Configuring Emacs, lsp-mode and the python language server.

The language server protocol was proposed by Microsoft as a way for different editors and development environments to share language analysis backends

This post describes how to configure Emacs, lsp-mode and the palantir python-language-server for improved code intelligence when working on Python projects. (I’m planning a companion post for Emacs, C++ and the cquery language server.)

Goal

Before starting, it is motivating to see what we are working towards.

With a correctly configured setup, Emacs will sport, amongst others, improved completion with interactive documentation, imenu navigation, documentation on hover, and really snazzy find definitions (M-.) and find references.

See the following screenshots for some examples:

Pre-requisites on the Python side

Install the python-language-server into the virtual environment, or user environment, that you’re planning to use.

These days, I tend to use pipenv:

cd my_project
pipenv install python-language-server[all]

The [all] means that it installs all optional providers, e.g. yapf formatting.

Pre-requisites on the Emacs side

In Emacs, install the required and some optional packages using for example M-x package-install:

lsp-mode – the main language server protocol package
lsp-ui – UI-related LSP extras, such as the sideline info, docs, flycheck, etc.
company-lsp – company-backend for LSP-based code completion.
projectile or find-file-in-project – we use a single function from here to determine the root directory of a project.

Emacs configuration

Add the following to your Emacs init.el, and don’t forget to read the comments.

If you’re not yet using use-package now would be a good time to upgrade.

(use-package lsp-mode
  :ensure t
  :config

  ;; make sure we have lsp-imenu everywhere we have LSP
  (require 'lsp-imenu)
  (add-hook 'lsp-after-open-hook 'lsp-enable-imenu)  
  ;; get lsp-python-enable defined
  ;; NB: use either projectile-project-root or ffip-get-project-root-directory
  ;;     or any other function that can be used to find the root directory of a project
  (lsp-define-stdio-client lsp-python "python"
                           #'projectile-project-root
                           '("pyls"))

  ;; make sure this is activated when python-mode is activated
  ;; lsp-python-enable is created by macro above 
  (add-hook 'python-mode-hook
            (lambda ()
              (lsp-python-enable)))

  ;; lsp extras
  (use-package lsp-ui
    :ensure t
    :config
    (setq lsp-ui-sideline-ignore-duplicate t)
    (add-hook 'lsp-mode-hook 'lsp-ui-mode))

  (use-package company-lsp
    :config
    (push 'company-lsp company-backends))

  ;; NB: only required if you prefer flake8 instead of the default
  ;; send pyls config via lsp-after-initialize-hook -- harmless for
  ;; other servers due to pyls key, but would prefer only sending this
  ;; when pyls gets initialised (:initialize function in
  ;; lsp-define-stdio-client is invoked too early (before server
  ;; start)) -- cpbotha
  (defun lsp-set-cfg ()
    (let ((lsp-cfg `(:pyls (:configurationSources ("flake8")))))
      ;; TODO: check lsp--cur-workspace here to decide per server / project
      (lsp--set-configuration lsp-cfg)))

  (add-hook 'lsp-after-initialize-hook 'lsp-set-cfg))

Putting it all together

Importantly, use pyvenv or something similar to switch to the relevant virtualenv before opening the first Python file.

When you open the file, the pyls should be automatically started up, and you can edit away with LSP-powered code intelligence.

This often gives better and more detailed results than elpy, probably because pyls uses a mix of static and dynamic (introspection-based) analysis.

Furthermore, the handling of LSP servers in Emacs can be unified, giving the same consistent level of support across a whole range of programming languages.

Interactive programming with Fennel Lua Lisp, Emacs and Lisp Game Jam winner EXO_encounter 667

Phil Hagelberg recently won the Lisp Game Jam 2018 with his entry EXO_encounter 667.

What I found most interesting however, was his interactive programming setup.

He programmed his game in (and contributed new features to) a Lisp to Lua compiler called Fennel, and used the game programming library Löve.

With Emacs and some Lua thread magic, he was able to perform runtime changes and introspection to his live running game project. (See below for a demo!)

Based on past experience developing visualization and image processing algorithms, I learned how useful this sort of interactive / runtime programming could be.

Hagelberg wrote up his experience in three great blog posts:

… and he has made the full source code to EXO_encounter 667 available on gitlab, so I had to try the interactive programming setup out for myself.

Due to his great write-ups, this was surprisingly easy.

Below you’ll find a short screencast of the setup in action, the steps I took to get everything running, and finally some information on how he put the interactive programming parts of the game together.

Demonstration

Here is a short video demonstrating a live programming session:

Quickstart

Here are the steps I followed to get everything up and running:

Install löve, lua and fennel

brew install caskroom/cask/love
brew install lua
luarocks install --server=http://luarocks.org/dev fennel

Install the Emacs fennel-mode

This important piece of code is also by Hagelberg.

Check out fennel-mode where you usually work with github and gitlab checkouts:

mkdir ~/build && cd ~/build
git clone https://gitlab.com/technomancy/fennel-mode.git

Evaluate the following two lines in Emacs using for example M-x eval-region:

(autoload 'fennel-mode (expand-file-name "~/build/fennel-mode/fennel-mode") nil t)
(add-to-list 'auto-mode-alist '("\\.fnl\\'" . fennel-mode))

Get and start playing with EXO_encounter 667

git clone https://gitlab.com/technomancy/exo-encounter-667.git

Start by opening wrap.fnl in the root directory.

Then, as per the instructions, start the Fennel repl using C-u M-x run-lisp. This will ask you which lisp to use. Replace the default fennel --repl with love . (that’s love followed by space and a period)

At this point you will get a repl via which you can enter fennel commands. You can also edit any of the top-level fennel files, and type C-c C-k to reload the whole file, and watch the game change before your eyes.

More detail on how the interactive programming parts work

When you start love . from within Emacs fennel-mode, this runs the game, but starts an extra Lua thread to listen for input from Emacs.

(see the Interactive Development section in the blog post titled “in which a game jam is recounted further“)

Looking at the source, main.lua bootstraps fennel and loads in wrap.fnl which contains the familiar love.load, love.draw and love.update callbacks.

In love.load, it starts the repl, which is loaded from lib.stdio, which is where the extra listener thread is started up.

Emacs fennel-mode does the rest. Once you’ve done run-lisp with love ., you can use all the dynamic commands described on the fennel-mode gitlab page.

Asynchronous rsync with Emacs, dired and tramp.

tmtxt-dired-async by Trần Xuân Trường is an unfortunately lesser known Emacs package which extends dired, the Emacs file manager, to be able to run rsync and other commands (zip, unzip, downloading) asynchronously.

This means you can copy gigabytes of directories around whilst still happily continuing with all of your other tasks in the Emacs operating system.

It has a feature where you can add any number of files from different locations into a wait list with C-c C-a, and then asynchronously rsync the whole wait list into a final destination directory with C-c C-v. This alone is worth the price of admission.

For example here it is pointlessly rsyncing the arduino 1.9 beta archive to another directory:

When the process is complete, the window at the bottom will automatically be killed after 5 seconds. Here is a separate session right after the asynchronous unzipping of the above-mentioned arduino archive:

This package has further increased the utility of my dired configuration.

I just contributed a pull request that enables tmtxt-dired-async to rsync to remote tramp-based directories, and I immediately used this new functionality to sort a few gigabytes of new photos onto the Linux server.

To add tmtxt-dired-async to your config, download tmtxt-async-tasks.el (a required library) and tmtxt-dired-async.el (check that my PR is in there if you plan to use this with tramp) into your ~/.emacs.d/ and add the following to your config:

;; no MELPA packages of this, so we have to do a simple check here
(setq dired-async-el (expand-file-name "~/.emacs.d/tmtxt-dired-async.el"))
(when (file-exists-p dired-async-el)
  (load (expand-file-name "~/.emacs.d/tmtxt-async-tasks.el"))
  (load dired-async-el)
  (define-key dired-mode-map (kbd "C-c C-r") 'tda/rsync)
  (define-key dired-mode-map (kbd "C-c C-z") 'tda/zip)
  (define-key dired-mode-map (kbd "C-c C-u") 'tda/unzip)

  (define-key dired-mode-map (kbd "C-c C-a") 'tda/rsync-multiple-mark-file)
  (define-key dired-mode-map (kbd "C-c C-e") 'tda/rsync-multiple-empty-list)
  (define-key dired-mode-map (kbd "C-c C-d") 'tda/rsync-multiple-remove-item)
  (define-key dired-mode-map (kbd "C-c C-v") 'tda/rsync-multiple)

  (define-key dired-mode-map (kbd "C-c C-s") 'tda/get-files-size)

  (define-key dired-mode-map (kbd "C-c C-q") 'tda/download-to-current-dir))

Enjoy!

Developing Arduino sketches with JetBrains CLion: A minimal example.

The official Arduino Desktop IDE is fantastic at what it was made for. After downloading, opening your first sketch (say, blink.ino) and flashing this to your connected Arduino hardware takes all of 3 seconds.

However, once your sketches become a little more complex, a more sophisticated IDE with code navigation, documentation and context-sensitive completion can be a great help.

Currently, one of the better solutions is the Arduino extension for Visual Studio Code. You can be up and running quite quickly, and after adding the necessary include directories to your config, the built-in IntelliSense C++ helps immensely with code completion, navigation and inline documentation.

Overview of this barebones solution

However, this post is about getting CLion working with your Arduino projects, without using any additional software besides the Arduino IDE and CLion.

It’s slightly less straight-forward than with Visual Studio Code, but could be worth it, as CLion is arguably a better C++ and general programming IDE than Visual Studio Code.

An example of CLion editing the converted blink sketch with the m0 toolchain file installed. The documentation for digitalWrite is shown inline. At the left the file’s structure, and below the serial monitor plugin for CLion that can be used to see what the arduino is sending us.

The core of the solution is to create a cmake toolchain file for the Arduino, based on compilation parameters extracted from a verbose run of the Arduino Desktop IDE 1.8.5 in command-line mode.

Furthermore, it is important that all sketch code is moved out into .cpp files, and the necessary includes (importantly Arduino.h) and function prototypes are added. The main .ino (sketch) file has to be maintained with the same name as the containing directory, but it can be empty, which is what I usually do.

There are CLion arduino plugins which you could try (I could not get any of them working completely), but with this minimal example, you get to know exactly what is going on behind the scenes.

Up and running CLion, cmake and the blink example

To get you started, I have converted the minimal stock Arduino blink sketch to a CLion-compatible project, including toolchain files for the AVR-based Uno and for the ARM-based M0 boards, and made it available on github as arduino-clion-minimal. You can easily modify your existing projects by just copying and modifying the CMakeLists.txt and the relevant toolchain file.

To try out CLionized blink, clone this repo, then open it with CLion.

Importantly, go to Preferences | build, execution, deployment | CMake and then add -DCMAKE_TOOLCHAIN_FILE=arduino-uno-toolchain.cmake or -DCMAKE_TOOLCHAIN_FILE=arduino-m0plus-toolchain.cmake (my current favourite Arduino hardware!) depending on your hardware platform.

After opening, you should now have three build (Ctrl-F9 – the button to the left of the target selection, NOT run) targets at the top right of the CLion UI: arduino-clion-minimal for quick compile-only checks, verify for full arduino building, and upload for full arduino building and uploading:

Before you upload, make sure that you’ve selected the correct board type and port with the Arduino desktop IDE.

Conclusions

With this simple setup, you should have access to all of CLion’s programming facilities during the development of your Arduino sketches.

Furthermore, Dmitry Cherkas’s Serial Port Monitor plugin can be used as Arduino serial port monitor for a more fully integrated experience.

Let me know in the comments how it went!

Which jumper to set on the ITEAD XBee shield v1.1 for use with a 3.3V Arduino

I had to use the ITEAD Studio XBee shield v1.1 with an Arduino m0 (SAMD21) board, which is a 3.3V board, whereas the most common Arduinos are 5V.

At the time of this writing, the shield’s website was not very clear on how exactly to set the jumpers (zone 5: “When operated in 3.3V, install the jumper” — which one?!), and the rest of the internet also did not seem to know.

Fortunately their support did eventually get around to my requests, and the setup seems to work, so now I can put this information online to help future travellers.

Keeping the “xbee shield v1.1” text the right side up, only the left jumper in zone 5 should be set. Here’s a photo of the shield mounted on an Arduino M0 clone:

For use with a 3.3V Arduino, only the left jumper in block 5 should be installed.

Here is the illustration that was sent by itead tech support:

Let me know in the comments what your experiences were with this shield!