Configuring Emacs, lsp-mode and the python language server.

The language server protocol was proposed by Microsoft as a way for different editors and development environments to share language analysis backends

This post describes how to configure Emacs, lsp-mode and the palantir python-language-server for improved code intelligence when working on Python projects. (I’m planning a companion post for Emacs, C++ and the cquery language server.)

Goal

Before starting, it is motivating to see what we are working towards.

With a correctly configured setup, Emacs will sport, amongst others, improved completion with interactive documentation, imenu navigation, documentation on hover, and really snazzy find definitions (M-.) and find references.

See the following screenshots for some examples:

Screen Shot 2018-06-08 at 13.01.32_2018-06-08_14-29-25.png

lsp-python-imenu_2018-06-08_14-34-27.png

lsp-python-docs-on-hover_2018-06-08_14-34-43.png

Pre-requisites on the Python side

Install the python-language-server into the virtual environment, or user environment, that you’re planning to use.

These days, I tend to use pipenv:

cd my_project
pipenv install python-language-server[all]

The [all] means that it installs all optional providers, e.g. yapf formatting.

Pre-requisites on the Emacs side

In Emacs, install the required and some optional packages using for example M-x package-install:

  • lsp-mode – the main language server protocol package
  • lsp-ui – UI-related LSP extras, such as the sideline info, docs, flycheck, etc.
  • company-lsp – company-backend for LSP-based code completion.
  • projectile or find-file-in-project – we use a single function from here to determine the root directory of a project.

Emacs configuration

Add the following to your Emacs init.el, and don’t forget to read the comments.

If you’re not yet using use-package now would be a good time to upgrade.

(use-package lsp-mode
  :ensure t
  :config

  ;; make sure we have lsp-imenu everywhere we have LSP
  (require 'lsp-imenu)
  (add-hook 'lsp-after-open-hook 'lsp-enable-imenu)  
  ;; get lsp-python-enable defined
  ;; NB: use either projectile-project-root or ffip-get-project-root-directory
  ;;     or any other function that can be used to find the root directory of a project
  (lsp-define-stdio-client lsp-python "python"
                           #'projectile-project-root
                           '("pyls"))

  ;; make sure this is activated when python-mode is activated
  ;; lsp-python-enable is created by macro above 
  (add-hook 'python-mode-hook
            (lambda ()
              (lsp-python-enable)))

  ;; lsp extras
  (use-package lsp-ui
    :ensure t
    :config
    (setq lsp-ui-sideline-ignore-duplicate t)
    (add-hook 'lsp-mode-hook 'lsp-ui-mode))

  (use-package company-lsp
    :config
    (push 'company-lsp company-backends))

  ;; NB: only required if you prefer flake8 instead of the default
  ;; send pyls config via lsp-after-initialize-hook -- harmless for
  ;; other servers due to pyls key, but would prefer only sending this
  ;; when pyls gets initialised (:initialize function in
  ;; lsp-define-stdio-client is invoked too early (before server
  ;; start)) -- cpbotha
  (defun lsp-set-cfg ()
    (let ((lsp-cfg `(:pyls (:configurationSources ("flake8")))))
      ;; TODO: check lsp--cur-workspace here to decide per server / project
      (lsp--set-configuration lsp-cfg)))

  (add-hook 'lsp-after-initialize-hook 'lsp-set-cfg))

Putting it all together

Importantly, use pyvenv or something similar to switch to the relevant virtualenv before opening the first Python file.

When you open the file, the pyls should be automatically started up, and you can edit away with LSP-powered code intelligence.

This often gives better and more detailed results than elpy, probably because pyls uses a mix of static and dynamic (introspection-based) analysis.

Furthermore, the handling of LSP servers in Emacs can be unified, giving the same consistent level of support across a whole range of programming languages.

Interactive programming with Fennel Lua Lisp, Emacs and Lisp Game Jam winner EXO_encounter 667

Phil Hagelberg recently won the Lisp Game Jam 2018 with his entry EXO_encounter 667.

What I found most interesting however, was his interactive programming setup.

He programmed his game in (and contributed new features to) a Lisp to Lua compiler called Fennel, and used the game programming library Löve.

With Emacs and some Lua thread magic, he was able to perform runtime changes and introspection to his live running game project. (See below for a demo!)

Based on past experience developing visualization and image processing algorithms, I learned how useful this sort of interactive / runtime programming could be.

Hagelberg wrote up his experience in three great blog posts:

… and he has made the full source code to EXO_encounter 667 available on gitlab, so I had to try the interactive programming setup out for myself.

Due to his great write-ups, this was surprisingly easy.

Below you’ll find a short screencast of the setup in action, the steps I took to get everything running, and finally some information on how he put the interactive programming parts of the game together.

Demonstration

Here is a short video demonstrating a live programming session:

Quickstart

Here are the steps I followed to get everything up and running:

Install löve, lua and fennel

brew install caskroom/cask/love
brew install lua
luarocks install --server=http://luarocks.org/dev fennel

Install the Emacs fennel-mode

This important piece of code is also by Hagelberg.

Check out fennel-mode where you usually work with github and gitlab checkouts:

mkdir ~/build && cd ~/build
git clone https://gitlab.com/technomancy/fennel-mode.git

Evaluate the following two lines in Emacs using for example M-x eval-region:

(autoload 'fennel-mode (expand-file-name "~/build/fennel-mode/fennel-mode") nil t)
(add-to-list 'auto-mode-alist '("\\.fnl\\'" . fennel-mode))

Get and start playing with EXO_encounter 667

git clone https://gitlab.com/technomancy/exo-encounter-667.git

Start by opening wrap.fnl in the root directory.

Then, as per the instructions, start the Fennel repl using C-u M-x run-lisp. This will ask you which lisp to use. Replace the default fennel --repl with love . (that’s love followed by space and a period)

At this point you will get a repl via which you can enter fennel commands. You can also edit any of the top-level fennel files, and type C-c C-k to reload the whole file, and watch the game change before your eyes.

More detail on how the interactive programming parts work

When you start love . from within Emacs fennel-mode, this runs the game, but starts an extra Lua thread to listen for input from Emacs.

(see the Interactive Development section in the blog post titled “in which a game jam is recounted further“)

Looking at the source, main.lua bootstraps fennel and loads in wrap.fnl which contains the familiar love.load, love.draw and love.update callbacks.

In love.load, it starts the repl, which is loaded from lib.stdio, which is where the extra listener thread is started up.

Emacs fennel-mode does the rest. Once you’ve done run-lisp with love ., you can use all the dynamic commands described on the fennel-mode gitlab page.

Asynchronous rsync with Emacs, dired and tramp.

tmtxt-dired-async by Trần Xuân Trường is an unfortunately lesser known Emacs package which extends dired, the Emacs file manager, to be able to run rsync and other commands (zip, unzip, downloading) asynchronously.

This means you can copy gigabytes of directories around whilst still happily continuing with all of your other tasks in the Emacs operating system.

It has a feature where you can add any number of files from different locations into a wait list with C-c C-a, and then asynchronously rsync the whole wait list into a final destination directory with C-c C-v. This alone is worth the price of admission.

For example here it is pointlessly rsyncing the arduino 1.9 beta archive to another directory:

When the process is complete, the window at the bottom will automatically be killed after 5 seconds. Here is a separate session right after the asynchronous unzipping of the above-mentioned arduino archive:

This package has further increased the utility of my dired configuration.

I just contributed a pull request that enables tmtxt-dired-async to rsync to remote tramp-based directories, and I immediately used this new functionality to sort a few gigabytes of new photos onto the Linux server.

To add tmtxt-dired-async to your config, download tmtxt-async-tasks.el (a required library) and tmtxt-dired-async.el (check that my PR is in there if you plan to use this with tramp) into your ~/.emacs.d/ and add the following to your config:

;; no MELPA packages of this, so we have to do a simple check here
(setq dired-async-el (expand-file-name "~/.emacs.d/tmtxt-dired-async.el"))
(when (file-exists-p dired-async-el)
  (load (expand-file-name "~/.emacs.d/tmtxt-async-tasks.el"))
  (load dired-async-el)
  (define-key dired-mode-map (kbd "C-c C-r") 'tda/rsync)
  (define-key dired-mode-map (kbd "C-c C-z") 'tda/zip)
  (define-key dired-mode-map (kbd "C-c C-u") 'tda/unzip)

  (define-key dired-mode-map (kbd "C-c C-a") 'tda/rsync-multiple-mark-file)
  (define-key dired-mode-map (kbd "C-c C-e") 'tda/rsync-multiple-empty-list)
  (define-key dired-mode-map (kbd "C-c C-d") 'tda/rsync-multiple-remove-item)
  (define-key dired-mode-map (kbd "C-c C-v") 'tda/rsync-multiple)

  (define-key dired-mode-map (kbd "C-c C-s") 'tda/get-files-size)

  (define-key dired-mode-map (kbd "C-c C-q") 'tda/download-to-current-dir))

Enjoy!

 

Run code on remote ipython kernels with Emacs and orgmode.

As is briefly documented on the ob-ipython github, one can run code on remote ipython kernels.

In this post, I give a little more detail, and show that this also works wonderfully for remote generation but local embedding of graphics in Emacs Org mode.

As I hinted previously, the jupyter notebook is a great interface for computational coding, but Emacs and Org mode offer far more flexible editing and are more robust as a documentation format.

On to the show. (This whole blog post is a single Org mode file with org-babel source code blocks, the last of which is live.)

Start by starting the ipython kernel on the remote machine:

me@server$ jupyter --runtime-dir
>>> /run/user/1000/jupyter

me@server$ ipython kernel
>>> To connect another client to this kernel, use:
>>>    --existing kernel-11925.json

We have to copy that json connection file to the client machine, and then connect to it with the jupyter console:

me@client$ jupyter --runtime-dir
>>> /Users/cpbotha/Library/Jupyter/runtime

me@client$ cd /Users/cpbotha/Library/Jupyter/runtime
me@client$ scp me@server:/run/user/1000/jupyter/kernel-11925.json .
me@client$ jupyter console --existing kernel-12818.json --ssh meepz97
>>> [ZMQTerminalIPythonApp] To connect another client via this tunnel, use:
>>> [ZMQTerminalIPythonApp] --existing kernel-12818-ssh.json

Note that we copy the json file into our local jupyter runtime directory, which will create the ssh connection file there, and enable us to reference it by name.json only (vs its full path) in any ob-ipython source code blocks.

Now you can open ob-ipython org-babel source blocks which will connect to the remote kernel. They start like this:

#+BEGIN_SRC ipython :session kernel-12818-ssh.json :exports both :results raw drawer

Let’s try it out:

%matplotlib inline
# changed to png only for the blog post
%config InlineBackend.figure_format = 'png'

from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt
from matplotlib import cm

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
X, Y, Z = axes3d.get_test_data(0.05)
cset = ax.contour(X, Y, Z, cmap=cm.coolwarm)
ax.clabel(cset, fontsize=9, inline=1)

plt.show()

428ndb.png

Pretty amazing! The code is executed on the remote machine, and the resultant plot is piped back and displayed embedded in orgmode as SVG!

Getting ob-ipython to show documentation during company completion.

ob-ipython is an Emacs package that enables org-babel to talk to a running ipython kernel. The upshot of this is that you can use org-mode instead of the jupyter notebook for interspersing executable code, results and documentation.

The screenshot from the ob-ipython github shows it in action: ob-ipython-github-screenshot.jpg

Personally, I would like to use this for controlling ipython kernels on remote GPU- and deep learning-capable Linux machines, all via Emacs on my laptop. The juyter notebook is really fantastic, but it’s not Emacs.

You could also use ein for this, but then you would have to give up org-mode.

As I was testing ob-ipython yesterday, I noticed that its company-backend (completion system for Emacs) doc-buffer support was absent. Usually, as you’re exploring possible code completions, you can press <f1> or C-h to show help on the currently highlighted completion candidate.

Fast-forward an hour or two of Emacs Lisp surgery, and I was able to hook up the ob-ipython company-mode backend to ob-ipython’s inspection facility. Now pressing C-h gets you detailed help in a company-documentation buffer!

Here is my github pull request, and here is a screenshot of the company-mode ob-ipython documentation in action:

ob-ipython-company-doc-buffer.png

Hopefully this will be merged soon so it can find its way onto the Melpa package archives.

Here’s a bonus screenshot showing the ob-ipython notes from my org-mode journal where you can see embedded Python code that has been executed via the connected ipython kernel, with the resultant SVG format plot embedded and displayed inline:

ob-ipython-notes-example-nov-2017-3.png

P.S. I am currently disabling elpy-mode when the ob-ipython minor mode is active, until I figure out a better solution to elpy interfering with ob-ipython.

(use-package ob-ipython
  :config
  ;; for now I am disabling elpy only ob-ipython minor mode
  ;; what we should actually do, is just to ensure that
  ;; ob-ipython's company backend comes before elpy's (TODO)
  (add-hook 'ob-ipython-mode-hookp
            (lambda ()
              (elpy-mode 0)
              (company-mode 1)))
  (add-to-list 'company-backends 'company-ob-ipython)
  (add-to-list 'org-latex-minted-langs '(ipython "python")))