## Configuring Emacs, lsp-mode and Microsoft’s Visual Studio Code Python language server.

In a previous post I showed how to get Palantir’s Python Language Server working together with Emacs and lsp-mode.

In this post, we look at the brand new elephant in the room, Microsoft’s arguably far more powerful own Python Language Server, and how to integrate it with Emacs.

## Motivation

Since that previous post on Palantir’s language server, I’ve been using Emacs far more intensively for Python coding in tmux on remote machines with GPUs for deep learning.

The interactive programming possibilities of Emacs (remember that Lisp programmers have been doing this since the 60s) make for a great development solution: I can interact with my remotely running neural network code, start a long-running training, and then detach from the running tmux. When I re-attach the next morning (or the next), I can continue interactively experimenting with the still-running Python instance, for example further fine-tuning the training.

Palantir’s Python Language Server can become sluggish at times, so I switched back to elpy which is usually quite snappy, and affords an impressive suite of code intelligence features for such a small package.

However, the elpy RPC Python process has a tendency to die quite often, and even M-x elpy-rpc-restart simply stops working at some point.

This, together with the fact that Microsoft’s own Python Language Server, the one used for both Visual Studio Code and also in Microsoft’s flagship Visual Studio IDE, has enjoyed and will continue to enjoy a far larger share of developer mindshare and attention, encouraged me to try and get this language server also working with Emacs.

This exercise cost many more hours than I was planning to spend, but everything seems to be working now.

/In this post, I will explain step-by-step how you too can enjoy the highly multi-threaded and actively developed Microsoft Python Language Server in your Emacs./

## The Goal

If you follow the steps set out further down in this blog post, your Emacs too could end up looking like the one showed in the following screenshots.

## Two methods: The new and improved, and the old and complicated

Shortly after I wrote this blog post, Andrew Christianson turned everything here into a far easier to use Emacs package which you can conveniently download from github.

You can safely ignore the next subsection, which I’m keeping here purely for historical purposes.

It’s really great when you put together a blog post, and then someone else takes that ball and RUNS with it!

### The OLD way: Four steps to combining Emacs with Microsoft’s Python Language Server

Read through all of the steps first, and then follow them carefully.

Let me know in the comments if anything should be further clarified.

#### Build the Microsoft Python Language Server using dotnet

Next, build the language server:

mkdir ~/build
cd ~/build
git clone https://github.com/Microsoft/python-language-server.git
cd python-language-server/src/LanguageServer/Impl
dotnet build -c Release


#### Install required Emacs packages

In Emacs, install the required and some optional packages using for example M-x package-install:

• lsp-mode – the main language server protocol package
• lsp-ui – UI-related LSP extras, such as the sideline info, docs, flycheck, etc.
• company-lsp – company-backend for LSP-based code completion.
• projectile or find-file-in-project – we use a single function from here to determine the root directory of a project.

#### Required change to lsp-ui-doc.el

Microsoft’s Python Language Server likes to replace spaces in the documentation it returns with &nbsp; HTML entities.

Furthermore, there seems to be an additional misunderstanding between Emacs lsp-mode and the MS PyLS with regard to the interpretation of markdown and plaintext docstrings.

Both of these issues impact the blue documentation overlay, and should be worked around by editing the lsp-ui-doc-extract function in lsp-ui-doc.el.

Right before the line with

((gethash "kind" contents) (gethash "value" contents)) ;; MarkupContent


;; cpbotha: with numpy functions, e.g. np.array for example,
;; kind=markdown and docs are in markdown, but in default
;; lsp-ui-20181031 this is rendered as plaintext see
;; https://microsoft.github.io/language-server-protocol/specification#markupcontent

;; not only that, MS PyLS turns all spaces into &nbsp; instances,
;; which we remove here this single additional cond clause fixes all
;; of this for hover

;; as if that was not enough: import pandas as pd - pandas is returned
;; with kind plaintext but contents markdown, whereas pd is returned
;; with kind markdown. fortunately, handling plaintext with the
;; markdown viewer still looks good, so here we are.
((member (gethash "kind" contents) '("markdown" "plaintext"))
(replace-regexp-in-string "&nbsp;" " " (lsp-ui-doc--extract-marked-string contents)))


#### Emacs configuration

Add the following to your Emacs init.el, and don’t forget to read the comments.

If you’re not yet using use-package now would be a good time to upgrade.

(use-package lsp-mode
:ensure t
:config

;; change nil to 't to enable logging of packets between emacs and the LS
;; this was invaluable for debugging communication with the MS Python Language Server
;; and comparing this with what vs.code is doing
(setq lsp-print-io nil)

;; lsp-ui gives us the blue documentation boxes and the sidebar info
(use-package lsp-ui
:ensure t
:config
(setq lsp-ui-sideline-ignore-duplicate t)

;; make sure we have lsp-imenu everywhere we have LSP

;; install LSP company backend for LSP-driven completion
(use-package company-lsp
:ensure t
:config
(push 'company-lsp company-backends))

;; dir containing Microsoft.Python.LanguageServer.dll
(setq ms-pyls-dir (expand-file-name "~/build/python-language-server/output/bin/Release/"))

;; this gets called when we do lsp-describe-thing-at-point in lsp-methods.el
;; we remove all of the "&nbsp;" entities that MS PYLS adds
;; this is mostly harmless for other language servers
(defun render-markup-content (kind content)
(message kind)
(replace-regexp-in-string "&nbsp;" " " content))
(setq lsp-render-markdown-markup-content #'render-markup-content)

;; it's crucial that we send the correct Python version to MS PYLS, else it returns no docs in many cases
;; furthermore, we send the current Python's (can be virtualenv) sys.path as searchPaths
(defun get-python-ver-and-syspath (workspace-root)
"return list with pyver-string and json-encoded list of python search paths."
(let ((python (executable-find python-shell-interpreter))
(init "from __future__ import print_function; import sys; import json;")
(ver "print(\"%s.%s\" % (sys.version_info[0], sys.version_info[1]));")
(sp (concat "sys.path.insert(0, '" workspace-root "'); print(json.dumps(sys.path))")))
(with-temp-buffer
(call-process python nil t nil "-c" (concat init ver sp))
(subseq (split-string (buffer-string) "\n") 0 2))))

;; I based most of this on the vs.code implementation:
;; https://github.com/Microsoft/vscode-python/blob/master/src/client/activation/languageServer/languageServer.ts#L219
;; (it still took quite a while to get right, but here we are!)
(defun ms-pyls-extra-init-params (workspace)
(destructuring-bind (pyver pysyspath) (get-python-ver-and-syspath (lsp--workspace-root workspace))
(:interpreter (
:properties (
:InterpreterPath ,(executable-find python-shell-interpreter)
;; this database dir will be created if required
:DatabasePath ,(expand-file-name (concat ms-pyls-dir "db/"))
:Version ,pyver))
;; preferredFormat "markdown" or "plaintext"
;; experiment to find what works best -- over here mostly plaintext
:displayOptions (
:preferredFormat "plaintext"
:trimDocumentationLines :json-false
:maxDocumentationLineLength 0
:trimDocumentationText :json-false
:maxDocumentationTextLength 0)

(lsp-define-stdio-client lsp-python "python"
#'ffip-get-project-root-directory
("dotnet" ,(concat ms-pyls-dir "Microsoft.Python.LanguageServer.dll"))
:extra-init-params #'ms-pyls-extra-init-params)

;; lsp-python-enable is created by macro above
(lambda ()
(lsp-python-enable)))



## Conclusions

Although I would have preferred to do this with the two lsp-mode work-arounds, I am pretty satisfied with this setup.

With the number of users and development effort Microsoft’s Python Language Server has been enjoying and will probably continue to enjoy, it’s great knowing we can make use of this functionality in Emacs.

I am curious how well eglot, a smaller Emacs LSP package than lsp-mode, would do based on the integration above. (hint hint…)

### 2018-12-26

Andrew Christianson converted all of the complicated instructions in this blog post into a far easier to use Emacs package called lsp-python-ms!

### 2018-11-22

Uploaded new version of emacs-lisp init code with two improvements:

• Thanks to reddit user cyanj for the print_function import suggestion, and commenter Erik Hetzner below for the old-style string interpolation suggestion, this setup should now also work with Python 2, in addition to 3.
• The ms-pyls database directory is now created as a subdirectory of the bindir.

### 2018-11-20

So far, I have logged two issues, one with MS PyLS and one with lsp-mode, so that we can hopefully one day remove some of the work-arounds detailed above:

## Importing all of your orgmode notes into Apple Notes for mobile access.

Over the years, I’ve built up quite a collection of notes as Org mode text files. So far, it has proven to be the most expressive and the most robust note-taking modality out of a long list of candidates that I’ve tried.

Note-taking using Org mode has one big drawback however: Mobile accessibility.

In other words, consulting one’s org mode notes database from a mobile device is painful. This should not be the case; notes should be always and instantly available, even on mobile.

In this blog post, I show you how to import your complete org mode notes database into Apple Notes, including typesetting and attachments, using the org mode publishing functionality.

To be clear: Org mode on the desktop remains my primary note-taking system. The goal of importing all of my notes into Apple Notes is only to have my personal knowledge base accessible from my mobile devices.

## End result

After configuring and running org-publish, and then importing the whole directory of exported HTML files and attachments into Apple Notes on macOS, your notes will look like one of the two examples below: First macOS, then iOS on the phone.

Note the Emacs-supplied syntax highlighting, and the inline image.

If you import these to your icloud account (the default) the notes will be available on all of your iOS mobile devices.

These imported notes are fully indexed, and hence searchable from all of your devices.

## org-publish configuration

In order to make this happen, we make use of the org-publish functionality. We also configure one or two Apple Notes-specific changes to improve rendering.

Add the configuration below to your init.el.

There are two publish targets: One for the org files (called pkb4000 below), and one for all of the attachments (called pkb4000-static below).

As an aside, pkb4000 is short for Personal Knowledge Base 4000. I chose the name as a joke, as this synced directory of org mode files felt like just the Nth in a long series of knowledge base iterations. Little did I know how well this one would stick.

Remember to change both the :base-directory properties to the top-level directory of your notes database. :publishing-directory is anywhere convenient where you would like to store the published HTML files and attachments.

;; https://orgmode.org/manual/HTML-preamble-and-postamble.html
;; disable author + date + validate link at end of HTML exports
(setq org-html-postamble nil)

(defun org-html-publish-to-html-for-apple-notes (plist filename pub-dir)
"Convert blank lines to <br /> and remove <h1> titles."
;; temporarily configure export to convert math to images because
;; apple notes obviously can't use mathjax (the default)
(let* ((org-html-with-latex 'imagemagick)
(outfile
(org-publish-org-to 'html filename
(concat "." (or (plist-get plist :html-extension)
org-html-extension
"html"))
plist pub-dir)))
;; 1. apple notes handles <p> paras badly, so we have to replace all blank
;;    lines (which the orgmode export accurately leaves for us) with
;;    <br /> tags to get apple notes to actually render blank lines between
;;    paragraphs
;; 2. remove large h1 with title, as apple notes already adds <title> as
;; the note title
(shell-command
(format "sed -i \"\" -e 's/^$/<br \\/>/' -e 's/<h1 class=\"title\">.*<\\/h1>$//' %s"
outfile))
outfile))

(setq org-publish-project-alist
'(("pkb4000"
:base-directory "~/Dropbox/notes/pkb4000/"
:recursive t
:publishing-function org-html-publish-to-html-for-apple-notes
:section-numbers nil
:with-toc nil)
("pkb4000-static"
:base-directory "~/Dropbox/notes/pkb4000/"
:base-extension "css\\|js\\|png\\|jpg\\|gif\\|pdf\\|mp3\\|ogg\\|swf"
:recursive t
:publishing-function org-publish-attachment
)))


## Publish your database and import to Apple Notes

Start the process by M-x org-publish, at which point Emacs will ask you to select a target. You have to org-publish both of the targets.

If you have a large database, Emacs might report errors in your org files. Fix these, and restart the process. org-publish caches its output, so re-runs should not take that long.

After a successful publish, import the whole :publishing-directory into Apple Notes on macOS by selecting File | Import from the menu.

Apple Notes will create a new Imported Notes folder containing your whole notes hierarchy.

This process can be easily repeated when you want to refresh the database on your Apple Notes, but unfortunately Notes will create a new Imported Notes N folder.

Alternatively, you can re-publish, and then import just changed HTML files one by one, after which you’ll have to move them back into the correct Apple Notes folder.

## Conclusion

This solves the problem of being able to search rapidly and consult your whole org mode notes database using your iOS mobile device.

However, it does not yet solve the problem of importing Apple Notes you create on the mobile device back into Orgmode. This is something one could consider trying to solve using AppleScript.

Whatever the case may be, this is still a nice improvement over my previous workflow!

## Bonus round: Convert orgmode buffer to Apple Note using AppleScript

Before I tried org-publish, I worked on some emacs-lisp and AppleScript to convert the current org mode buffer to HTML, and then to inject that into Apple Notes using Apple Script.

I am posting this here in case it might be useful to someone. However it is NOT required for the org-publish workflow described above.

This assumes that you have an orgmode folder.

Although far more humble than org-publish, this code will only create a new note if it does not exist yet. If the note already exists, it will simply update its contents to the current org file.

;; https://www.emacswiki.org/emacs/string-utils.el
(defun string-utils-escape-double-quotes (str-val)
"Return STR-VAL with every double-quote escaped with backslash."
(save-match-data
(replace-regexp-in-string "\"" "\\\\\"" str-val)))

(defun string-utils-escape-backslash (str-val)
"Return STR-VAL with every backslash escaped with an additional backslash."
(save-match-data
(replace-regexp-in-string "\\\\" "\\\\\\\\" str-val)))

(setq as-tmpl "set TITLE to \"%s\"
set NBODY to \"%s\"
tell application \"Notes\"
tell folder \"orgmode\"
if not (note named TITLE exists) then
make new note with properties {name:TITLE}
end if
set body of note TITLE to NBODY
end tell
end tell")

(defun oan-export ()
(interactive)
(let ((title (file-name-base (buffer-file-name))))
(with-current-buffer (org-export-to-buffer 'html "*orgmode-to-apple-notes*")
(let ((body (string-utils-escape-double-quotes
(string-utils-escape-backslash (buffer-string)))))
;; install title + body into template above and send to notes
(do-applescript (format as-tmpl title body))
;; get rid of temp orgmode-to-apple-notes buffer
(kill-buffer))
)))


p

## Configuring Emacs, lsp-mode and the python language server.

The language server protocol was proposed by Microsoft as a way for different editors and development environments to share language analysis backends

This post describes how to configure Emacs, lsp-mode and the palantir python-language-server for improved code intelligence when working on Python projects. (I’m planning a companion post for Emacs, C++ and the cquery language server.)

## Goal

Before starting, it is motivating to see what we are working towards.

With a correctly configured setup, Emacs will sport, amongst others, improved completion with interactive documentation, imenu navigation, documentation on hover, and really snazzy find definitions (M-.) and find references.

See the following screenshots for some examples:

## Pre-requisites on the Python side

Install the python-language-server into the virtual environment, or user environment, that you’re planning to use.

These days, I tend to use pipenv:

cd my_project
pipenv install python-language-server[all]


The [all] means that it installs all optional providers, e.g. yapf formatting.

## Pre-requisites on the Emacs side

In Emacs, install the required and some optional packages using for example M-x package-install:

• lsp-mode – the main language server protocol package
• lsp-ui – UI-related LSP extras, such as the sideline info, docs, flycheck, etc.
• company-lsp – company-backend for LSP-based code completion.
• projectile or find-file-in-project – we use a single function from here to determine the root directory of a project.

## Emacs configuration

Add the following to your Emacs init.el, and don’t forget to read the comments.

If you’re not yet using use-package now would be a good time to upgrade.

(use-package lsp-mode
:ensure t
:config

;; make sure we have lsp-imenu everywhere we have LSP
;; get lsp-python-enable defined
;; NB: use either projectile-project-root or ffip-get-project-root-directory
;;     or any other function that can be used to find the root directory of a project
(lsp-define-stdio-client lsp-python "python"
#'projectile-project-root
'("pyls"))

;; make sure this is activated when python-mode is activated
;; lsp-python-enable is created by macro above
(lambda ()
(lsp-python-enable)))

;; lsp extras
(use-package lsp-ui
:ensure t
:config
(setq lsp-ui-sideline-ignore-duplicate t)

(use-package company-lsp
:config
(push 'company-lsp company-backends))

;; NB: only required if you prefer flake8 instead of the default
;; send pyls config via lsp-after-initialize-hook -- harmless for
;; other servers due to pyls key, but would prefer only sending this
;; when pyls gets initialised (:initialize function in
;; lsp-define-stdio-client is invoked too early (before server
;; start)) -- cpbotha
(defun lsp-set-cfg ()
(let ((lsp-cfg (:pyls (:configurationSources ("flake8")))))
;; TODO: check lsp--cur-workspace here to decide per server / project
(lsp--set-configuration lsp-cfg)))



## Putting it all together

Importantly, use pyvenv or something similar to switch to the relevant virtualenv before opening the first Python file.

When you open the file, the pyls should be automatically started up, and you can edit away with LSP-powered code intelligence.

This often gives better and more detailed results than elpy, probably because pyls uses a mix of static and dynamic (introspection-based) analysis.

Furthermore, the handling of LSP servers in Emacs can be unified, giving the same consistent level of support across a whole range of programming languages.

## Interactive programming with Fennel Lua Lisp, Emacs and Lisp Game Jam winner EXO_encounter 667

Phil Hagelberg recently won the Lisp Game Jam 2018 with his entry EXO_encounter 667.

What I found most interesting however, was his interactive programming setup.

He programmed his game in (and contributed new features to) a Lisp to Lua compiler called Fennel, and used the game programming library Löve.

With Emacs and some Lua thread magic, he was able to perform runtime changes and introspection to his live running game project. (See below for a demo!)

Based on past experience developing visualization and image processing algorithms, I learned how useful this sort of interactive / runtime programming could be.

Hagelberg wrote up his experience in three great blog posts:

… and he has made the full source code to EXO_encounter 667 available on gitlab, so I had to try the interactive programming setup out for myself.

Due to his great write-ups, this was surprisingly easy.

Below you’ll find a short screencast of the setup in action, the steps I took to get everything running, and finally some information on how he put the interactive programming parts of the game together.

## Demonstration

Here is a short video demonstrating a live programming session:

## Quickstart

Here are the steps I followed to get everything up and running:

### Install löve, lua and fennel

brew install caskroom/cask/love
brew install lua
luarocks install --server=http://luarocks.org/dev fennel


### Install the Emacs fennel-mode

This important piece of code is also by Hagelberg.

Check out fennel-mode where you usually work with github and gitlab checkouts:

mkdir ~/build && cd ~/build
git clone https://gitlab.com/technomancy/fennel-mode.git


Evaluate the following two lines in Emacs using for example M-x eval-region:

(autoload 'fennel-mode (expand-file-name "~/build/fennel-mode/fennel-mode") nil t)


### Get and start playing with EXO_encounter 667

git clone https://gitlab.com/technomancy/exo-encounter-667.git


Start by opening wrap.fnl in the root directory.

Then, as per the instructions, start the Fennel repl using C-u M-x run-lisp. This will ask you which lisp to use. Replace the default fennel --repl with love . (that’s love followed by space and a period)

At this point you will get a repl via which you can enter fennel commands. You can also edit any of the top-level fennel files, and type C-c C-k to reload the whole file, and watch the game change before your eyes.

## More detail on how the interactive programming parts work

When you start love . from within Emacs fennel-mode, this runs the game, but starts an extra Lua thread to listen for input from Emacs.

(see the Interactive Development section in the blog post titled “in which a game jam is recounted further“)

Looking at the source, main.lua bootstraps fennel and loads in wrap.fnl which contains the familiar love.load, love.draw and love.update callbacks.

In love.load, it starts the repl, which is loaded from lib.stdio, which is where the extra listener thread is started up.

Emacs fennel-mode does the rest. Once you’ve done run-lisp with love ., you can use all the dynamic commands described on the fennel-mode gitlab page.

## Asynchronous rsync with Emacs, dired and tramp.

tmtxt-dired-async by Trần Xuân Trường is an unfortunately lesser known Emacs package which extends dired, the Emacs file manager, to be able to run rsync and other commands (zip, unzip, downloading) asynchronously.

This means you can copy gigabytes of directories around whilst still happily continuing with all of your other tasks in the Emacs operating system.

It has a feature where you can add any number of files from different locations into a wait list with C-c C-a, and then asynchronously rsync the whole wait list into a final destination directory with C-c C-v. This alone is worth the price of admission.

For example here it is pointlessly rsyncing the arduino 1.9 beta archive to another directory:

When the process is complete, the window at the bottom will automatically be killed after 5 seconds. Here is a separate session right after the asynchronous unzipping of the above-mentioned arduino archive:

This package has further increased the utility of my dired configuration.

I just contributed a pull request that enables tmtxt-dired-async to rsync to remote tramp-based directories, and I immediately used this new functionality to sort a few gigabytes of new photos onto the Linux server.

To add tmtxt-dired-async to your config, download tmtxt-async-tasks.el (a required library) and tmtxt-dired-async.el (check that my PR is in there if you plan to use this with tramp) into your ~/.emacs.d/ and add the following to your config:

;; no MELPA packages of this, so we have to do a simple check here
(setq dired-async-el (expand-file-name "~/.emacs.d/tmtxt-dired-async.el"))
(when (file-exists-p dired-async-el)
(define-key dired-mode-map (kbd "C-c C-r") 'tda/rsync)
(define-key dired-mode-map (kbd "C-c C-z") 'tda/zip)
(define-key dired-mode-map (kbd "C-c C-u") 'tda/unzip)

(define-key dired-mode-map (kbd "C-c C-a") 'tda/rsync-multiple-mark-file)
(define-key dired-mode-map (kbd "C-c C-e") 'tda/rsync-multiple-empty-list)
(define-key dired-mode-map (kbd "C-c C-d") 'tda/rsync-multiple-remove-item)
(define-key dired-mode-map (kbd "C-c C-v") 'tda/rsync-multiple)

(define-key dired-mode-map (kbd "C-c C-s") 'tda/get-files-size)

(define-key dired-mode-map (kbd "C-c C-q") 'tda/download-to-current-dir))`

Enjoy!