Configuring Emacs, lsp-mode and the python language server.

The language server protocol was proposed by Microsoft as a way for different editors and development environments to share language analysis backends

This post describes how to configure Emacs, lsp-mode and the palantir python-language-server for improved code intelligence when working on Python projects. (I’m planning a companion post for Emacs, C++ and the cquery language server.)

Goal

Before starting, it is motivating to see what we are working towards.

With a correctly configured setup, Emacs will sport, amongst others, improved completion with interactive documentation, imenu navigation, documentation on hover, and really snazzy find definitions (M-.) and find references.

See the following screenshots for some examples:

Screen Shot 2018-06-08 at 13.01.32_2018-06-08_14-29-25.png

lsp-python-imenu_2018-06-08_14-34-27.png

lsp-python-docs-on-hover_2018-06-08_14-34-43.png

Pre-requisites on the Python side

Install the python-language-server into the virtual environment, or user environment, that you’re planning to use.

These days, I tend to use pipenv:

cd my_project
pipenv install python-language-server[all]

The [all] means that it installs all optional providers, e.g. yapf formatting.

Pre-requisites on the Emacs side

In Emacs, install the required and some optional packages using for example M-x package-install:

  • lsp-mode – the main language server protocol package
  • lsp-ui – UI-related LSP extras, such as the sideline info, docs, flycheck, etc.
  • company-lsp – company-backend for LSP-based code completion.
  • projectile or find-file-in-project – we use a single function from here to determine the root directory of a project.

Emacs configuration

Add the following to your Emacs init.el, and don’t forget to read the comments.

If you’re not yet using use-package now would be a good time to upgrade.

(use-package lsp-mode
  :ensure t
  :config

  ;; make sure we have lsp-imenu everywhere we have LSP
  (require 'lsp-imenu)
  (add-hook 'lsp-after-open-hook 'lsp-enable-imenu)  
  ;; get lsp-python-enable defined
  ;; NB: use either projectile-project-root or ffip-get-project-root-directory
  ;;     or any other function that can be used to find the root directory of a project
  (lsp-define-stdio-client lsp-python "python"
                           #'projectile-project-root
                           '("pyls"))

  ;; make sure this is activated when python-mode is activated
  ;; lsp-python-enable is created by macro above 
  (add-hook 'python-mode-hook
            (lambda ()
              (lsp-python-enable)))

  ;; lsp extras
  (use-package lsp-ui
    :ensure t
    :config
    (setq lsp-ui-sideline-ignore-duplicate t)
    (add-hook 'lsp-mode-hook 'lsp-ui-mode))

  (use-package company-lsp
    :config
    (push 'company-lsp company-backends))

  ;; NB: only required if you prefer flake8 instead of the default
  ;; send pyls config via lsp-after-initialize-hook -- harmless for
  ;; other servers due to pyls key, but would prefer only sending this
  ;; when pyls gets initialised (:initialize function in
  ;; lsp-define-stdio-client is invoked too early (before server
  ;; start)) -- cpbotha
  (defun lsp-set-cfg ()
    (let ((lsp-cfg `(:pyls (:configurationSources ("flake8")))))
      ;; TODO: check lsp--cur-workspace here to decide per server / project
      (lsp--set-configuration lsp-cfg)))

  (add-hook 'lsp-after-initialize-hook 'lsp-set-cfg))

Putting it all together

Importantly, use pyvenv or something similar to switch to the relevant virtualenv before opening the first Python file.

When you open the file, the pyls should be automatically started up, and you can edit away with LSP-powered code intelligence.

This often gives better and more detailed results than elpy, probably because pyls uses a mix of static and dynamic (introspection-based) analysis.

Furthermore, the handling of LSP servers in Emacs can be unified, giving the same consistent level of support across a whole range of programming languages.

Use the Google Cloud Speech API to transcribe a podcast

As I was listening to the December 21 episode of the CPPCast, together with TWiML&AI my two most favourite podcasts, I couldn’t help but be a little bewildered by the number of times the guest used the word “like” during their interview.

Most of these were examples of speech disfluency, or filler words, but I have to admit that they detracted somewhat from an otherwise interesting discourse.

During another CPPCast episode which I recently listened to, the hosts coincidentally discussed the idea of making available transcriptions of the casts.

These two occurrences, namely the abundance of the “like” disfluency and the mention of transcription, connected in the back of my mind, and produced the idea of finding out how one could go about to use a publically available speech API to transcribe the podcast, and count the number of utterances of the word “like”.

Due to the golden age of information we find ourselves in, this was not that hard at all.

Selecting the API

After a short investigation of Microsoft’s offerings seemed to indicate that I would not be able to transcribe just under an hour of speech, I turned to Google.

The Google Cloud Speech API has specific support for the asynchronous transcription of speech recordings of up to 3 hours.

Setting up the project and service account

Make sure that you can access the Google Cloud Dashboard with your google account. I created a new project for this experiment called cppcast-speech-to-text.

Within that project, select APIs & Services dashboard from the menu on the left, and then enable the Speech API for that project by selecting the Enable APIs and Services link at the top.

Next, go to IAM & Admin and Service Accounts via the main menu, and create a service account for this project.

Remember to select the download JSON private key checkbox.

Transcode and upload the audio

For the Speech API, you will have to transcode the MP3 to FLAC, and you will have to upload the file to a Google Cloud Storage bucket.

I transcoded the MP3 to a 16kHz mono FLAC (preferred by the API) as follows:

ffmpeg -i cppcast-131.mp3 -y -vn -acodec flac -ar 16000 -ac 1 cppcast-131.flac

This turned my 39MB MP3 into a 61MB FLAC file.

Create a storage bucket via the cloud dashboard main menu’s StorageBrowser menus, and then upload the FLAC file to that bucket via the web interface.

Note down the BUCKETNAME and the FILENAME, you’ll need these later when starting the transcription job.

Transcribe!

I used the Asynchronous Speech Recognition API, as this is the only API supporting speech segments this long.

First startup the Google Cloud Shell by clicking on the boxed >_ icon at the top left. In this super convenient Debian Linux shell, gcloud is already installed, which is why I chose to use it.

Upload your service account JSON private key, and activate it by doing the following:

export GOOGLE_APPLICATION_CREDENTIALS=~/your-service-account-key.json

Using one of the installed editors, or just uploading, create a file called async-request.json in your home:

{
  "config": {
      "encoding": "FLAC",
      "sampleRateHertz": 16000,
      "language_code": "en-US"
  },
  "audio":{
    "uri":"gs://BUCKETNAME/FILENAME"
  }
}

You are now ready to make the request using curl, and the async-request.json file you created:

curl -X POST \
     -H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
     -H "Content-Type: application/json; charset=utf-8" \
     --data @async-request.json "https://speech.googleapis.com/v1/speech:longrunningrecognize"

You should see a response looking something like this:

{
  "name": "LONG_JOB_NUMBER"
}

Soon after this, you can start seeing how your job is progressing:

curl -H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
     -H "Content-Type: application/json; charset=utf-8" \
     "https://speech.googleapis.com/v1/operations/LONG_JOB_NUMBER"

The response will look like this while your request is being processed:

{
  "name": "LONG_JOB_NUMBER",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.speech.v1.LongRunningRecognizeMetadata",
    "progressPercent": 29,
    "startTime": "2018-02-14T20:17:05.885941Z",
    "lastUpdateTime": "2018-02-14T20:22:26.830868Z"
  }
}

In my case, the 56 minute podcast was transcribed in just under 17 minutes.

When the job is done, the response to the above curl request will contain the transcribed text. It looks something like this:

{
  "name": "LONG_JOB_NUMBER",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.speech.v1.LongRunningRecognizeMetadata",
    "progressPercent": 100,
    "startTime": "2018-02-14T20:17:05.885941Z",
    "lastUpdateTime": "2018-02-14T20:35:16.404144Z"
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.cloud.speech.v1.LongRunningRecognizeResponse",
    "results": [
      {
        "alternatives": [
          {
            "transcript": "I said 130 want to see PP cast with guest Nicole mazzuca recorded September 14th 2017",
            "confidence": 0.8874592
          }
        ]
      },

// and so on for the whole recording

You can download the full transcription here.

Too many likes?

I wrote the following Python to tally up the total number of words, and the total number of “like” utterances.

import json

with open('/Users/cpbotha/Downloads/cppcast-131-text.json') as f:
    # results: a list of dicts, each with 'alternatives', which is a list of transcripts
    res = json.load(f)['response']['results']

num_like = 0
num_words = 0
for r in res:
    alts = r['alternatives']
    # ensure that we only have one alternative per result
    assert len(alts) == 1
    # break into lowercase words
    t = alts[0]['transcript'].strip().lower().split()
    # tally up total number of words
    num_words += len(t)
    # count the like utterances
    num_like += sum(1 for w in t if w == 'like')

In this 56 minute long episode of CPPCast, 7411 words were detected, 214 of which were the word “like”.

This is not quite as many as I imagined, but still comes down to 3.82 likes per minute, which is enough to be quite noticeable.

Conclusions

  • We should try to use “like” and other speech disfluencies far less often. Inserting a small pause makes more sense: The speaker and the listeners get a little break to process the ongoing speech, and the speech comes across as more measured.
  • All in all, it took me about 2 hours from idea to transcribed text. I find it wonderful that machine learning for speech-to-text has become so democratised.
  • After my transcription job was complete, I saw that it was possible to supply phrase hints to the API. I could have uploaded a list of words we expect to occur during this podcast, such as “CPPCast” and “C++”, and this would have been used by the API to further improve its transcription.

Creating a Django migration for a GiST / GIN index with a special index operator.

In order to add a GiST index on a Postgres database that could be used to accelerate trigram matches using the pg_trgm module and the special gist_trgm_ops operator, I had to code up a special Django Index

Django will hopefully soon support custom index operators, but if you need the functionality right now, this example will do the trick.

The special GiST index class looks like this:

from django.contrib.postgres.indexes import GistIndex

class GistIndexTrgrmOps(GistIndex):
    def create_sql(self, model, schema_editor):
        # - this Statement is instantiated by the _create_index_sql()
        #   method of django.db.backends.base.schema.BaseDatabaseSchemaEditor.
        #   using sql_create_index template from
        #   django.db.backends.postgresql.schema.DatabaseSchemaEditor
        # - the template has original value:
        #   "CREATE INDEX %(name)s ON %(table)s%(using)s (%(columns)s)%(extra)s"
        statement = super().create_sql(model, schema_editor)
        # - however, we want to use a GIST index to accelerate trigram
        #   matching, so we want to add the gist_trgm_ops index operator
        #   class
        # - so we replace the template with:
        #   "CREATE INDEX %(name)s ON %(table)s%(using)s (%(columns)s gist_trgrm_ops)%(extra)s"
        statement.template =\
            "CREATE INDEX %(name)s ON %(table)s%(using)s (%(columns)s gist_trgm_ops)%(extra)s"

        return statement

Which you can then use in your model class like this:

class YourModel(models.Model):
    some_field = models.TextField(...)

    class Meta:
        indexes = [
            GistIndexTrgrmOps(fields=['some_field'])
        ]

The migration will then generate SQL (use manage.py sqlmigrate to inspect the SQL) which looks like this:

CREATE INDEX "app_somefield_139a28_gist" ON "app_yourmodel" USING gist ("some_field" gist_trgrm_ops);

You can easily modify this pattern for other (Postgres) indices and their operators.

How to debug PyInstaller DLL / PYD load failed issues on Windows

TL;DR

When debugging DLL load errors on Windows, use lucasg’s open source and more modern rewrite of the old Dependency Walker software. Very importantly, keep on drilling down through indirect dependencies until you find the missing DLLs.

The Problem

Recently I had to package up a wxPython and VTK-based app for standalone deployment on Windows. Because of great experience with PyInstaller, I opted to use this tool.

With the first try with the freshly built package on the deployment machine, it refused to start up due to an ImportError: DLL load failed: The specified module could not be found., and specifically with the vtk.vtkCommonCorePython.pyd Python extension DLL.

What was frustrating, is that the relevant file was definitely present and in the right place, namely the same folder as the exe file.

A test app that imports only wx (4.0.0rc) and vtk (8.0.1) generated the following traceback:

[3096] LOADER: Running pyiboot01_bootstrap.py
[3096] LOADER: Running t1.py
Traceback (most recent call last):
  File "site-packages\vtk\vtkCommonCore.py", line 5, in <module>
  File "C:\Users\cpbotha\Miniconda3\envs\env1\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 718, in load_module
ImportError: DLL load failed: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "t1.py", line 9, in <module>
  File "C:\Users\cpbotha\Miniconda3\envs\env1\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 631, in exec_module
  File "site-packages\vtk\__init__.py", line 41, in <module>
  File "C:\Users\cpbotha\Miniconda3\envs\env1\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 631, in exec_module
  File "site-packages\vtk\vtkCommonCore.py", line 9, in <module>
  File "C:\Users\cpbotha\Miniconda3\envs\env1\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 718, in load_module
ImportError: DLL load failed: The specified module could not be found.
[3096] Failed to execute script t1
[3096] LOADER: OK.
[3096] LOADER: Cleaning up Python interpreter.

Digging in using pdb and ctypes.WinDLL

By adding import pdb; pdb.set_trace() at a strategic point, I could use the Python debugger to try and investigate why it was not able to load the DLL which was clearly in the right place.

First I confirmed that it could load other namespaced PYDs in the same directory:

(Pdb) import ctypes
(Pdb) ctypes.WinDLL("wx.siplib.pyd")
<PyInstallerWinDLL 'C:\Users\cpbotha\Downloads\t1\wx.siplib.pyd', handle 7fff66320000 at 0x151a6781048>

However, using the same invocation on the offending PYD would still raise an error. By the way, I had to disable PyInstaller’s silly exceptions, because they masked the underlying OSError exception.

Great deal of good that did me, because Windows 10 gets me no additional information other than reporting “error 126” on the top-level DLL. Why can’t such a mature system not offer a little more guidance?

(Pdb) ctypes.WinDLL("vtk.vtkCommonCorePython.pyd")
>>> OSError: [WinError 126] The specified module could not be found
(Pdb) ctypes.WinDLL("C:\\Users\\cpbotha\\Downloads\\t1\\vtk.vtkCommonCorePython.pyd")
>>> OSError: [WinError 126] The specified module could not be found
(Pdb) import os
(Pdb) os.path.exists("C:\\Users\\cpbotha\\Downloads\\t1\\vtk.vtkCommonCorePython.pyd")
True
(Pdb) ctypes.WinDLL("vtkCommonCorePython.pyd")
>>> OSError: [WinError 126] The specified module could not be found
(Pdb) os.path.exists("vtkCommonCorePython.pyd")
True

I did spend more time than I should have in pdb tracing through the whole complicated PyInstaller and Python import code innards. The fact that some PYDs loaded and some did not should have more quickly pushed me in the direction of nested dependencies.

Seeing the light with lucasg’s Dependencies

Although I had at an earlier stage checked DLL loading first with Dependency Walker (this gets very confused by the new Windows api-ms-win-* DLLs) and later with lucasg’s improved utility, I did not drill down far enough into the dependency tree.

It’s important to keep on drilling down until you see missing DLLs, the application won’t automatically traverse the tree.

Anyways, drilling down from the offending vtk.vtkCommonCorePython.pyd soon enough led me to the culprit: The Intel Threading Building Blocks (TBB) DLL conda package I was using was accidentally built in debug mode, and the debug runtimes it relied one were obviously not being deployed:

screenshot_2017-12-06_13-52-48.png

After switching to a TBB conda package from a different channel, the app was finally able to start up and run on the deployment machine.

Fixed position of any Scatter child widget in Kivy

Just in case you were wondering how one could go about having any child widget of a Kivy Scatter widget that would stick in a single window-relative position whilst the scatter itself was being translated and rotated, I thought I’d post this solution.

With a scatter, one can very easily implement a canvas-like object that can be rotated, translated and zoomed, whilst all of the widgets it contains are transformed along with it. This is usually great, unless you’d like, for some or other reason, to display a widget at a fixed position, for example any kind of overlay, such as a label.

The short answer is that this can be done by using a MatrixInstruction that is executed before that widget is drawn (in the canvas.before context) and that effectively reverses the parent Scatter’s transformation.

The slightly more involved answer is the minimal working example below. First we have the Python part of the ScatterOverlayApp. This only defines the MyScatter class which we’ll flesh out in the KV file, and the main app class which instantiates the top-level widget, in our case MyScatter:

from kivy.app import App
from kivy.uix.scatter import Scatter

class MyScatter(Scatter):
    pass

class ScatterOverlayApp(App):
    def build(self):
        return MyScatter()

if __name__ == '__main__':
    ScatterOverlayApp().run()

In the KV Language part of this example, the <MyScatter> rule defines two buttons, and then the Label which we shall configure to be the fixed position overlay. The Label stores the current matrix, and then applies the inverse transformation matrix of its parent, in other words that of the MyScatter.

This will result in the widget drawing in the space of MyScatter‘s parent, in our case the window. So when we specify the label to be in the middle of the root widget, it will actually be drawn right in the middle of the window. Even when we rotate and scale MyScatter (see screenshot below), the label will remain exactly where we put it.

<MyScatter>:
    Button:
        text: "Click me"
        pos: 100, 100

    Button:
        text: "Don't click me"
        pos: 150, 200

    Label:
        pos: root.width / 2.0, root.height / 2.0
        text: "I am an overlay label!"
        font_size: 32

        canvas.before:
            # store current matrix
            PushMatrix
            MatrixInstruction:
                # reverse MyScatter's transformation matrix
                matrix: self.parent.transform_inv

        canvas.after:
            # restore matrix so that other widgets draw normally
            PopMatrix

Here’s a screenshot showing the situation after the containing MyScatter widget has been rotated, translated and zoomed. Both the buttons it contains have also been transformed, but the label is exactly where we placed it initially.

scatteroverlay.jpg