Run code on remote ipython kernels with Emacs and orgmode.

As is briefly documented on the ob-ipython github, one can run code on remote ipython kernels.

In this post, I give a little more detail, and show that this also works wonderfully for remote generation but local embedding of graphics in Emacs Org mode.

As I hinted previously, the jupyter notebook is a great interface for computational coding, but Emacs and Org mode offer far more flexible editing and are more robust as a documentation format.

On to the show. (This whole blog post is a single Org mode file with org-babel source code blocks, the last of which is live.)

Start by starting the ipython kernel on the remote machine:

me@server$ jupyter --runtime-dir
>>> /run/user/1000/jupyter

me@server$ ipython kernel
>>> To connect another client to this kernel, use:
>>>    --existing kernel-11925.json

We have to copy that json connection file to the client machine, and then connect to it with the jupyter console:

me@client$ jupyter --runtime-dir
>>> /Users/cpbotha/Library/Jupyter/runtime

me@client$ cd /Users/cpbotha/Library/Jupyter/runtime
me@client$ scp me@server:/run/user/1000/jupyter/kernel-11925.json .
me@client$ jupyter console --existing kernel-12818.json --ssh meepz97
>>> [ZMQTerminalIPythonApp] To connect another client via this tunnel, use:
>>> [ZMQTerminalIPythonApp] --existing kernel-12818-ssh.json

Note that we copy the json file into our local jupyter runtime directory, which will create the ssh connection file there, and enable us to reference it by name.json only (vs its full path) in any ob-ipython source code blocks.

Now you can open ob-ipython org-babel source blocks which will connect to the remote kernel. They start like this:

#+BEGIN_SRC ipython :session kernel-12818-ssh.json :exports both :results raw drawer

Let’s try it out:

%matplotlib inline
# changed to png only for the blog post
%config InlineBackend.figure_format = 'png'

from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt
from matplotlib import cm

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
X, Y, Z = axes3d.get_test_data(0.05)
cset = ax.contour(X, Y, Z, cmap=cm.coolwarm)
ax.clabel(cset, fontsize=9, inline=1)


Pretty amazing! The code is executed on the remote machine, and the resultant plot is piped back and displayed embedded in orgmode as SVG!

Getting ob-ipython to show documentation during company completion.

ob-ipython is an Emacs package that enables org-babel to talk to a running ipython kernel. The upshot of this is that you can use org-mode instead of the jupyter notebook for interspersing executable code, results and documentation.

The screenshot from the ob-ipython github shows it in action: ob-ipython-github-screenshot.jpg

Personally, I would like to use this for controlling ipython kernels on remote GPU- and deep learning-capable Linux machines, all via Emacs on my laptop. The juyter notebook is really fantastic, but it’s not Emacs.

You could also use ein for this, but then you would have to give up org-mode.

As I was testing ob-ipython yesterday, I noticed that its company-backend (completion system for Emacs) doc-buffer support was absent. Usually, as you’re exploring possible code completions, you can press <f1> or C-h to show help on the currently highlighted completion candidate.

Fast-forward an hour or two of Emacs Lisp surgery, and I was able to hook up the ob-ipython company-mode backend to ob-ipython’s inspection facility. Now pressing C-h gets you detailed help in a company-documentation buffer!

Here is my github pull request, and here is a screenshot of the company-mode ob-ipython documentation in action:


Hopefully this will be merged soon so it can find its way onto the Melpa package archives.

Here’s a bonus screenshot showing the ob-ipython notes from my org-mode journal where you can see embedded Python code that has been executed via the connected ipython kernel, with the resultant SVG format plot embedded and displayed inline:


P.S. I am currently disabling elpy-mode when the ob-ipython minor mode is active, until I figure out a better solution to elpy interfering with ob-ipython.

(use-package ob-ipython
  ;; for now I am disabling elpy only ob-ipython minor mode
  ;; what we should actually do, is just to ensure that
  ;; ob-ipython's company backend comes before elpy's (TODO)
  (add-hook 'ob-ipython-mode-hookp
            (lambda ()
              (elpy-mode 0)
              (company-mode 1)))
  (add-to-list 'company-backends 'company-ob-ipython)
  (add-to-list 'org-latex-minted-langs '(ipython "python")))

Inline GraphViz DOT evaluation for graphs using Emacs, org-mode and org-babel

With Emacs, org mode and org-babel, it’s possible to evaluate source code samples embedded in your org files and have the output of said evaluation appear inline. This makes for a beautiful literate programming environment. It also enables one to include graphs in one’s documents (org mode, PDF, HTML presentations or blog posts) by using for example GraphViz.

This blog post (obviously authored using Emacs and Org mode) contains short instructions for doing so.

First follow the org-babel documentation and enable source code evaluation for dot by adding the following to your Emacs init.el:

‘((dot . t)))

Then create a new document, and add something like the following dot source code sample to it:

#+BEGIN_SRC dot :file dot_success.png :cmdline -Kdot -Tpng
  digraph {
  // graph from left to right
  node [shape=box];

  id [label="Install Graphviz"]
  conf [label="Configure org-babel"]
  dot [label="DOT in org-mode"]

  id -> conf
  conf -> dot
  dot -> "Profit"
  dot -> "Success" [style=dotted]

Now press C-c C-c to evaluate this code. Emacs will generate the configured output file dot_success.png and then link to it in an automatically created #+RESULTS section right below it.

You can press C-c C-x C-v to toggle display of inline images to see it directly in Emacs. Alternatively, use M-x org-display-inline-images to switch this on. Whenever you change the DOT source code (press C-c ‘ to edit the dot source code in a separate buffer), just press C-c C-c to re-execute the updated source.

In any exported documents (for examlpe this blog post), only the output graph itself will appear, like this:

Exported as a PDF with C-e l o it looks like this.

In your Emacs, it should look like this:

This approach should work for any of the many languages supported by org-babel. Let me know in the comments what you come up with!