Run code on remote ipython kernels with Emacs and orgmode.

As is briefly documented on the ob-ipython github, one can run code on remote ipython kernels.

In this post, I give a little more detail, and show that this also works wonderfully for remote generation but local embedding of graphics in Emacs Org mode.

As I hinted previously, the jupyter notebook is a great interface for computational coding, but Emacs and Org mode offer far more flexible editing and are more robust as a documentation format.

On to the show. (This whole blog post is a single Org mode file with org-babel source code blocks, the last of which is live.)

Start by starting the ipython kernel on the remote machine:

me@server$ jupyter --runtime-dir
>>> /run/user/1000/jupyter

me@server$ ipython kernel
>>> To connect another client to this kernel, use:
>>>    --existing kernel-11925.json

We have to copy that json connection file to the client machine, and then connect to it with the jupyter console:

me@client$ jupyter --runtime-dir
>>> /Users/cpbotha/Library/Jupyter/runtime

me@client$ cd /Users/cpbotha/Library/Jupyter/runtime
me@client$ scp me@server:/run/user/1000/jupyter/kernel-11925.json .
me@client$ jupyter console --existing kernel-12818.json --ssh meepz97
>>> [ZMQTerminalIPythonApp] To connect another client via this tunnel, use:
>>> [ZMQTerminalIPythonApp] --existing kernel-12818-ssh.json

Note that we copy the json file into our local jupyter runtime directory, which will create the ssh connection file there, and enable us to reference it by name.json only (vs its full path) in any ob-ipython source code blocks.

Now you can open ob-ipython org-babel source blocks which will connect to the remote kernel. They start like this:

#+BEGIN_SRC ipython :session kernel-12818-ssh.json :exports both :results raw drawer

Let’s try it out:

%matplotlib inline
# changed to png only for the blog post
%config InlineBackend.figure_format = 'png'

from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt
from matplotlib import cm

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
X, Y, Z = axes3d.get_test_data(0.05)
cset = ax.contour(X, Y, Z, cmap=cm.coolwarm)
ax.clabel(cset, fontsize=9, inline=1)

plt.show()

428ndb.png

Pretty amazing! The code is executed on the remote machine, and the resultant plot is piped back and displayed embedded in orgmode as SVG!

7 thoughts on “Run code on remote ipython kernels with Emacs and orgmode.”

  1. Thank you for this quick and informative blog post!

    This is exactly what I am looking for but for R. Have you managed to make it work with R or send me in the right direction?

    Thank in advance!

      1. FYI, it’s much simpler to get the remote kernel working by using ‘remote_ikernel’ (https://github.com/cfriedline/remote_ikernel). You can simply define the remote kernel using remote_ikernel, and specify it in ob-ipython using :kernel keyword.
        This approach also works for R and other language kernels. I found it greatly simplified my workflow.

        One thing that hold me from fully switching to ob-ipython is actually the auto-completion is not usable when using a remote kernel: sometimes it need 10s to show the completion, and it’s not async. EIN and jupyter notebook in browser have much better completion experience when using with remote kernel.

        1. Thank you very much for the super useful tip! I will try this out soon.

          With regard to the auto-completion: Do you think it would help to cache the completion responses on the client?

          1. I guess that’s what jupyter does internally: seems when we import a particular library, they use ‘jedi’ to scan and cache all completion items.
            I’m not sure how to do that in Emacs though… Maybe those python package like Elpy can provide some clues.

  2. Thanks for inspiring me to try ob-ipython!

    I also recommend a shortcut for typing snippets:

    (add-to-list ‘org-structure-template-alist
    ‘(“p” “#+BEGIN_SRC ipython :session :exports both :async t :results raw drawer\n?\n#+END_COMMENT”))

    Then you can type `<p`, press TAB, and get a new "code cell".

    1. Great to see you Stéfan and thanks for the tip!

      I’m hoping that between us all we can do more PRs to further improve ob-ipython. I’m hearing (here and on reddit) that another pain-point is the speed of completion when connected to a remote kernel. *hint hint*

Leave a Reply

Your email address will not be published. Required fields are marked *