Run code on remote ipython kernels with Emacs and orgmode.

As is briefly documented on the ob-ipython github, one can run code on remote ipython kernels.

In this post, I give a little more detail, and show that this also works wonderfully for remote generation but local embedding of graphics in Emacs Org mode.

As I hinted previously, the jupyter notebook is a great interface for computational coding, but Emacs and Org mode offer far more flexible editing and are more robust as a documentation format.

On to the show. (This whole blog post is a single Org mode file with org-babel source code blocks, the last of which is live.)

Start by starting the ipython kernel on the remote machine:

me@server$ jupyter --runtime-dir
>>> /run/user/1000/jupyter

me@server$ ipython kernel
>>> To connect another client to this kernel, use:
>>>    --existing kernel-11925.json

We have to copy that json connection file to the client machine, and then connect to it with the jupyter console:

me@client$ jupyter --runtime-dir
>>> /Users/cpbotha/Library/Jupyter/runtime

me@client$ cd /Users/cpbotha/Library/Jupyter/runtime
me@client$ scp me@server:/run/user/1000/jupyter/kernel-11925.json .
me@client$ jupyter console --existing kernel-12818.json --ssh meepz97
>>> [ZMQTerminalIPythonApp] To connect another client via this tunnel, use:
>>> [ZMQTerminalIPythonApp] --existing kernel-12818-ssh.json

Note that we copy the json file into our local jupyter runtime directory, which will create the ssh connection file there, and enable us to reference it by name.json only (vs its full path) in any ob-ipython source code blocks.

Now you can open ob-ipython org-babel source blocks which will connect to the remote kernel. They start like this:

#+BEGIN_SRC ipython :session kernel-12818-ssh.json :exports both :results raw drawer

Let’s try it out:

%matplotlib inline
# changed to png only for the blog post
%config InlineBackend.figure_format = 'png'

from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt
from matplotlib import cm

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
X, Y, Z = axes3d.get_test_data(0.05)
cset = ax.contour(X, Y, Z, cmap=cm.coolwarm)
ax.clabel(cset, fontsize=9, inline=1)

plt.show()

428ndb.png

Pretty amazing! The code is executed on the remote machine, and the resultant plot is piped back and displayed embedded in orgmode as SVG!

Getting ob-ipython to show documentation during company completion.

ob-ipython is an Emacs package that enables org-babel to talk to a running ipython kernel. The upshot of this is that you can use org-mode instead of the jupyter notebook for interspersing executable code, results and documentation.

The screenshot from the ob-ipython github shows it in action: ob-ipython-github-screenshot.jpg

Personally, I would like to use this for controlling ipython kernels on remote GPU- and deep learning-capable Linux machines, all via Emacs on my laptop. The juyter notebook is really fantastic, but it’s not Emacs.

You could also use ein for this, but then you would have to give up org-mode.

As I was testing ob-ipython yesterday, I noticed that its company-backend (completion system for Emacs) doc-buffer support was absent. Usually, as you’re exploring possible code completions, you can press <f1> or C-h to show help on the currently highlighted completion candidate.

Fast-forward an hour or two of Emacs Lisp surgery, and I was able to hook up the ob-ipython company-mode backend to ob-ipython’s inspection facility. Now pressing C-h gets you detailed help in a company-documentation buffer!

Here is my github pull request, and here is a screenshot of the company-mode ob-ipython documentation in action:

ob-ipython-company-doc-buffer.png

Hopefully this will be merged soon so it can find its way onto the Melpa package archives.

Here’s a bonus screenshot showing the ob-ipython notes from my org-mode journal where you can see embedded Python code that has been executed via the connected ipython kernel, with the resultant SVG format plot embedded and displayed inline:

ob-ipython-notes-example-nov-2017-3.png

P.S. I am currently disabling elpy-mode when the ob-ipython minor mode is active, until I figure out a better solution to elpy interfering with ob-ipython.

(use-package ob-ipython
  :config
  ;; for now I am disabling elpy only ob-ipython minor mode
  ;; what we should actually do, is just to ensure that
  ;; ob-ipython's company backend comes before elpy's (TODO)
  (add-hook 'ob-ipython-mode-hookp
            (lambda ()
              (elpy-mode 0)
              (company-mode 1)))
  (add-to-list 'company-backends 'company-ob-ipython)
  (add-to-list 'org-latex-minted-langs '(ipython "python")))

Date-sorted interactive recursive search with ivy, counsel and ag

A core part of my note-taking strategy is a growing collection of Emacs Org mode files containing my monthly lab journals, project summaries and various technical documents.

Usually I rely on Projectile’s Helm integration to perform recursive searches with The Silver Searcher (ag) using helm-projectile-ag (trigged by the completely muscle-memorised C-c p s s) through all of the org files in my notes hierarchy. Recursive regular expression search results are shown interactively as you type. Wonderful!

However, recently I found that I really missed the ability to see my search results sorted by last modified date, so that the most recently touched files would show up first. For a chronological notes database, this sorting makes the most sense.

A few frustrating hours later, interspersed with bouts of seriously pondering the exact nature of my relationship with Emacs, I managed to attain the desired result using the Emacs packages ivy and counsel.

Although I’m sticking with Helm for the rest, in this case the ivy API made it much simpler to do what I needed to do. It is understandable that this is not a straight-forward endeavour, as ag is heavily parallelised and furthermore is handled asynchronously by both helm and ivy / counsel. Sorting the results slows matters down, but for this specific case the slowdown is a reasonable price to pay.

Read on for more detail.

Video demo of end result

If you follow the instruction below, you too will be able to achieve the following feats of recall:

Search results are returned as I type, as is usually the case with counsel-ag and helm-ag. However, they are now sorted by file modified date, last modified file at the top. This usually means I find what I need faster.

Show me the code

The ivy-read function (this is ivy’s single API entrypoint) has two required parameters, one of which is the collection. This is a function returning the list of items that the user wants to narrow down, or the list itself.

Because we need to tell ivy to sort the results by associating a sorting function with a specific collection function, we wrap the supplied counsel-ag-function collection function.

We also supply a new sorting function that will take two of the ag-generated search results, each of the form relative_filename:line_number:line_contents, derive the filenames from that, determine the modified timestamps of both the files and then return t or nil depending on which file is newer.

We finally (for this first snippet) associate the sorting function with our wrapped collection function.

(defun cpb/ag-collection (string)
  "Search for pattern STRING using ag.
We only have this as a separate function so we can assoc with sort function."
  ;; this will use counsel--async-command to run asynchronously
  (counsel-ag-function string counsel-ag-base-command "")) 

;; modified from https://github.com/abo-abo/swiper/wiki/Sort-files-by-mtime
;; * directory to counsel--git-grep-dir: that's where AG is working
;; * removed directory logic: ag returns files
;; * parsed out filename from ag results
(defun eh-ivy-sort-file-by-mtime (x y)
  "Determine if AG sort result X is newer than Y."
  (let* ((x (concat counsel--git-grep-dir (car (split-string x ":"))))
         (y (concat counsel--git-grep-dir (car (split-string y ":"))))
         (x-mtime (nth 5 (file-attributes x)))
         (y-mtime (nth 5 (file-attributes y))))
    (time-less-p y-mtime x-mtime)))

;; ivy uses the ivy-sort-functions-alist to look up suitable sort
;; functions for any given collection function
;; we add a cons cell specifying eh-ivy-sort-file-by-mtime as the sort
;; function to go with our collection function
(add-to-list 'ivy-sort-functions-alist
             '(cpb/ag-collection . eh-ivy-sort-file-by-mtime))

In the second and final snippet, we create a new version of the counsel-ag function where we specify our own collection function, and we supply a truthful :sort argument.

(defun cpb/counsel-ag (&optional initial-input initial-directory extra-ag-args ag-prompt)
  "Grep for a string in the current directory using ag.
INITIAL-INPUT can be given as the initial minibuffer input.
INITIAL-DIRECTORY, if non-nil, is used as the root directory for search.
EXTRA-AG-ARGS string, if non-nil, is appended to `counsel-ag-base-command'.
AG-PROMPT, if non-nil, is passed as `ivy-read' prompt argument.

Modified by cpbotha: Sort results last modified file first."
  (interactive)
  (when current-prefix-arg
    (setq initial-directory
          (or initial-directory
              (read-directory-name (concat
                                    (car (split-string counsel-ag-base-command))
                                    " in directory: "))))
    (setq extra-ag-args
          (or extra-ag-args
              (let* ((pos (cl-position ?  counsel-ag-base-command))
                     (command (substring-no-properties counsel-ag-base-command 0 pos))
                     (ag-args (replace-regexp-in-string
                               "%s" "" (substring-no-properties counsel-ag-base-command pos))))
                (read-string (format "(%s) args:" command) ag-args)))))
  (ivy-set-prompt 'cpb/counsel-ag counsel-prompt-function)
  (setq counsel--git-grep-dir (or initial-directory default-directory))
  (ivy-read (or ag-prompt (car (split-string counsel-ag-base-command)))
            #'cpb/ag-collection
            :initial-input initial-input
            :dynamic-collection t
            ;; yes, we want to sort the results
            :sort t
            :keymap counsel-ag-map
            :history 'counsel-git-grep-history
            :action #'counsel-git-grep-action
            :unwind (lambda ()
                      (counsel-delete-process)
                      (swiper--cleanup))
            :caller 'cpb/counsel-ag))

Closing

You can invoke cpb/counsel-ag interactively to search from the current directory, or you can point it at a directory of your choosing by passing in the initial-directory argument.

In my Emacs initialisation, I have bound another wrapper function, bound to a global shortcut key, that invokes cpb/counsel-ag on my notes directory, so I can instantly search through my notes in reverse chronological order.

It looks like this:

;; wrapper function to invoke chrono-sorted ag search on my notes dir
(defun cpb/counsel-ag-notes ()
  "Search my notes file hierarchy using counsel and ivy."
  (interactive)
  (cpb/counsel-ag nil "~/Dropbox/notes/pkb4000/" nil "PKB4000: "))

;; I invoke search by pressing "Ctrl-C 4"
(global-set-key (kbd "C-c 4") 'cpb/counsel-ag-notes)

I believe that my relationship with Emacs is safe for now.

Updates

  • 2017-02-28: Fixed (interactive) invocation in wrapper function.
  • 2017-02-27: Updated cpb/ag-collection for latest counsel version. Added example of wrapper function and global shortcut.