Run code on remote ipython kernels with Emacs and orgmode.

As is briefly documented on the ob-ipython github, one can run code on remote ipython kernels.

In this post, I give a little more detail, and show that this also works wonderfully for remote generation but local embedding of graphics in Emacs Org mode.

As I hinted previously, the jupyter notebook is a great interface for computational coding, but Emacs and Org mode offer far more flexible editing and are more robust as a documentation format.

On to the show. (This whole blog post is a single Org mode file with org-babel source code blocks, the last of which is live.)

Start by starting the ipython kernel on the remote machine:

me@server$ jupyter --runtime-dir
>>> /run/user/1000/jupyter

me@server$ ipython kernel
>>> To connect another client to this kernel, use:
>>>    --existing kernel-11925.json

We have to copy that json connection file to the client machine, and then connect to it with the jupyter console:

me@client$ jupyter --runtime-dir
>>> /Users/cpbotha/Library/Jupyter/runtime

me@client$ cd /Users/cpbotha/Library/Jupyter/runtime
me@client$ scp me@server:/run/user/1000/jupyter/kernel-11925.json .
me@client$ jupyter console --existing kernel-12818.json --ssh meepz97
>>> [ZMQTerminalIPythonApp] To connect another client via this tunnel, use:
>>> [ZMQTerminalIPythonApp] --existing kernel-12818-ssh.json

Note that we copy the json file into our local jupyter runtime directory, which will create the ssh connection file there, and enable us to reference it by name.json only (vs its full path) in any ob-ipython source code blocks.

Now you can open ob-ipython org-babel source blocks which will connect to the remote kernel. They start like this:

#+BEGIN_SRC ipython :session kernel-12818-ssh.json :exports both :results raw drawer

Let’s try it out:

%matplotlib inline
# changed to png only for the blog post
%config InlineBackend.figure_format = 'png'

from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt
from matplotlib import cm

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
X, Y, Z = axes3d.get_test_data(0.05)
cset = ax.contour(X, Y, Z, cmap=cm.coolwarm)
ax.clabel(cset, fontsize=9, inline=1)

plt.show()

428ndb.png

Pretty amazing! The code is executed on the remote machine, and the resultant plot is piped back and displayed embedded in orgmode as SVG!

Getting ob-ipython to show documentation during company completion.

ob-ipython is an Emacs package that enables org-babel to talk to a running ipython kernel. The upshot of this is that you can use org-mode instead of the jupyter notebook for interspersing executable code, results and documentation.

The screenshot from the ob-ipython github shows it in action: ob-ipython-github-screenshot.jpg

Personally, I would like to use this for controlling ipython kernels on remote GPU- and deep learning-capable Linux machines, all via Emacs on my laptop. The juyter notebook is really fantastic, but it’s not Emacs.

You could also use ein for this, but then you would have to give up org-mode.

As I was testing ob-ipython yesterday, I noticed that its company-backend (completion system for Emacs) doc-buffer support was absent. Usually, as you’re exploring possible code completions, you can press <f1> or C-h to show help on the currently highlighted completion candidate.

Fast-forward an hour or two of Emacs Lisp surgery, and I was able to hook up the ob-ipython company-mode backend to ob-ipython’s inspection facility. Now pressing C-h gets you detailed help in a company-documentation buffer!

Here is my github pull request, and here is a screenshot of the company-mode ob-ipython documentation in action:

ob-ipython-company-doc-buffer.png

Hopefully this will be merged soon so it can find its way onto the Melpa package archives.

Here’s a bonus screenshot showing the ob-ipython notes from my org-mode journal where you can see embedded Python code that has been executed via the connected ipython kernel, with the resultant SVG format plot embedded and displayed inline:

ob-ipython-notes-example-nov-2017-3.png

P.S. I am currently disabling elpy-mode when the ob-ipython minor mode is active, until I figure out a better solution to elpy interfering with ob-ipython.

(use-package ob-ipython
  :config
  ;; for now I am disabling elpy only ob-ipython minor mode
  ;; what we should actually do, is just to ensure that
  ;; ob-ipython's company backend comes before elpy's (TODO)
  (add-hook 'ob-ipython-mode-hookp
            (lambda ()
              (elpy-mode 0)
              (company-mode 1)))
  (add-to-list 'company-backends 'company-ob-ipython)
  (add-to-list 'org-latex-minted-langs '(ipython "python")))

Querying RESTful webservices into Emacs orgmode tables

In this post, I’ll show you how you can use Emacs and orgmode to query live data from any RESTful webservice, and then use that data in orgmode tables, a really great way to get live table-based calculation and display functionality in your rich orgmode-based documentation.

As an example, we will query live ticker data from the Kraken cryptocurrency exchange, and then use the current trading values of two different cryptocurrencies to calculate a fictitious investor’s position.

There’s a short YouTube video accompanying this post that demonstrates how the whole business works. Read on for more details!

RESTful webservice: Kraken ticker info

Ticker info can be easily and freely pulled from the Kraken API. For example, if you go to https://api.kraken.com/0/public/Ticker?pair=ETHEUR,XBTEUR using your browser, you should see returned JSON looking something like the following extremely redacted example:

{
   "error" : [],
   "result" : {
      "XXBTZEUR" : {
         "c" : [
            "2239.99000",
            "0.01870867"
         ],
         "a" : [
            "2239.99000",
            "3",
            "3.000"
         ],
      },
      "XETHZEUR" : {
         "c" : [
            "196.40030",
            "2.70918471"
         ],
         "a" : [
            "196.89291",
            "6",
            "6.000"
         ],
      }
   }
}

I can specify any number of pairs, but for the sake of exposition we’re going to work with Bitcoin in Euro, and Ethereum in Euro.

Query the ticker webservice using emacs-lisp

Next we’ll write some emacs-lisp code to embed in our orgmode file. In this case, this blog post is actually an orgmode file which I shall later export to wordpress.

The code makes use of TKF’s great request.el package, installable from MELPA. I started with one of the examples on the request.el github, and then used the shiny new let-alist macro to extract the first element of the c key (the last traded price) of the result - PAIR hierarchy.

The eth-eur and btc-eur pairs are stored in the cpb-kraken-etheur and cpb-kraken-xbteur variables respectively.

(require 'request)

(defun timestamp ()
  (format-time-string "%Y-%m-%dT%H:%M:%S"))

(request
 "https://api.kraken.com/0/public/Ticker"
 :params '(("pair" . "ETHEUR,XBTEUR"))
 :parser 'json-read
 :success (cl-function
           (lambda (&key data &allow-other-keys)
             ;; get out the last successful trade "c"
             (let-alist data
               (setq cpb-kraken-etheur
                     (string-to-number (aref .result.XETHZEUR.c 0)))
               (setq cpb-kraken-xbteur
                     (string-to-number (aref .result.XXBTZEUR.c 0)))
               (setq cpb-kraken-timestamp (timestamp))
               )
             (org-table-iterate-buffer-tables)
             (message "Retrieved Kraken ticker values at %s. ETHEUR: %f XBTEUR: %f"
                      cpb-kraken-timestamp cpb-kraken-etheur cpb-kraken-xbteur)
             )))

The code is embedded in this orgmode document using org-babel, i.e. in an emacs-lisp source code block:

#+BEGIN_SRC emacs-lisp :results none
(pretty lisp (code (that you see above ok?)))
#+END_SRC

Whenever I press C-c C-c with my cursor anywhere over the code, it will retrieve the current values into emacs-lisp variables, and the recalculate the table shown below.

Use the ticker data in a spreadsheet-like table

Next, we construct the orgmode table. If you have never done this with Emacs orgmode, you should try it. The UX for quickly making and maintaining text-mode tables is breathtaking. The table as you see it below is the HTML representation (generated by Emacs and orgmode) of the plaintext table as it lives in this org file:

coinunitscurr unit pricecurr valfrac
eth15.231197.0000003000.5070.36
btc2.3351132252.1000005258.90800.64
2017-06-03T14:10:14  8259.415 

Everything in column 3 and to the right, and in the last row, is calculated based on the ticker values we have pulled in using the emacs-lisp code above. The table will update whenever I press C-c C-c on the embedded code block, as it ends with (org-table-iterate-buffer-tables), meaning to recalculate all tables in this file, until convergence.

The orgmode formula editor (shortcut C-c '), enables you to edit all cell formulas with live highlighting of the references. The editor interface looks like this:

# Field and Range Formulas
@2$3 = '(format "%f" cpb-kraken-etheur)
@2$4 = $2*$3
@2$5 = $4/@II$4;%.2f
@3$3 = '(format "%f" cpb-kraken-xbteur)
@3$4 = $2*$3
@3$5 = $4/@II$4;%.2f
@4$1 = '(format "%s" cpb-kraken-timestamp)
@4$4 = vsum(@I..@II)

As you can see, I pull in the three variables retrieved and calculated by the emacs-lisp code using snippets of lisp. The other formulas are the standard spreadsheet fair with an orgmode flavour. @2$3 for example refers to the third column in the second row.

I can make any number of tables in this same file, depending on values from either the lisp code, or other tables. As per usual, I can export the file to PDF, HTML, ODF or even to a wordpress site, as I’m doing right now.

Fixing crux-open-with on Ubuntu

The crux package for Emacs contains, amongst a list of useful functions (crux does stand for A Collection of Ridiculously Useful eXtensions for Emacs after all), the function crux-open-with. This interactive function opens the file you currently have open, or the file under your cursor in dired, using the system application for that file.

On Ubuntu 16.04, this function unfortunately does not work. It looks like the application tries to start (icon visible in the dash) and then dies.

After some debugging I discovered that on Ubuntu, the system tool xdg-open (which is invoked by crux) starts gvfs-open which on its part does not wait for the app to finish, but returns immediately.

The internal Emacs function start-process used by crux-open-with already runs the child process asynchronously. The early return of gvfs-open results in the sub-process closing, which means your app never gets off the ground.

To fix this, we have to get xdg-open to work in generic (i.e. non-Gnome / GVFS) mode and thus to wait for the app it spawns to return. We do this by setting the environment variable XDG_CURRENT_DESKTOP to X-Generic.

Fortunately, we can do this fairly neatly by wrapping the crux-open-with invocation in a let where we temporarily change the environment.

In my case it looks like this:

(use-package crux
  :config
  (defun cpb-crux-open-with (arg)
    (interactive "P")
    (let ((process-environment (cons "XDG_CURRENT_DESKTOP=X-Generic" process-environment)))
      (crux-open-with arg)))
  (global-set-key (kbd "C-c o") 'cpb-crux-open-with))

If you don’t also use use-package (you really should!) just copy out the part after :config.

This has been tested to work on Ubuntu 16.04 in both Unity and Gnome Shell 3.18.

Updates

  • 2017-04-18: Fixed environment variable + tested in Gnome Shell.

Recursive text search in project without projectile

If you’re not using projectile, but you would like to be able to perform interactive, search-as-you-type, recursive searches through the current project, this is pretty easy to do if you have counsel and find-file-in-project installed.

Add the following Emacs Lisp to your init:

(defun cpb/counsel-ag-in-project ()
  "Use `ffip' and `counsel-ag' for quick project-wide text searching."
  (interactive)
  (let ((project-root (ffip-project-root)))
    ;; if ffip could not find project-root, it will already have
    ;; shown an error message. We only have to check for non-nil.
    (if project-root
        (counsel-ag nil project-root nil
                    (format "Search in PRJ %s" project-root)))))

(global-set-key (kbd "C-c s") 'cpb/counsel-ag-in-project)

This function, which I have bound to ctrl-c s, finds the project root using find-file-in-project's function for this, and then invokes the handy counsel-ag on its result.