Querying RESTful webservices into Emacs orgmode tables

In this post, I’ll show you how you can use Emacs and orgmode to query live data from any RESTful webservice, and then use that data in orgmode tables, a really great way to get live table-based calculation and display functionality in your rich orgmode-based documentation.

As an example, we will query live ticker data from the Kraken cryptocurrency exchange, and then use the current trading values of two different cryptocurrencies to calculate a fictitious investor’s position.

There’s a short YouTube video accompanying this post that demonstrates how the whole business works. Read on for more details!

RESTful webservice: Kraken ticker info

Ticker info can be easily and freely pulled from the Kraken API. For example, if you go to https://api.kraken.com/0/public/Ticker?pair=ETHEUR,XBTEUR using your browser, you should see returned JSON looking something like the following extremely redacted example:

{
   "error" : [],
   "result" : {
      "XXBTZEUR" : {
         "c" : [
            "2239.99000",
            "0.01870867"
         ],
         "a" : [
            "2239.99000",
            "3",
            "3.000"
         ],
      },
      "XETHZEUR" : {
         "c" : [
            "196.40030",
            "2.70918471"
         ],
         "a" : [
            "196.89291",
            "6",
            "6.000"
         ],
      }
   }
}

I can specify any number of pairs, but for the sake of exposition we’re going to work with Bitcoin in Euro, and Ethereum in Euro.

Query the ticker webservice using emacs-lisp

Next we’ll write some emacs-lisp code to embed in our orgmode file. In this case, this blog post is actually an orgmode file which I shall later export to wordpress.

The code makes use of TKF’s great request.el package, installable from MELPA. I started with one of the examples on the request.el github, and then used the shiny new let-alist macro to extract the first element of the c key (the last traded price) of the result - PAIR hierarchy.

The eth-eur and btc-eur pairs are stored in the cpb-kraken-etheur and cpb-kraken-xbteur variables respectively.

(require 'request)

(defun timestamp ()
  (format-time-string "%Y-%m-%dT%H:%M:%S"))

(request
 "https://api.kraken.com/0/public/Ticker"
 :params '(("pair" . "ETHEUR,XBTEUR"))
 :parser 'json-read
 :success (cl-function
           (lambda (&key data &allow-other-keys)
             ;; get out the last successful trade "c"
             (let-alist data
               (setq cpb-kraken-etheur
                     (string-to-number (aref .result.XETHZEUR.c 0)))
               (setq cpb-kraken-xbteur
                     (string-to-number (aref .result.XXBTZEUR.c 0)))
               (setq cpb-kraken-timestamp (timestamp))
               )
             (org-table-iterate-buffer-tables)
             (message "Retrieved Kraken ticker values at %s. ETHEUR: %f XBTEUR: %f"
                      cpb-kraken-timestamp cpb-kraken-etheur cpb-kraken-xbteur)
             )))

The code is embedded in this orgmode document using org-babel, i.e. in an emacs-lisp source code block:

#+BEGIN_SRC emacs-lisp :results none
(pretty lisp (code (that you see above ok?)))
#+END_SRC

Whenever I press C-c C-c with my cursor anywhere over the code, it will retrieve the current values into emacs-lisp variables, and the recalculate the table shown below.

Use the ticker data in a spreadsheet-like table

Next, we construct the orgmode table. If you have never done this with Emacs orgmode, you should try it. The UX for quickly making and maintaining text-mode tables is breathtaking. The table as you see it below is the HTML representation (generated by Emacs and orgmode) of the plaintext table as it lives in this org file:

coinunitscurr unit pricecurr valfrac
eth15.231197.0000003000.5070.36
btc2.3351132252.1000005258.90800.64
2017-06-03T14:10:14  8259.415 

Everything in column 3 and to the right, and in the last row, is calculated based on the ticker values we have pulled in using the emacs-lisp code above. The table will update whenever I press C-c C-c on the embedded code block, as it ends with (org-table-iterate-buffer-tables), meaning to recalculate all tables in this file, until convergence.

The orgmode formula editor (shortcut C-c '), enables you to edit all cell formulas with live highlighting of the references. The editor interface looks like this:

# Field and Range Formulas
@2$3 = '(format "%f" cpb-kraken-etheur)
@2$4 = $2*$3
@2$5 = $4/@II$4;%.2f
@3$3 = '(format "%f" cpb-kraken-xbteur)
@3$4 = $2*$3
@3$5 = $4/@II$4;%.2f
@4$1 = '(format "%s" cpb-kraken-timestamp)
@4$4 = vsum(@I..@II)

As you can see, I pull in the three variables retrieved and calculated by the emacs-lisp code using snippets of lisp. The other formulas are the standard spreadsheet fair with an orgmode flavour. @2$3 for example refers to the third column in the second row.

I can make any number of tables in this same file, depending on values from either the lisp code, or other tables. As per usual, I can export the file to PDF, HTML, ODF or even to a wordpress site, as I’m doing right now.

Fixing crux-open-with on Ubuntu

The crux package for Emacs contains, amongst a list of useful functions (crux does stand for A Collection of Ridiculously Useful eXtensions for Emacs after all), the function crux-open-with. This interactive function opens the file you currently have open, or the file under your cursor in dired, using the system application for that file.

On Ubuntu 16.04, this function unfortunately does not work. It looks like the application tries to start (icon visible in the dash) and then dies.

After some debugging I discovered that on Ubuntu, the system tool xdg-open (which is invoked by crux) starts gvfs-open which on its part does not wait for the app to finish, but returns immediately.

The internal Emacs function start-process used by crux-open-with already runs the child process asynchronously. The early return of gvfs-open results in the sub-process closing, which means your app never gets off the ground.

To fix this, we have to get xdg-open to work in generic (i.e. non-Gnome / GVFS) mode and thus to wait for the app it spawns to return. We do this by setting the environment variable XDG_CURRENT_DESKTOP to X-Generic.

Fortunately, we can do this fairly neatly by wrapping the crux-open-with invocation in a let where we temporarily change the environment.

In my case it looks like this:

(use-package crux
  :config
  (defun cpb-crux-open-with (arg)
    (interactive "P")
    (let ((process-environment (cons "XDG_CURRENT_DESKTOP=X-Generic" process-environment)))
      (crux-open-with arg)))
  (global-set-key (kbd "C-c o") 'cpb-crux-open-with))

If you don’t also use use-package (you really should!) just copy out the part after :config.

This has been tested to work on Ubuntu 16.04 in both Unity and Gnome Shell 3.18.

Updates

  • 2017-04-18: Fixed environment variable + tested in Gnome Shell.

Recursive text search in project without projectile

If you’re not using projectile, but you would like to be able to perform interactive, search-as-you-type, recursive searches through the current project, this is pretty easy to do if you have counsel and find-file-in-project installed.

Add the following Emacs Lisp to your init:

(defun cpb/counsel-ag-in-project ()
  "Use `ffip' and `counsel-ag' for quick project-wide text searching."
  (interactive)
  (let ((project-root (ffip-project-root)))
    ;; if ffip could not find project-root, it will already have
    ;; shown an error message. We only have to check for non-nil.
    (if project-root
        (counsel-ag nil project-root nil
                    (format "Search in PRJ %s" project-root)))))

(global-set-key (kbd "C-c s") 'cpb/counsel-ag-in-project)

This function, which I have bound to ctrl-c s, finds the project root using find-file-in-project's function for this, and then invokes the handy counsel-ag on its result.

From org file with local bibtex to LaTeX and PDF

vxlabs-emacs-org-ref-pdf-example.png
Screenshot of orgmode source, PDF preview on the right, interactive citation selection in the minibuffer. Click for full resolution.

I have (co-)written a few LaTeX documents in my time.

However, as I have been writing my life and lab notes and many of my technical blog posts in Emacs orgmode for the past few years, I wanted to see how one would go about using BiBTeX references in orgmode files (using John Kitchin’s org-ref package) such that they would render correctly in orgmode’s export LaTeX and PDF.

My use case is of course slightly different than the norm: I maintain my master bibliography using the wonderful Zotero software, and I prefer exporting document-specific BiBTeX files from that main database.

This post shows you how to do it.

Pre-requisites

  • Make sure that latexmk is installed on your system. This is one of the easiest ways to make sure that pdflatex and bibtex are executed in the correct sequence, and a sufficient number of times.
  • With M-x package-install RET org-ref install the org-ref package. This will enable interactive selection and insertion of bibtex references from your local bib file.
  • You should also have the ox-latex Emacs package installed. This is what gives orgmode its LaTeX output powers.

Emacs setup

In your init.el somewhere, use the following code to ensure that the orgmode LaTeX exporter invokes latexmk in the correct way:

(setq org-latex-pdf-process
    '("latexmk -pdflatex='pdflatex -interaction nonstopmode' -pdf -bibtex -f %f"))

A minimal but complete orgmode example

The usual org-ref workflow is that you use it to manage a single main BiBTeX file, which is configured in your init.el. As mentioned above, I prefer using small document-specific BiBTeX files.

This is what the first three lines sets up using an Emacs file variable. Note that the local bib file is specified without quotes of any kind. Also, because there’s no path specified, it lives together with the orgmode document itself, which I prefer.

# Local Variables:
# org-ref-default-bibliography: local-bibtex-file.bib
# End:

After this, I have two LaTeX-specific lines ensuring that the output PDF looks prettier than default. You can thank me later.

The third LATEX_HEADER line imports the natbib package, which means I have extra cite commands to differentiate between for example textual citing (the first example) and parenthetical citing (the second example).

#+LATEX_CLASS_OPTIONS: [a4paper, 11pt, colorlinks=true, citecolor=., linkcolor=black, urlcolor=black]
#+LATEX_HEADER: \usepackage{fourier}
#+LATEX_HEADER: \usepackage{natbib}

#+TITLE: Example orgmode to LaTeX

* Introduction

  This is that really difficult intro spection, as was also
  demonstrated by cite:huang_identification_2016.

* Conclusions

  This is the even more difficult final section. We will now use a
  parenthetical citation citep:meyer_four-level_2012.

Finally, very importantly, I have the orgmode orgmode markup to select a suitable bibliography style, and again to specify the bib file. This will get automatically translated to the suitable LaTeX or even HTML versions.

Putting it all together

With something like the example above in your orgmode file, and a suitable bib-file, you can do C-c C-e l o to export to LaTeX, execute the latexmk command we specified above, and view the PDF.

In the screenshot at the start of this post, you can see what this looks like on my setup. I am using PDFTools for viewing the PDF inside of Emacs (this gets updated automatically whenever I rebuild the PDF).

You can also see the ivy-based completion as I’m in the process of interactively inserting a new citation into the document.

The default helm-based completion is more attractive, but it takes over the whole display, which is why I made use of ivy for instruction’s sake.

mu4e 0.9.18: E-Mailing with Emacs now even better.

In this post I talk about three great new features in the latest release of mu4e, an email programme that runs in Emacs. I also show my mu4e configuration as an example to others who would like a similar setup.

mu4e-0.9.18-screenshot.png
mu4e 0.9.18 screenshot showing selected context (bottom right of main) and visual-line-mode (long lines) which is activated by the format=flowed support.

After recently discovering that plaintext format=flowed in Thunderbird works only partially, it was time to check back in on mu4e after my previous happy stint using it.

I was pleasantly surprised to see that mu4e development had been quite active, and that this Emacs mail user agent had acquired a number of highly useful new features.

Three features I find particularly interesting are:

Full and deterministic format=flowed support

As I explained in another blog post, format=flowed is a really clever extension to old-school plain text emails that enables any receiving mail app that supports the standard to reflow emails for better display on smaller or larger displays.

mu4e now has first-class and deterministic support of the format=flowed feature.

I do understand that HTML has won the email format war.

However, HTML formatting gives users entirely too much freedom when writing their emails, resulting in misguided typesetting and other unnecessary visual elements, analogous to and just as irritating as chart junk, that regularly damage sensitive eyes.

This is why I prefer writing in plain text. With reliable format=flowed support, mu4e enables me to focus on the content whilst fixing plain-text email’s only real flaw: Reflowing on mobile devices.

Compose emails in a separate window

This, just like anything else, one could previously implement by oneself in emacs-lisp. However, it’s now a welcome core feature activated via the mu4e-compose-in-new-frame variable.

When you press C to compose a new email, a separate frame (this is what Emacs calls a window) is opened. After sending the email with C-c C-c the frame is automatically closed.

Contexts

In my previous setup, I had made a bunch of emacs-lisp functions to switch between different email identities (from addresses).

This worked, but it was not nearly as convenient and well thought-out as mu4e’s new contexts.

I now have a separate context defined for each email identity. I can switch between them using ; followed by the self-configured shortcut character.

They also activate automatically based on a configurable match function, for example examining the destination address of an email I am replying to. My configuration example below shows this in a little more detail.

My configuration

Because of these new features and a number of other changes, I put together this new configuration from scratch based on mu4e’s excellent documentation and examples.

I am still using `offlineimap` to synchronise my email with the IMAP server (fastmail in my case), but I’m currently sending email using the Emacs smtpmail package.

Here is a slightly shortened and simplified version of my configuration:

;; I installed mu4e from the 0.9.18 tarball
;; this is where make install put the emacs bits
(add-to-list 'load-path "/usr/local/share/emacs/site-lisp/mu4e")

(require 'mu4e)

;; I want my format=flowed thank you very much
;; mu4e sets up visual-line-mode and also fill (M-q) to do the right thing
;; each paragraph is a single long line; at sending, emacs will add the
;; special line continuation characters.
(setq mu4e-compose-format-flowed t)

;; every new email composition gets its own frame! (window)
(setq mu4e-compose-in-new-frame t)

;; give me ISO(ish) format date-time stamps in the header list
(setq mu4e-headers-date-format "%Y-%m-%d %H:%M")

;; show full addresses in view message (instead of just names)
;; toggle per name with M-RET
(setq mu4e-view-show-addresses 't)

;; path to our Maildir directory
(setq mu4e-maildir "~/Maildir")

;; the next are relative to `mu4e-maildir'
;; instead of strings, they can be functions too, see
;; their docstring or the chapter 'Dynamic folders'
(setq mu4e-sent-folder   "/Sent"
      mu4e-drafts-folder "/Drafts"
      mu4e-trash-folder  "/Trash")

;; the maildirs you use frequently; access them with 'j' ('jump')
(setq   mu4e-maildir-shortcuts
    '(("/Archive"     . ?a)
      ("/INBOX"       . ?i)
      ("/Sent"        . ?s)))

;; the list of all of my e-mail addresses
(setq mu4e-user-mail-address-list '("me@home.com"
                                    "me@work.com"
                                    "me@org.org"))

;; the headers to show in the headers list -- a pair of a field
;; and its width, with `nil' meaning 'unlimited'
;; (better only use that for the last field.
;; These are the defaults:
(setq mu4e-headers-fields
    '( (:date          .  25)    ;; alternatively, use :human-date
       (:flags         .   6)
       (:from          .  22)
       (:subject       .  nil))) ;; alternatively, use :thread-subject

;; program to get mail; alternatives are 'fetchmail', 'getmail'
;; isync or your own shellscript. called when 'U' is pressed in
;; main view.

;; If you get your mail without an explicit command,
;; use "true" for the command (this is the default)
;; when I press U in the main view, or C-c C-u elsewhere,
;; this command is called, followed by the mu indexer
(setq mu4e-get-mail-command "offlineimap")

;; not using smtp-async yet
;; some of these variables will get overridden by the contexts
(setq
 send-mail-function 'smtpmail-send-it
 message-send-mail-function 'smtpmail-send-it
 smtpmail-smtp-server "smtp.fastmail.com"
 smtpmail-smtp-service 465
 smtpmail-stream-type 'ssl
 )

;; don't keep message buffers around
(setq message-kill-buffer-on-exit t)

;; here come the contexts
;; I have about 5 of these, chopped down to 2 for demonstration purposes
;; each context can set any number of variables (see :vars)
;; for example below here I'm using two different SMTP servers depending on identity
(setq mu4e-contexts
      `( ,(make-mu4e-context
           :name "c me@home.com"
           :enter-func (lambda () (mu4e-message "Enter me@home.com context"))
           :leave-func (lambda () (mu4e-message "Leave me@home.com context"))
           ;; we match based on the contact-fields of the message (that we are replying to)
           ;; https://www.djcbsoftware.nl/code/mu/mu4e/What-are-contexts.html#What-are-contexts
           :match-func (lambda (msg)
                         (when msg 
                           (mu4e-message-contact-field-matches msg 
                                                               :to "me@home.com")))
           :vars '( ( user-mail-address      . "me@home.com"  )
                    ( user-full-name         . "Charl P. Botha" )
                    ( smtpmail-smtp-server   . "smtp.fastmail.com" )
                    ( mu4e-compose-signature .
                                             (concat
                                              "dr. charl p. botha\n"
                                              "http://charlbotha.com/\n"))))

         ,(make-mu4e-context
           :name "s me@org.org"
           :enter-func (lambda () (mu4e-message "Enter me@org.org context"))
           ;; no leave-func
           ;; we match based on the contact-fields of the message
           :match-func (lambda (msg)
                         (when msg 
                           (mu4e-message-contact-field-matches msg 
                                                               :to "me@org.org")))
           :vars '( ( user-mail-address       . "me@org.org" )
                    ( user-full-name          . "Charl P. Botha" )
                    ( smtpmail-smtp-server    . "smtp.gmail.com" )
                    ( mu4e-compose-signature  .
                                              (concat
                                               "Dr. Charl P. Botha\n"
                                               "Science Officer :: Stone Three Venture Technology\n"
                                               "https://www.stonethree.com\n"))))         

         ))

;; start with the first (default) context; 
(setq mu4e-context-policy 'pick-first)

;; compose with the current context if no context matches;
(setq mu4e-compose-context-policy nil)

;; these are the standard mu4e search bookmarks
;; I've only added the fourth one to pull up flagged emails in my inbox
;; I sometimes use this to shortlist emails I need to get around to ASAP
(setq mu4e-bookmarks
  `( ,(make-mu4e-bookmark
       :name  "Unread messages"
       :query "flag:unread AND NOT flag:trashed"
       :key ?u)
     ,(make-mu4e-bookmark
       :name "Today's messages"
       :query "date:today..now"
       :key ?t)
     ,(make-mu4e-bookmark
       :name "Last 7 days"
       :query "date:7d..now"
       :key ?w)
     ,(make-mu4e-bookmark
       :name "Flagged in INBOX"
       :query "maildir:\"/INBOX\" and flag:flagged"
       :key ?f)))

Conclusion

This is currently running smoothly on two of my Linux workstations as well as my early 2015 retina MacBook Pro.

The general setup procedure is to install offlineimap using pip2 install --user offlineimap (the offlineimap configuration file hasn’t changed much since the previous post), run offlineimap for the first time, then build and install mu4e from its tarball, and finally to run mu index to initialise the index.

After this, your mail, and plain-text nirvana, is a simple M-x mu4e away.