## Querying RESTful webservices into Emacs orgmode tables

In this post, I’ll show you how you can use Emacs and orgmode to query live data from any RESTful webservice, and then use that data in orgmode tables, a really great way to get live table-based calculation and display functionality in your rich orgmode-based documentation.

As an example, we will query live ticker data from the Kraken cryptocurrency exchange, and then use the current trading values of two different cryptocurrencies to calculate a fictitious investor’s position.

There’s a short YouTube video accompanying this post that demonstrates how the whole business works. Read on for more details!

## RESTful webservice: Kraken ticker info

Ticker info can be easily and freely pulled from the Kraken API. For example, if you go to https://api.kraken.com/0/public/Ticker?pair=ETHEUR,XBTEUR using your browser, you should see returned JSON looking something like the following extremely redacted example:

{
"error" : [],
"result" : {
"XXBTZEUR" : {
"c" : [
"2239.99000",
"0.01870867"
],
"a" : [
"2239.99000",
"3",
"3.000"
],
},
"XETHZEUR" : {
"c" : [
"196.40030",
"2.70918471"
],
"a" : [
"196.89291",
"6",
"6.000"
],
}
}
}


I can specify any number of pairs, but for the sake of exposition we’re going to work with Bitcoin in Euro, and Ethereum in Euro.

## Query the ticker webservice using emacs-lisp

Next we’ll write some emacs-lisp code to embed in our orgmode file. In this case, this blog post is actually an orgmode file which I shall later export to wordpress.

The code makes use of TKF’s great request.el package, installable from MELPA. I started with one of the examples on the request.el github, and then used the shiny new let-alist macro to extract the first element of the c key (the last traded price) of the result - PAIR hierarchy.

The eth-eur and btc-eur pairs are stored in the cpb-kraken-etheur and cpb-kraken-xbteur variables respectively.

(require 'request)

(defun timestamp ()
(format-time-string "%Y-%m-%dT%H:%M:%S"))

(request
"https://api.kraken.com/0/public/Ticker"
:params '(("pair" . "ETHEUR,XBTEUR"))
:success (cl-function
(lambda (&key data &allow-other-keys)
;; get out the last successful trade "c"
(let-alist data
(setq cpb-kraken-etheur
(string-to-number (aref .result.XETHZEUR.c 0)))
(setq cpb-kraken-xbteur
(string-to-number (aref .result.XXBTZEUR.c 0)))
(setq cpb-kraken-timestamp (timestamp))
)
(org-table-iterate-buffer-tables)
(message "Retrieved Kraken ticker values at %s. ETHEUR: %f XBTEUR: %f"
cpb-kraken-timestamp cpb-kraken-etheur cpb-kraken-xbteur)
)))


The code is embedded in this orgmode document using org-babel, i.e. in an emacs-lisp source code block:

#+BEGIN_SRC emacs-lisp :results none
(pretty lisp (code (that you see above ok?)))
#+END_SRC


Whenever I press C-c C-c with my cursor anywhere over the code, it will retrieve the current values into emacs-lisp variables, and the recalculate the table shown below.

## Use the ticker data in a spreadsheet-like table

Next, we construct the orgmode table. If you have never done this with Emacs orgmode, you should try it. The UX for quickly making and maintaining text-mode tables is breathtaking. The table as you see it below is the HTML representation (generated by Emacs and orgmode) of the plaintext table as it lives in this org file:

coinunitscurr unit pricecurr valfrac
eth15.231197.0000003000.5070.36
btc2.3351132252.1000005258.90800.64
2017-06-03T14:10:14  8259.415

Everything in column 3 and to the right, and in the last row, is calculated based on the ticker values we have pulled in using the emacs-lisp code above. The table will update whenever I press C-c C-c on the embedded code block, as it ends with (org-table-iterate-buffer-tables), meaning to recalculate all tables in this file, until convergence.

The orgmode formula editor (shortcut C-c '), enables you to edit all cell formulas with live highlighting of the references. The editor interface looks like this:

# Field and Range Formulas
@2$3 = '(format "%f" cpb-kraken-etheur) @2$4 = $2*$3
@2$5 =$4/@II$4;%.2f @3$3 = '(format "%f" cpb-kraken-xbteur)
@3$4 =$2*$3 @3$5 = $4/@II$4;%.2f
@4$1 = '(format "%s" cpb-kraken-timestamp) @4$4 = vsum(@I..@II)


As you can see, I pull in the three variables retrieved and calculated by the emacs-lisp code using snippets of lisp. The other formulas are the standard spreadsheet fair with an orgmode flavour. @2\$3 for example refers to the third column in the second row.

I can make any number of tables in this same file, depending on values from either the lisp code, or other tables. As per usual, I can export the file to PDF, HTML, ODF or even to a wordpress site, as I’m doing right now.

## From org file with local bibtex to LaTeX and PDF

I have (co-)written a few LaTeX documents in my time.

However, as I have been writing my life and lab notes and many of my technical blog posts in Emacs orgmode for the past few years, I wanted to see how one would go about using BiBTeX references in orgmode files (using John Kitchin’s org-ref package) such that they would render correctly in orgmode’s export LaTeX and PDF.

My use case is of course slightly different than the norm: I maintain my master bibliography using the wonderful Zotero software, and I prefer exporting document-specific BiBTeX files from that main database.

This post shows you how to do it.

## Pre-requisites

• Make sure that latexmk is installed on your system. This is one of the easiest ways to make sure that pdflatex and bibtex are executed in the correct sequence, and a sufficient number of times.
• With M-x package-install RET org-ref install the org-ref package. This will enable interactive selection and insertion of bibtex references from your local bib file.
• You should also have the ox-latex Emacs package installed. This is what gives orgmode its LaTeX output powers.

## Emacs setup

In your init.el somewhere, use the following code to ensure that the orgmode LaTeX exporter invokes latexmk in the correct way:

(setq org-latex-pdf-process
'("latexmk -pdflatex='pdflatex -interaction nonstopmode' -pdf -bibtex -f %f"))


## A minimal but complete orgmode example

The usual org-ref workflow is that you use it to manage a single main BiBTeX file, which is configured in your init.el. As mentioned above, I prefer using small document-specific BiBTeX files.

This is what the first three lines sets up using an Emacs file variable. Note that the local bib file is specified without quotes of any kind. Also, because there’s no path specified, it lives together with the orgmode document itself, which I prefer.

# Local Variables:
# org-ref-default-bibliography: local-bibtex-file.bib
# End:


After this, I have two LaTeX-specific lines ensuring that the output PDF looks prettier than default. You can thank me later.

The third LATEX_HEADER line imports the natbib package, which means I have extra cite commands to differentiate between for example textual citing (the first example) and parenthetical citing (the second example).

#+LATEX_CLASS_OPTIONS: [a4paper, 11pt, colorlinks=true, citecolor=., linkcolor=black, urlcolor=black]

#+TITLE: Example orgmode to LaTeX

* Introduction

This is that really difficult intro spection, as was also
demonstrated by cite:huang_identification_2016.

* Conclusions

This is the even more difficult final section. We will now use a
parenthetical citation citep:meyer_four-level_2012.


Finally, very importantly, I have the orgmode orgmode markup to select a suitable bibliography style, and again to specify the bib file. This will get automatically translated to the suitable LaTeX or even HTML versions.

bibliographystyle:abbrvnat
bibliography:bigmove-ml-phase1.bib


## Putting it all together

With something like the example above in your orgmode file, and a suitable bib-file, you can do C-c C-e l o to export to LaTeX, execute the latexmk command we specified above, and view the PDF.

In the screenshot at the start of this post, you can see what this looks like on my setup. I am using PDFTools for viewing the PDF inside of Emacs (this gets updated automatically whenever I rebuild the PDF).

You can also see the ivy-based completion as I’m in the process of interactively inserting a new citation into the document.

The default helm-based completion is more attractive, but it takes over the whole display, which is why I made use of ivy for instruction’s sake.

## Sending emails with math and source code

Org mode is great for authoring rich documents with syntax highlighted source code, LaTeX math and images. It even supports evaluating live snippets of code embedded in the text. It does all of this whilst remaining a plain text format.

Imagine how useful it would be to author programming-related or technical emails using this functionality?

Imagine no more! org-mime, part of the org mode contrib, does this for a number of emacs-based mail clients. However, our preference is for mu4e, which is not part of that list.

Fortunately, it seems mu4e has orgmode support integrated. To get this working, configure the following in your ~/.emacs.d/init.el:

;; configure orgmode support in mu4e
(require 'org-mu4e)
;; when mail is sent, automatically convert org body to HTML
(setq org-mu4e-convert-to-html t)


When composing a new email, switch on the special mu4e / orgmode mode with M-x org~mu4e-mime-switch-headers-or-body (it will automatically switch between compose mode and orgmode depending on whether your cursor is on the headers or in the body).

You should add the following options to your org mode body. The first configures LaTeX (math) to be converted using imagemagick (instead of MathJax or dvipng), and the second removes the table of contents.

#+OPTIONS: tex:imagemagick
#+OPTIONS: toc:0


When you want to send the email, move to the headers, and then do C-c C-c. Because org-mu4e-convert-to-html is set to true, the orgmode body will automatically be converted to HTML before being sent.

Until my pull request is merged in, you will have to make the following change to function org~mu4e-mime-convert-to-html() in org-mu4e.el:

(insert (org~mu4e-mime-multipart
body html (mapconcat 'identity html-images "\n")))


should be:

(insert (org~mu4e-mime-multipart
raw-body html (mapconcat 'identity html-images "\n")))


In other words, body becomes raw-body, else your outgoing mails will have HTML in their text parts, instead of plain text.

In that same function, you might also want to change:

(org-export-preserve-breaks t)


to

(org-export-preserve-breaks nil)


So that the HTML email does not get hard linebreaks.

Following are examples of some of the elements you might include in such a rich text email. They should appear in your rich text capable email client as follows:

Here are the originals:

## A section with some math

We can easily include math using LaTeX:

$C(p,t) = \lbrace x_0(p,t), x_1(p,t), \cdots, x_{N-1}(p,t) \rbrace\tag{1}$

### Some syntax highlighted C++

Syntax highlighted source code, for all of the languages supported by emacs (a very long list):

int main() {
// Recently discovered trick, local functions in C++!
struct Kludge {
static int doSomething(int x) {
return x * 2;
}
};

for (int i=0; i < 10; i++) {
cout << Kludge::doSomething(i) << endl;
}

return 0;
}


## Inline GraphViz DOT evaluation for graphs using Emacs, org-mode and org-babel

With Emacs, org mode and org-babel, it’s possible to evaluate source code samples embedded in your org files and have the output of said evaluation appear inline. This makes for a beautiful literate programming environment. It also enables one to include graphs in one’s documents (org mode, PDF, HTML presentations or blog posts) by using for example GraphViz.

This blog post (obviously authored using Emacs and Org mode) contains short instructions for doing so.

First follow the org-babel documentation and enable source code evaluation for dot by adding the following to your Emacs init.el:

‘((dot . t)))

Then create a new document, and add something like the following dot source code sample to it:

#+BEGIN_SRC dot :file dot_success.png :cmdline -Kdot -Tpng
digraph {
// graph from left to right
rankdir=LR;
splines=true;
node [shape=box];

id [label="Install Graphviz"]
conf [label="Configure org-babel"]
dot [label="DOT in org-mode"]

id -> conf
conf -> dot
dot -> "Profit"
dot -> "Success" [style=dotted]
}
#+END_SRC


Now press C-c C-c to evaluate this code. Emacs will generate the configured output file dot_success.png and then link to it in an automatically created #+RESULTS section right below it.

You can press C-c C-x C-v to toggle display of inline images to see it directly in Emacs. Alternatively, use M-x org-display-inline-images to switch this on. Whenever you change the DOT source code (press C-c ‘ to edit the dot source code in a separate buffer), just press C-c C-c to re-execute the updated source.

In any exported documents (for examlpe this blog post), only the output graph itself will appear, like this:

Exported as a PDF with C-e l o it looks like this.

In your Emacs, it should look like this:

This approach should work for any of the many languages supported by org-babel. Let me know in the comments what you come up with!