Importing all of your orgmode notes into Apple Notes for mobile access.

Over the years, I’ve built up quite a collection of notes as Org mode text files. So far, it has proven to be the most expressive and the most robust note-taking modality out of a long list of candidates that I’ve tried.

Note-taking using Org mode has one big drawback however: Mobile accessibility.

In other words, consulting one’s org mode notes database from a mobile device is painful. This should not be the case; notes should be always and instantly available, even on mobile.

In this blog post, I show you how to import your complete org mode notes database into Apple Notes, including typesetting and attachments, using the org mode publishing functionality.

To be clear: Org mode on the desktop remains my primary note-taking system. The goal of importing all of my notes into Apple Notes is only to have my personal knowledge base accessible from my mobile devices.

End result

After configuring and running org-publish, and then importing the whole directory of exported HTML files and attachments into Apple Notes on macOS, your notes will look like one of the two examples below: First macOS, then iOS on the phone.

Note the Emacs-supplied syntax highlighting, and the inline image.

If you import these to your icloud account (the default) the notes will be available on all of your iOS mobile devices.

These imported notes are fully indexed, and hence searchable from all of your devices.

screenshot_2018-10-29_13-21-06.png
An example note in Apple Notes on macOS.
screenshot_2018-10-29_13-44-23.png
The same note as above, but now in Apple Notes on iOS.

org-publish configuration

In order to make this happen, we make use of the org-publish functionality. We also configure one or two Apple Notes-specific changes to improve rendering.

Add the configuration below to your init.el.

There are two publish targets: One for the org files (called pkb4000 below), and one for all of the attachments (called pkb4000-static below).

As an aside, pkb4000 is short for Personal Knowledge Base 4000. I chose the name as a joke, as this synced directory of org mode files felt like just the Nth in a long series of knowledge base iterations. Little did I know how well this one would stick.

Remember to change both the :base-directory properties to the top-level directory of your notes database. :publishing-directory is anywhere convenient where you would like to store the published HTML files and attachments.

;; https://orgmode.org/manual/HTML-preamble-and-postamble.html
;; disable author + date + validate link at end of HTML exports
(setq org-html-postamble nil)

(defun org-html-publish-to-html-for-apple-notes (plist filename pub-dir)
  "Convert blank lines to <br /> and remove <h1> titles."
  ;; temporarily configure export to convert math to images because
  ;; apple notes obviously can't use mathjax (the default)
  (let* ((org-html-with-latex 'imagemagick)
         (outfile
          (org-publish-org-to 'html filename
                              (concat "." (or (plist-get plist :html-extension)
                                              org-html-extension
                                              "html"))
                              plist pub-dir)))
    ;; 1. apple notes handles <p> paras badly, so we have to replace all blank
    ;;    lines (which the orgmode export accurately leaves for us) with
    ;;    <br /> tags to get apple notes to actually render blank lines between
    ;;    paragraphs
    ;; 2. remove large h1 with title, as apple notes already adds <title> as
    ;; the note title
    (shell-command
     (format "sed -i \"\" -e 's/^$/<br \\/>/' -e 's/<h1 class=\"title\">.*<\\/h1>$//' %s"
             outfile))
    outfile))

(setq org-publish-project-alist
      '(("pkb4000"
         :base-directory "~/Dropbox/notes/pkb4000/"
         :publishing-directory "~/Downloads/pkb4000"
         :recursive t
         :publishing-function org-html-publish-to-html-for-apple-notes
         :section-numbers nil
         :with-toc nil)
        ("pkb4000-static"
         :base-directory "~/Dropbox/notes/pkb4000/"
         :base-extension "css\\|js\\|png\\|jpg\\|gif\\|pdf\\|mp3\\|ogg\\|swf"
         :publishing-directory "~/Downloads/pkb4000/"
         :recursive t
         :publishing-function org-publish-attachment
         )))

Publish your database and import to Apple Notes

Start the process by M-x org-publish, at which point Emacs will ask you to select a target. You have to org-publish both of the targets.

If you have a large database, Emacs might report errors in your org files. Fix these, and restart the process. org-publish caches its output, so re-runs should not take that long.

After a successful publish, import the whole :publishing-directory into Apple Notes on macOS by selecting File | Import from the menu.

Apple Notes will create a new Imported Notes folder containing your whole notes hierarchy.

This process can be easily repeated when you want to refresh the database on your Apple Notes, but unfortunately Notes will create a new Imported Notes N folder.

Alternatively, you can re-publish, and then import just changed HTML files one by one, after which you’ll have to move them back into the correct Apple Notes folder.

Conclusion

This solves the problem of being able to search rapidly and consult your whole org mode notes database using your iOS mobile device.

However, it does not yet solve the problem of importing Apple Notes you create on the mobile device back into Orgmode. This is something one could consider trying to solve using AppleScript.

Whatever the case may be, this is still a nice improvement over my previous workflow!

Bonus round: Convert orgmode buffer to Apple Note using AppleScript

Before I tried org-publish, I worked on some emacs-lisp and AppleScript to convert the current org mode buffer to HTML, and then to inject that into Apple Notes using Apple Script.

I am posting this here in case it might be useful to someone. However it is NOT required for the org-publish workflow described above.

This assumes that you have an orgmode folder.

Although far more humble than org-publish, this code will only create a new note if it does not exist yet. If the note already exists, it will simply update its contents to the current org file.

;; https://www.emacswiki.org/emacs/string-utils.el
(defun string-utils-escape-double-quotes (str-val)
  "Return STR-VAL with every double-quote escaped with backslash."
  (save-match-data
    (replace-regexp-in-string "\"" "\\\\\"" str-val)))

(defun string-utils-escape-backslash (str-val)
  "Return STR-VAL with every backslash escaped with an additional backslash."
  (save-match-data
    (replace-regexp-in-string "\\\\" "\\\\\\\\" str-val)))

(setq as-tmpl "set TITLE to \"%s\"
set NBODY to \"%s\"
tell application \"Notes\"
        tell folder \"orgmode\"
                if not (note named TITLE exists) then
                        make new note with properties {name:TITLE}
                end if
                set body of note TITLE to NBODY
        end tell
end tell")

(defun oan-export ()
  (interactive)
  (let ((title (file-name-base (buffer-file-name))))
    (with-current-buffer (org-export-to-buffer 'html "*orgmode-to-apple-notes*")
      (let ((body (string-utils-escape-double-quotes
                   (string-utils-escape-backslash (buffer-string)))))
        ;; install title + body into template above and send to notes
        (do-applescript (format as-tmpl title body))
        ;; get rid of temp orgmode-to-apple-notes buffer
        (kill-buffer))
      )))

p

Querying RESTful webservices into Emacs orgmode tables

In this post, I’ll show you how you can use Emacs and orgmode to query live data from any RESTful webservice, and then use that data in orgmode tables, a really great way to get live table-based calculation and display functionality in your rich orgmode-based documentation.

As an example, we will query live ticker data from the Kraken cryptocurrency exchange, and then use the current trading values of two different cryptocurrencies to calculate a fictitious investor’s position.

There’s a short YouTube video accompanying this post that demonstrates how the whole business works. Read on for more details!

RESTful webservice: Kraken ticker info

Ticker info can be easily and freely pulled from the Kraken API. For example, if you go to https://api.kraken.com/0/public/Ticker?pair=ETHEUR,XBTEUR using your browser, you should see returned JSON looking something like the following extremely redacted example:

{
   "error" : [],
   "result" : {
      "XXBTZEUR" : {
         "c" : [
            "2239.99000",
            "0.01870867"
         ],
         "a" : [
            "2239.99000",
            "3",
            "3.000"
         ],
      },
      "XETHZEUR" : {
         "c" : [
            "196.40030",
            "2.70918471"
         ],
         "a" : [
            "196.89291",
            "6",
            "6.000"
         ],
      }
   }
}

I can specify any number of pairs, but for the sake of exposition we’re going to work with Bitcoin in Euro, and Ethereum in Euro.

Query the ticker webservice using emacs-lisp

Next we’ll write some emacs-lisp code to embed in our orgmode file. In this case, this blog post is actually an orgmode file which I shall later export to wordpress.

The code makes use of TKF’s great request.el package, installable from MELPA. I started with one of the examples on the request.el github, and then used the shiny new let-alist macro to extract the first element of the c key (the last traded price) of the result - PAIR hierarchy.

The eth-eur and btc-eur pairs are stored in the cpb-kraken-etheur and cpb-kraken-xbteur variables respectively.

(require 'request)

(defun timestamp ()
  (format-time-string "%Y-%m-%dT%H:%M:%S"))

(request
 "https://api.kraken.com/0/public/Ticker"
 :params '(("pair" . "ETHEUR,XBTEUR"))
 :parser 'json-read
 :success (cl-function
           (lambda (&key data &allow-other-keys)
             ;; get out the last successful trade "c"
             (let-alist data
               (setq cpb-kraken-etheur
                     (string-to-number (aref .result.XETHZEUR.c 0)))
               (setq cpb-kraken-xbteur
                     (string-to-number (aref .result.XXBTZEUR.c 0)))
               (setq cpb-kraken-timestamp (timestamp))
               )
             (org-table-iterate-buffer-tables)
             (message "Retrieved Kraken ticker values at %s. ETHEUR: %f XBTEUR: %f"
                      cpb-kraken-timestamp cpb-kraken-etheur cpb-kraken-xbteur)
             )))

The code is embedded in this orgmode document using org-babel, i.e. in an emacs-lisp source code block:

#+BEGIN_SRC emacs-lisp :results none
(pretty lisp (code (that you see above ok?)))
#+END_SRC

Whenever I press C-c C-c with my cursor anywhere over the code, it will retrieve the current values into emacs-lisp variables, and the recalculate the table shown below.

Use the ticker data in a spreadsheet-like table

Next, we construct the orgmode table. If you have never done this with Emacs orgmode, you should try it. The UX for quickly making and maintaining text-mode tables is breathtaking. The table as you see it below is the HTML representation (generated by Emacs and orgmode) of the plaintext table as it lives in this org file:

coin units curr unit price curr val frac
eth 15.231 197.000000 3000.507 0.36
btc 2.335113 2252.100000 5258.9080 0.64
2017-06-03T14:10:14     8259.415  

Everything in column 3 and to the right, and in the last row, is calculated based on the ticker values we have pulled in using the emacs-lisp code above. The table will update whenever I press C-c C-c on the embedded code block, as it ends with (org-table-iterate-buffer-tables), meaning to recalculate all tables in this file, until convergence.

The orgmode formula editor (shortcut C-c '), enables you to edit all cell formulas with live highlighting of the references. The editor interface looks like this:

# Field and Range Formulas
@2$3 = '(format "%f" cpb-kraken-etheur)
@2$4 = $2*$3
@2$5 = $4/@II$4;%.2f
@3$3 = '(format "%f" cpb-kraken-xbteur)
@3$4 = $2*$3
@3$5 = $4/@II$4;%.2f
@4$1 = '(format "%s" cpb-kraken-timestamp)
@4$4 = vsum(@I..@II)

As you can see, I pull in the three variables retrieved and calculated by the emacs-lisp code using snippets of lisp. The other formulas are the standard spreadsheet fair with an orgmode flavour. @2$3 for example refers to the third column in the second row.

I can make any number of tables in this same file, depending on values from either the lisp code, or other tables. As per usual, I can export the file to PDF, HTML, ODF or even to a wordpress site, as I’m doing right now.

From org file with local bibtex to LaTeX and PDF

vxlabs-emacs-org-ref-pdf-example.png
Screenshot of orgmode source, PDF preview on the right, interactive citation selection in the minibuffer. Click for full resolution.

I have (co-)written a few LaTeX documents in my time.

However, as I have been writing my life and lab notes and many of my technical blog posts in Emacs orgmode for the past few years, I wanted to see how one would go about using BiBTeX references in orgmode files (using John Kitchin’s org-ref package) such that they would render correctly in orgmode’s export LaTeX and PDF.

My use case is of course slightly different than the norm: I maintain my master bibliography using the wonderful Zotero software, and I prefer exporting document-specific BiBTeX files from that main database.

This post shows you how to do it.

Pre-requisites

  • Make sure that latexmk is installed on your system. This is one of the easiest ways to make sure that pdflatex and bibtex are executed in the correct sequence, and a sufficient number of times.
  • With M-x package-install RET org-ref install the org-ref package. This will enable interactive selection and insertion of bibtex references from your local bib file.
  • You should also have the ox-latex Emacs package installed. This is what gives orgmode its LaTeX output powers.

Emacs setup

In your init.el somewhere, use the following code to ensure that the orgmode LaTeX exporter invokes latexmk in the correct way:

(setq org-latex-pdf-process
    '("latexmk -pdflatex='pdflatex -interaction nonstopmode' -pdf -bibtex -f %f"))

A minimal but complete orgmode example

The usual org-ref workflow is that you use it to manage a single main BiBTeX file, which is configured in your init.el. As mentioned above, I prefer using small document-specific BiBTeX files.

This is what the first three lines sets up using an Emacs file variable. Note that the local bib file is specified without quotes of any kind. Also, because there’s no path specified, it lives together with the orgmode document itself, which I prefer.

# Local Variables:
# org-ref-default-bibliography: local-bibtex-file.bib
# End:

After this, I have two LaTeX-specific lines ensuring that the output PDF looks prettier than default. You can thank me later.

The third LATEX_HEADER line imports the natbib package, which means I have extra cite commands to differentiate between for example textual citing (the first example) and parenthetical citing (the second example).

#+LATEX_CLASS_OPTIONS: [a4paper, 11pt, colorlinks=true, citecolor=., linkcolor=black, urlcolor=black]
#+LATEX_HEADER: \usepackage{fourier}
#+LATEX_HEADER: \usepackage{natbib}

#+TITLE: Example orgmode to LaTeX

* Introduction

  This is that really difficult intro spection, as was also
  demonstrated by cite:huang_identification_2016.

* Conclusions

  This is the even more difficult final section. We will now use a
  parenthetical citation citep:meyer_four-level_2012.

Finally, very importantly, I have the orgmode orgmode markup to select a suitable bibliography style, and again to specify the bib file. This will get automatically translated to the suitable LaTeX or even HTML versions.

Putting it all together

With something like the example above in your orgmode file, and a suitable bib-file, you can do C-c C-e l o to export to LaTeX, execute the latexmk command we specified above, and view the PDF.

In the screenshot at the start of this post, you can see what this looks like on my setup. I am using PDFTools for viewing the PDF inside of Emacs (this gets updated automatically whenever I rebuild the PDF).

You can also see the ivy-based completion as I’m in the process of interactively inserting a new citation into the document.

The default helm-based completion is more attractive, but it takes over the whole display, which is why I made use of ivy for instruction’s sake.

Sending emails with math and source code

Org mode is great for authoring rich documents with syntax highlighted source code, LaTeX math and images. It even supports evaluating live snippets of code embedded in the text. It does all of this whilst remaining a plain text format.

Imagine how useful it would be to author programming-related or technical emails using this functionality?

Imagine no more! org-mime, part of the org mode contrib, does this for a number of emacs-based mail clients. However, our preference is for mu4e, which is not part of that list.

Fortunately, it seems mu4e has orgmode support integrated. To get this working, configure the following in your ~/.emacs.d/init.el:

;; configure orgmode support in mu4e
(require 'org-mu4e)
;; when mail is sent, automatically convert org body to HTML
(setq org-mu4e-convert-to-html t)

When composing a new email, switch on the special mu4e / orgmode mode with M-x org~mu4e-mime-switch-headers-or-body (it will automatically switch between compose mode and orgmode depending on whether your cursor is on the headers or in the body).

You should add the following options to your org mode body. The first configures LaTeX (math) to be converted using imagemagick (instead of MathJax or dvipng), and the second removes the table of contents.

#+OPTIONS: tex:imagemagick
#+OPTIONS: toc:0

When you want to send the email, move to the headers, and then do C-c C-c. Because org-mu4e-convert-to-html is set to true, the orgmode body will automatically be converted to HTML before being sent.

Until my pull request is merged in, you will have to make the following change to function org~mu4e-mime-convert-to-html() in org-mu4e.el:

(insert (org~mu4e-mime-multipart
         body html (mapconcat 'identity html-images "\n")))

should be:

(insert (org~mu4e-mime-multipart
         raw-body html (mapconcat 'identity html-images "\n")))

In other words, body becomes raw-body, else your outgoing mails will have HTML in their text parts, instead of plain text.

In that same function, you might also want to change:

(org-export-preserve-breaks t)

to

(org-export-preserve-breaks nil)

So that the HTML email does not get hard linebreaks.

Following are examples of some of the elements you might include in such a rich text email. They should appear in your rich text capable email client as follows:

org-mu4e-example.png

Here are the originals:

A section with some math

We can easily include math using LaTeX:

\[ C(p,t) = \lbrace x_0(p,t), x_1(p,t), \cdots, x_{N-1}(p,t) \rbrace\tag{1} \]

Some syntax highlighted C++

Syntax highlighted source code, for all of the languages supported by emacs (a very long list):

int main() {
    // Recently discovered trick, local functions in C++!
    struct Kludge {
        static int doSomething(int x) {
            return x * 2;
        }
    };

    for (int i=0; i < 10; i++) {
        cout << Kludge::doSomething(i) << endl;
    }

    return 0;
}

Inline GraphViz DOT evaluation for graphs using Emacs, org-mode and org-babel

With Emacs, org mode and org-babel, it’s possible to evaluate source code samples embedded in your org files and have the output of said evaluation appear inline. This makes for a beautiful literate programming environment. It also enables one to include graphs in one’s documents (org mode, PDF, HTML presentations or blog posts) by using for example GraphViz.

This blog post (obviously authored using Emacs and Org mode) contains short instructions for doing so.

First follow the org-babel documentation and enable source code evaluation for dot by adding the following to your Emacs init.el:

(org-babel-do-load-languages
‘org-babel-load-languages
‘((dot . t)))

Then create a new document, and add something like the following dot source code sample to it:

#+BEGIN_SRC dot :file dot_success.png :cmdline -Kdot -Tpng
  digraph {
  // graph from left to right
  rankdir=LR;
  splines=true;
  node [shape=box];

  id [label="Install Graphviz"]
  conf [label="Configure org-babel"]
  dot [label="DOT in org-mode"]

  id -> conf
  conf -> dot
  dot -> "Profit"
  dot -> "Success" [style=dotted]
  }
#+END_SRC

Now press C-c C-c to evaluate this code. Emacs will generate the configured output file dot_success.png and then link to it in an automatically created #+RESULTS section right below it.

You can press C-c C-x C-v to toggle display of inline images to see it directly in Emacs. Alternatively, use M-x org-display-inline-images to switch this on. Whenever you change the DOT source code (press C-c ‘ to edit the dot source code in a separate buffer), just press C-c C-c to re-execute the updated source.

In any exported documents (for examlpe this blog post), only the output graph itself will appear, like this:

Exported as a PDF with C-e l o it looks like this.

In your Emacs, it should look like this:

This approach should work for any of the many languages supported by org-babel. Let me know in the comments what you come up with!