From org file with local bibtex to LaTeX and PDF

vxlabs-emacs-org-ref-pdf-example.png
Screenshot of orgmode source, PDF preview on the right, interactive citation selection in the minibuffer. Click for full resolution.

I have (co-)written a few LaTeX documents in my time.

However, as I have been writing my life and lab notes and many of my technical blog posts in Emacs orgmode for the past few years, I wanted to see how one would go about using BiBTeX references in orgmode files (using John Kitchin’s org-ref package) such that they would render correctly in orgmode’s export LaTeX and PDF.

My use case is of course slightly different than the norm: I maintain my master bibliography using the wonderful Zotero software, and I prefer exporting document-specific BiBTeX files from that main database.

This post shows you how to do it.

Pre-requisites

  • Make sure that latexmk is installed on your system. This is one of the easiest ways to make sure that pdflatex and bibtex are executed in the correct sequence, and a sufficient number of times.
  • With M-x package-install RET org-ref install the org-ref package. This will enable interactive selection and insertion of bibtex references from your local bib file.
  • You should also have the ox-latex Emacs package installed. This is what gives orgmode its LaTeX output powers.

Emacs setup

In your init.el somewhere, use the following code to ensure that the orgmode LaTeX exporter invokes latexmk in the correct way:

(setq org-latex-pdf-process
    '("latexmk -pdflatex='pdflatex -interaction nonstopmode' -pdf -bibtex -f %f"))

A minimal but complete orgmode example

The usual org-ref workflow is that you use it to manage a single main BiBTeX file, which is configured in your init.el. As mentioned above, I prefer using small document-specific BiBTeX files.

This is what the first three lines sets up using an Emacs file variable. Note that the local bib file is specified without quotes of any kind. Also, because there’s no path specified, it lives together with the orgmode document itself, which I prefer.

# Local Variables:
# org-ref-default-bibliography: local-bibtex-file.bib
# End:

After this, I have two LaTeX-specific lines ensuring that the output PDF looks prettier than default. You can thank me later.

The third LATEX_HEADER line imports the natbib package, which means I have extra cite commands to differentiate between for example textual citing (the first example) and parenthetical citing (the second example).

#+LATEX_CLASS_OPTIONS: [a4paper, 11pt, colorlinks=true, citecolor=., linkcolor=black, urlcolor=black]
#+LATEX_HEADER: \usepackage{fourier}
#+LATEX_HEADER: \usepackage{natbib}

#+TITLE: Example orgmode to LaTeX

* Introduction

  This is that really difficult intro spection, as was also
  demonstrated by cite:huang_identification_2016.

* Conclusions

  This is the even more difficult final section. We will now use a
  parenthetical citation citep:meyer_four-level_2012.

Finally, very importantly, I have the orgmode orgmode markup to select a suitable bibliography style, and again to specify the bib file. This will get automatically translated to the suitable LaTeX or even HTML versions.

Putting it all together

With something like the example above in your orgmode file, and a suitable bib-file, you can do C-c C-e l o to export to LaTeX, execute the latexmk command we specified above, and view the PDF.

In the screenshot at the start of this post, you can see what this looks like on my setup. I am using PDFTools for viewing the PDF inside of Emacs (this gets updated automatically whenever I rebuild the PDF).

You can also see the ivy-based completion as I’m in the process of interactively inserting a new citation into the document.

The default helm-based completion is more attractive, but it takes over the whole display, which is why I made use of ivy for instruction’s sake.

mu4e 0.9.18: E-Mailing with Emacs now even better.

In this post I talk about three great new features in the latest release of mu4e, an email programme that runs in Emacs. I also show my mu4e configuration as an example to others who would like a similar setup.

mu4e-0.9.18-screenshot.png
mu4e 0.9.18 screenshot showing selected context (bottom right of main) and visual-line-mode (long lines) which is activated by the format=flowed support.

After recently discovering that plaintext format=flowed in Thunderbird works only partially, it was time to check back in on mu4e after my previous happy stint using it.

I was pleasantly surprised to see that mu4e development had been quite active, and that this Emacs mail user agent had acquired a number of highly useful new features.

Three features I find particularly interesting are:

Full and deterministic format=flowed support

As I explained in another blog post, format=flowed is a really clever extension to old-school plain text emails that enables any receiving mail app that supports the standard to reflow emails for better display on smaller or larger displays.

mu4e now has first-class and deterministic support of the format=flowed feature.

I do understand that HTML has won the email format war.

However, HTML formatting gives users entirely too much freedom when writing their emails, resulting in misguided typesetting and other unnecessary visual elements, analogous to and just as irritating as chart junk, that regularly damage sensitive eyes.

This is why I prefer writing in plain text. With reliable format=flowed support, mu4e enables me to focus on the content whilst fixing plain-text email’s only real flaw: Reflowing on mobile devices.

Compose emails in a separate window

This, just like anything else, one could previously implement by oneself in emacs-lisp. However, it’s now a welcome core feature activated via the mu4e-compose-in-new-frame variable.

When you press C to compose a new email, a separate frame (this is what Emacs calls a window) is opened. After sending the email with C-c C-c the frame is automatically closed.

Contexts

In my previous setup, I had made a bunch of emacs-lisp functions to switch between different email identities (from addresses).

This worked, but it was not nearly as convenient and well thought-out as mu4e’s new contexts.

I now have a separate context defined for each email identity. I can switch between them using ; followed by the self-configured shortcut character.

They also activate automatically based on a configurable match function, for example examining the destination address of an email I am replying to. My configuration example below shows this in a little more detail.

My configuration

Because of these new features and a number of other changes, I put together this new configuration from scratch based on mu4e’s excellent documentation and examples.

I am still using `offlineimap` to synchronise my email with the IMAP server (fastmail in my case), but I’m currently sending email using the Emacs smtpmail package.

Here is a slightly shortened and simplified version of my configuration:

;; I installed mu4e from the 0.9.18 tarball
;; this is where make install put the emacs bits
(add-to-list 'load-path "/usr/local/share/emacs/site-lisp/mu4e")

(require 'mu4e)

;; I want my format=flowed thank you very much
;; mu4e sets up visual-line-mode and also fill (M-q) to do the right thing
;; each paragraph is a single long line; at sending, emacs will add the
;; special line continuation characters.
(setq mu4e-compose-format-flowed t)

;; every new email composition gets its own frame! (window)
(setq mu4e-compose-in-new-frame t)

;; give me ISO(ish) format date-time stamps in the header list
(setq mu4e-headers-date-format "%Y-%m-%d %H:%M")

;; show full addresses in view message (instead of just names)
;; toggle per name with M-RET
(setq mu4e-view-show-addresses 't)

;; path to our Maildir directory
(setq mu4e-maildir "~/Maildir")

;; the next are relative to `mu4e-maildir'
;; instead of strings, they can be functions too, see
;; their docstring or the chapter 'Dynamic folders'
(setq mu4e-sent-folder   "/Sent"
      mu4e-drafts-folder "/Drafts"
      mu4e-trash-folder  "/Trash")

;; the maildirs you use frequently; access them with 'j' ('jump')
(setq   mu4e-maildir-shortcuts
    '(("/Archive"     . ?a)
      ("/INBOX"       . ?i)
      ("/Sent"        . ?s)))

;; the list of all of my e-mail addresses
(setq mu4e-user-mail-address-list '("me@home.com"
                                    "me@work.com"
                                    "me@org.org"))

;; the headers to show in the headers list -- a pair of a field
;; and its width, with `nil' meaning 'unlimited'
;; (better only use that for the last field.
;; These are the defaults:
(setq mu4e-headers-fields
    '( (:date          .  25)    ;; alternatively, use :human-date
       (:flags         .   6)
       (:from          .  22)
       (:subject       .  nil))) ;; alternatively, use :thread-subject

;; program to get mail; alternatives are 'fetchmail', 'getmail'
;; isync or your own shellscript. called when 'U' is pressed in
;; main view.

;; If you get your mail without an explicit command,
;; use "true" for the command (this is the default)
;; when I press U in the main view, or C-c C-u elsewhere,
;; this command is called, followed by the mu indexer
(setq mu4e-get-mail-command "offlineimap")

;; not using smtp-async yet
;; some of these variables will get overridden by the contexts
(setq
 send-mail-function 'smtpmail-send-it
 message-send-mail-function 'smtpmail-send-it
 smtpmail-smtp-server "smtp.fastmail.com"
 smtpmail-smtp-service 465
 smtpmail-stream-type 'ssl
 )

;; don't keep message buffers around
(setq message-kill-buffer-on-exit t)

;; here come the contexts
;; I have about 5 of these, chopped down to 2 for demonstration purposes
;; each context can set any number of variables (see :vars)
;; for example below here I'm using two different SMTP servers depending on identity
(setq mu4e-contexts
      `( ,(make-mu4e-context
           :name "c me@home.com"
           :enter-func (lambda () (mu4e-message "Enter me@home.com context"))
           :leave-func (lambda () (mu4e-message "Leave me@home.com context"))
           ;; we match based on the contact-fields of the message (that we are replying to)
           ;; https://www.djcbsoftware.nl/code/mu/mu4e/What-are-contexts.html#What-are-contexts
           :match-func (lambda (msg)
                         (when msg 
                           (mu4e-message-contact-field-matches msg 
                                                               :to "me@home.com")))
           :vars '( ( user-mail-address      . "me@home.com"  )
                    ( user-full-name         . "Charl P. Botha" )
                    ( smtpmail-smtp-server   . "smtp.fastmail.com" )
                    ( mu4e-compose-signature .
                                             (concat
                                              "dr. charl p. botha\n"
                                              "http://charlbotha.com/\n"))))

         ,(make-mu4e-context
           :name "s me@org.org"
           :enter-func (lambda () (mu4e-message "Enter me@org.org context"))
           ;; no leave-func
           ;; we match based on the contact-fields of the message
           :match-func (lambda (msg)
                         (when msg 
                           (mu4e-message-contact-field-matches msg 
                                                               :to "me@org.org")))
           :vars '( ( user-mail-address       . "me@org.org" )
                    ( user-full-name          . "Charl P. Botha" )
                    ( smtpmail-smtp-server    . "smtp.gmail.com" )
                    ( mu4e-compose-signature  .
                                              (concat
                                               "Dr. Charl P. Botha\n"
                                               "Science Officer :: Stone Three Venture Technology\n"
                                               "https://www.stonethree.com\n"))))         

         ))

;; start with the first (default) context; 
(setq mu4e-context-policy 'pick-first)

;; compose with the current context if no context matches;
(setq mu4e-compose-context-policy nil)

;; these are the standard mu4e search bookmarks
;; I've only added the fourth one to pull up flagged emails in my inbox
;; I sometimes use this to shortlist emails I need to get around to ASAP
(setq mu4e-bookmarks
  `( ,(make-mu4e-bookmark
       :name  "Unread messages"
       :query "flag:unread AND NOT flag:trashed"
       :key ?u)
     ,(make-mu4e-bookmark
       :name "Today's messages"
       :query "date:today..now"
       :key ?t)
     ,(make-mu4e-bookmark
       :name "Last 7 days"
       :query "date:7d..now"
       :key ?w)
     ,(make-mu4e-bookmark
       :name "Flagged in INBOX"
       :query "maildir:\"/INBOX\" and flag:flagged"
       :key ?f)))

Conclusion

This is currently running smoothly on two of my Linux workstations as well as my early 2015 retina MacBook Pro.

The general setup procedure is to install offlineimap using pip2 install --user offlineimap (the offlineimap configuration file hasn’t changed much since the previous post), run offlineimap for the first time, then build and install mu4e from its tarball, and finally to run mu index to initialise the index.

After this, your mail, and plain-text nirvana, is a simple M-x mu4e away.

Date-sorted interactive recursive search with ivy, counsel and ag

A core part of my note-taking strategy is a growing collection of Emacs Org mode files containing my monthly lab journals, project summaries and various technical documents.

Usually I rely on Projectile’s Helm integration to perform recursive searches with The Silver Searcher (ag) using helm-projectile-ag (trigged by the completely muscle-memorised C-c p s s) through all of the org files in my notes hierarchy. Recursive regular expression search results are shown interactively as you type. Wonderful!

However, recently I found that I really missed the ability to see my search results sorted by last modified date, so that the most recently touched files would show up first. For a chronological notes database, this sorting makes the most sense.

A few frustrating hours later, interspersed with bouts of seriously pondering the exact nature of my relationship with Emacs, I managed to attain the desired result using the Emacs packages ivy and counsel.

Although I’m sticking with Helm for the rest, in this case the ivy API made it much simpler to do what I needed to do. It is understandable that this is not a straight-forward endeavour, as ag is heavily parallelised and furthermore is handled asynchronously by both helm and ivy / counsel. Sorting the results slows matters down, but for this specific case the slowdown is a reasonable price to pay.

Read on for more detail.

Video demo of end result

If you follow the instruction below, you too will be able to achieve the following feats of recall:

Search results are returned as I type, as is usually the case with counsel-ag and helm-ag. However, they are now sorted by file modified date, last modified file at the top. This usually means I find what I need faster.

Show me the code

The ivy-read function (this is ivy’s single API entrypoint) has two required parameters, one of which is the collection. This is a function returning the list of items that the user wants to narrow down, or the list itself.

Because we need to tell ivy to sort the results by associating a sorting function with a specific collection function, we wrap the supplied counsel-ag-function collection function.

We also supply a new sorting function that will take two of the ag-generated search results, each of the form relative_filename:line_number:line_contents, derive the filenames from that, determine the modified timestamps of both the files and then return t or nil depending on which file is newer.

We finally (for this first snippet) associate the sorting function with our wrapped collection function.

(defun cpb/ag-collection (string)
  "Search for pattern STRING using ag.
We only have this as a separate function so we can assoc with sort function."
  ;; this will use counsel--async-command to run asynchronously
  (counsel-ag-function string ""))

;; modified from https://github.com/abo-abo/swiper/wiki/Sort-files-by-mtime
;; * directory to counsel--git-grep-dir: that's where AG is working
;; * removed directory logic: ag returns files
;; * parsed out filename from ag results
(defun eh-ivy-sort-file-by-mtime (x y)
  "Determine if AG sort result X is newer than Y."
  (let* ((x (concat counsel--git-grep-dir (car (split-string x ":"))))
         (y (concat counsel--git-grep-dir (car (split-string y ":"))))
         (x-mtime (nth 5 (file-attributes x)))
         (y-mtime (nth 5 (file-attributes y))))
    (time-less-p y-mtime x-mtime)))

(add-to-list 'ivy-sort-functions-alist
             '(cpb/ag-collection . eh-ivy-sort-file-by-mtime))

In the second and final snippet, we create a new version of the counsel-ag function where we specify our own collection function, and we supply a truthful :sort argument.

(defun cpb/counsel-ag (&optional initial-input initial-directory
                                 extra-ag-args ag-prompt)
  "Grep for a string in the current directory using ag.
INITIAL-INPUT can be given as the initial minibuffer input.
INITIAL-DIRECTORY, if non-nil, is used as the root directory for search.
EXTRA-AG-ARGS string, if non-nil, is appended to `counsel-ag-base-command'.
AG-PROMPT, if non-nil, is passed as `ivy-read' prompt argument.

Modified by cpbotha: Sort results last modified file first."
  (interactive
   (list nil
         (when current-prefix-arg
           (read-directory-name (concat
                                 (car (split-string counsel-ag-base-command))
                                 " in directory: ")))))
  (ivy-set-prompt 'counsel-ag counsel-prompt-function)
  (setq counsel--git-grep-dir (or initial-directory default-directory))
  (ivy-read (or ag-prompt (car (split-string counsel-ag-base-command)))
            ;; specify collection function so we can assoc with sort function
            #'cpb/ag-collection
            :initial-input initial-input
            :dynamic-collection t
            ;; yes, we want to sort the results
            :sort t
            :keymap counsel-ag-map
            :history 'counsel-git-grep-history
            :action #'counsel-git-grep-action
            :unwind (lambda ()
                      (counsel-delete-process)
                      (swiper--cleanup))
            :caller 'cpb/counsel-ag))

Closing

You can invoke cpb/counsel-ag interactively to search from the current directory, or you can point it at a directory of your choosing by passing in the initial-directory argument.

In my Emacs initialisation, I have bound another wrapper function that invokes cpb/counsel-ag on my notes directory, so I can instantly search through my notes in reverse chronological order.

I believe that my relationship with Emacs is safe for now.

Step-by-step guide to C++ navigation and completion with Emacs and the Clang-based rtags

If you want C++ completion and navigation (jump to definition, jump to declaration, and so forth), there are several good options for Emacs. For a QtQuick / C++ project I’m working on, I needed the best Emacs has to offer.

This turned out to be the Clang-based rtags system.

Rtags is not the easiest of the options to get going, hence this short tutorial. I initially configured irony-mode, which is also Clang-based and was significantly easier to get going, but it soon started hanging on the completion of for example QStringList methods in my project. Because it also doesn’t support navigation, I decided to try rtags. So far, rtags has been working quite well on a Qt 5.x project of slightly under 200K lines of C++.

Pre-requisites

On Ubuntu 14.04, I tested this with both libclang-3.6-dev from the standard repos and with libclang-3.8-dev from the LLVM repos. CMake should also be installed.

On OSX 10.11 (El Capitan), homebrew took care of the dependencies for me (see next step).

Get and build rtags

From the documentation, simply clone the rtags repo and build it:

git clone --recursive https://github.com/Andersbakken/rtags.git
cd rtags
cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=1 .
make

Make a compilation database for your project

rtags needs a compilation database file, usually called compile_commands.json, to figure out what compile options were used to compile which source files. If your project is using cmake, you only have to add the -DCMAKE_EXPORT_COMPILE_COMMANDS=1 to the cmake invocation to generate that file.

In my case, and perhaps in yours, you have to use something else like Qt’s qmake system. In this case, you can use a tool like Build Ear (BEAR) or the Python-rewrite of scan-build. In both cases, you invoke the utility with your normal build command. In my Qt project, I do:

bear --append make -j16

As your project builds, bear will capture all of the compilation commands and put them in compile_commands.json. With my project, the compile_commands.json finds itself in the out-of-source (shadow) build directory.

(The first time, you have to do this after a clean, or a make clean, so all of the compile commands can be captured.)

BEAR is preferable, but on the latest OSX with its new system integrity protection (SIP) and on some linuxes, it won’t work and you’ll end up with an empty compilation database. In these cases, use the intercept-build command from the scan-build package linked above.

Start the rtags daemon RDM

You can do this from emacs, but due to better system process handling, I invoke this from a normal terminal:

cd /where/you/built/rtags/bin
nohup ./rdm &

When working with rtags, always make sure that RDM is running.

Tell RDM about your project using RC

rc is the rtags command one uses to communicate with the rtags daemon RDM.

Importantly, make sure that rc is in your path.

We only have to tell RDM once about the location of any new project. Do this as follows:

cd /your/project/source
rc -J /dir/containing/compile_commands.json/

Note that the -J parameter is the directory containing the compile_commands.json file.

Configure your Emacs

Install the Emacs rtags package:

M-x package-install RET rtags RET

Make sure that you also have the company package installed.

Next, add the following to your Emacs init.el:

;; ensure that we use only rtags checking
;; https://github.com/Andersbakken/rtags#optional-1
(defun setup-flycheck-rtags ()
  (interactive)
  (flycheck-select-checker 'rtags)
  ;; RTags creates more accurate overlays.
  (setq-local flycheck-highlighting-mode nil)
  (setq-local flycheck-check-syntax-automatically nil))

;; only run this if rtags is installed
(when (require 'rtags nil :noerror)
  ;; make sure you have company-mode installed
  (require 'company)
  (define-key c-mode-base-map (kbd "M-.")
    (function rtags-find-symbol-at-point))
  (define-key c-mode-base-map (kbd "M-,")
    (function rtags-find-references-at-point))
  ;; disable prelude's use of C-c r, as this is the rtags keyboard prefix
  (define-key prelude-mode-map (kbd "C-c r") nil)
  ;; install standard rtags keybindings. Do M-. on the symbol below to
  ;; jump to definition and see the keybindings.
  (rtags-enable-standard-keybindings)
  ;; comment this out if you don't have or don't use helm
  (setq rtags-use-helm t)
  ;; company completion setup
  (setq rtags-autostart-diagnostics t)
  (rtags-diagnostics)
  (setq rtags-completions-enabled t)
  (push 'company-rtags company-backends)
  (global-company-mode)
  (define-key c-mode-base-map (kbd "<C-tab>") (function company-complete))
  ;; use rtags flycheck mode -- clang warnings shown inline
  (require 'flycheck-rtags)
  ;; c-mode-common-hook is also called by c++-mode
  (add-hook 'c-mode-common-hook #'setup-flycheck-rtags))

You can either just eval this new code, or restart your emacs.

Using rtags

I mostly use M-. to jump to any symbol’s definition. Pressing M-. again will jump to and fro between definition and declaration. Use =M-,= to see all references to the symbol under the cursor. This is where Helm comes in especially useful.

Use C-c r [ and C-c r ] to jump backwards and forwards through the position stack that rtags builds up as you jump between symbols. You can use C-c r I to get a list of rtags-extracted symbols in the current file, although semantic-mode with helm (C-c h i) gives you slightly less accurate but more colourful results. I use both.

As you type, company-mode will usually interrupt after you’ve entered the first three characters of a symbol or member. I can’t wait that long, so I press C-tab to get a list of completions immediately.

As an additional bonus, you’ll see Clang warnings indicated in bright red (usually) in your code window!

As you’re working, remember to do builds with bear --append as shown above. I do this via M-x compile. rdm will pick up all changes and re-analyse files as you work!

Conclusions and other rambling

I am quite impressed by how well rtags works on complex C++ projects! It is indeed clang under the hood, but rtags has overcome the considerable challenge of exposing this to Emacs in a robust and usable fashion.

With this setup, I am able to take longer breaks from using QtCreator, which I don’t like very much. QtCreator has great potential, but unfortunately its Emacs keyboard emulation is sorely lacking (this could be forgiven) and as a general C++ IDE it still has a long way to go in the usability department. I remain optimistic that with competition like JetBrains (when I’m not in Emacs, I’m in something by JetBrains; their Emacs emulation is bearable!) out there, QtCreator will improve over time.

Feel free to join the discussion in the comments below or on the reddit topic!

Fixing the Cordova browser platform Access-Control-Allow-Origin error

When developing a mobile app using Cordova or PhoneGap, the browser target platform can really speed up your development. You could serve the HTML files directly using cordova serve, but the browser platform, while being almost as fast, is much closer to the Android / IOS environments your app will eventually find itself in. It also has access to the Cordova APIs.

My general workflow is that I keep cordova run browser running in a terminal window (this will initially start a new instance of chrome), and periodically run cordova prepare browser as I’m developing. This last step packages up the “app”, along with the cordova.js bits, for the browser platform.

However, if your app contacts an external API or server at any point, for example with jQuery.ajax(), you will be greeted by the following error:

XMLHttpRequest cannot load http://some.api.com/api/endpoint. No
'Access-Control-Allow-Origin' header is present on the requested
resource. Origin 'http://localhost:8000' is therefore not allowed
access.

In short, your chrome browser is (wisely) refusing to access that outside resource (the API or server) for security reasons. If your browser did not do this, an attacker could quite easily access web services using your identity!

Reading up about this issue in the context of Cordova / PhoneGap, you’ll find the common solution to be either to modify the server that you’re accessing to allow explicitly this sort of cross-origin access, or to setup a proxy if the former is not possible. Often the former is indeed not possible (in my case, the app is talking to a hardware router API), but the latter is just way too much unnecessary effort.

The easy (but good) fix

Recall that cordova starts up a new instance of chrome which is only used to access your app. The app is completely under your control, so the risk of foul play by a third party is significantly reduced. Instead of setting up proxies, or changing servers, we could just instruct this special instance of chrome to ignore its cross-origin rules!

It took me a while to track this down, so here you go: Edit browser.js in yourapp/platforms/browser/cordova/node_modules/cordova-serve/src and change just the chromeArgs line so it looks like the one in the snippet below (you’re just adding the --disable-web-security argument):

function getBrowser(target, dataDir) {
    dataDir = dataDir || 'temp_chrome_user_data_dir_for_cordova';

    var chromeArgs = ' --user-data-dir=/tmp/' + dataDir + ' --disable-web-security';

Now make 100% sure that all of your Cordova chrome instances are stopped. The next time you do cordova run browser, note that the chrome window that appears has a yellow bar warning you about web security:

chrome-disable-web-security-warning.jpg

Only the special chrome instances started up by Cordova for apps where you have applied the above fix will have web-security disabled. By all means take note of the yellow warning, but also do enjoy all of your Cordova app requests successfully contacting the outside world!