Fixing crux-open-with on Ubuntu

The crux package for Emacs contains, amongst a list of useful functions (crux does stand for A Collection of Ridiculously Useful eXtensions for Emacs after all), the function crux-open-with. This interactive function opens the file you currently have open, or the file under your cursor in dired, using the system application for that file.

On Ubuntu 16.04, this function unfortunately does not work. It looks like the application tries to start (icon visible in the dash) and then dies.

After some debugging I discovered that on Ubuntu, the system tool xdg-open (which is invoked by crux) starts gvfs-open which on its part does not wait for the app to finish, but returns immediately.

The internal Emacs function start-process used by crux-open-with already runs the child process asynchronously. The early return of gvfs-open results in the sub-process closing, which means your app never gets off the ground.

To fix this, we have to get xdg-open to work in generic (i.e. non-Gnome / GVFS) mode and thus to wait for the app it spawns to return. We do this by setting the environment variable XDG_CURRENT_DESKTOP to X-Generic.

Fortunately, we can do this fairly neatly by wrapping the crux-open-with invocation in a let where we temporarily change the environment.

In my case it looks like this:

(use-package crux
  (defun cpb-crux-open-with (arg)
    (interactive "P")
    (let ((process-environment (cons "XDG_CURRENT_DESKTOP=X-Generic" process-environment)))
      (crux-open-with arg)))
  (global-set-key (kbd "C-c o") 'cpb-crux-open-with))

If you don’t also use use-package (you really should!) just copy out the part after :config.

This has been tested to work on Ubuntu 16.04 in both Unity and Gnome Shell 3.18.


  • 2017-04-18: Fixed environment variable + tested in Gnome Shell.

Miniconda3, TensorFlow, Keras on Google Compute Engine GPU instance: The step-by-step guide.

Google recently announced the availability of GPUs on Google Compute Engine instances. For my deep learning experiments, I often need more beefy GPUs than the puny GTX 750Ti in my desktop workstation, so this was good news. To make the GCE offering even more attractive, their GPU instances are also available in their EU datacenters, which is in terms of latency a big plus for me here on the Southern tip of the African continent.

Last night I had some time to try this out, and in this post I would like to share with you all the steps I took to:

  1. Get a GCE instance with GPU up and running with miniconda, TensorFlow and Keras
  2. Create a reusable disk image with all software pre-installed so that I could bring up new instances ready-to-roll at the drop of a hat.
  3. Apply the pre-trained Resnet50 deep neural network on images from the web, as a demonstration that the above works. Thanks to Keras, this step is fun and fantastically straight-forward.


I started by creating a project for this work. On the Compute Engine console, check that this project is active at the top.

Before I was able to allocate GPUs to my instance, I had to fill in the “request quote increase” form available from the Compute Engine quotas page. My request for two GPUs in the EU region was approved within minutes.

I installed my client workstation’s public SSH key as a project-wide SSH key via the metadata screen.

Start an instance for the first time

I configured my GPU instance as shown in the following screenshot:


  • Under Machine type switch to Customize to be able to select a GPU.
  • I selected an Ubuntu 16.04 image, and changed the persistent disk to SSD.
  • I selected the europe-west1-b zone. Choose whatever is closest for you. The interface will warn you if the selection does NOT support GPUs.

After this, click on the Create button and wait for your instance to become ready.

Once it’s up and running, you’ll be able to ssh to the displayed public IP. I used the ssh on my client workstation, but of course you could opt for the Google-supplied web-based versions.

Install NVIDIA drivers and CUDA

I used the following handy script from the relevant GCE documentation:

echo "Checking for CUDA and installing."
# Check for CUDA and try to install.
if ! dpkg-query -W cuda; then
  # The 16.04 installer works with 16.10.
  curl -O
  dpkg -i ./cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
  apt-get update
  apt-get install cuda -y

After this, download the CUDNN debs from the NVIDIA download site using your developer account. Install the two debs using dpkg -i.

To confirm that the drivers have been installed, run the nvidia-smi command:


Install miniconda, tensorflow and keras

I usually download the 64bit Linux miniconda installer from and then install it into ~/miniconda3 by running the downloaded .sh script.

After this, I installed TensorFlow 1.0.1 and Keras 2.0.1 into a new conda environment by doing:

conda create -n ml python=3.6
conda install jupyter pandas numpy scipy scikit-image
pip install --ignore-installed --upgrade
pip install keras h5py

The keras package also installed theano, which I then uninstalled using pip uninstall theano in the active ml environment.

To test, run ipython and then type import keras. It should look like this:


Note that it’s picking up the TensorFlow backend, and successfully loading all of the CUDA librarias, including CUDNN.

Save your disk as an image for later

You will get billed for each minute that the instance is running. You also get billed for persistent disks that are still around, even if they are not used by any instance.

Creating a reusable disk image will enable you to delete instances and disks, and later to restart an instance with all of your software already installed.

To do this, follow the steps in the documentation, which I paraphrase and extend here:

  1. Stop the instance.
  2. In the instance list, click on the instance name itself; this will take you to the edit screen.
  3. Click the edit button, and then uncheck Delete boot disk when instance is deleted.
  4. Click the save button.
  5. Delete the instance, but double-check that delete boot disk is unchecked in the confirmation dialog.
  6. Now go to the Images screen and select Create Image with the boot disk as source.

Next time, go to the Images screen, select your image and then select Create Instance. That instance will come with all of your goodies ready to go!

Apply the ResNet50 neural network on images from the interwebs

After connecting to the instance with an SSH port redirect:

ssh -L 8889:localhost:8888 cpbotha@EXTERNAL_IP

… and then starting a jupyter notebook on the GCE instance:

cpbotha@instance-1:~$ source ~/miniconda3/bin/activate ml
(ml) cpbotha@instance-1:~$ jupyter notebook

So that I can connect to the notebook on my localhost:8889, I enter and execute the following code (adapted from the Keras documentation) in a cell:

from keras.applications.resnet50 import ResNet50
from keras.preprocessing import image
from keras.applications.resnet50 import preprocess_input, decode_predictions
import numpy as np
from PIL import Image

model = ResNet50(weights='imagenet')

# adapted from
# to accept also a PIL image
def load_and_predict_image(img_or_path):
    target_size = (224,224)
    if type(img_or_path) is str:
        img = image.load_img(img_or_path, target_size=target_size)

        img = img_or_path.resize(target_size)

    x = image.img_to_array(img)
    x = np.expand_dims(x, axis=0)
    x = preprocess_input(x)

    preds = model.predict(x)
    # decode the results into a list of tuples (class, description, probability)
    # (one such list for each sample in the batch)
    print('Predicted:', decode_predictions(preds, top=3)[0])

In the next cell, I do:

from PIL import Image
import urllib.request

url1 = ""
url2 = ""
im =


To be greeted with the following results:


The pre-trained ResNet50 network identified the Kudu I gave it initially as an ostrich, so I decided to make it a bit easier for the poor network by actually giving it an ostrich, which it did identify with a 99.98% probability.

Looking at the photos, the former taken in the Kruger National Park and the second in Etosha, I can image that the network could identify the former as the latter due to similar background and foreground colouring, and clearly having not been trained on Kudu.

Let’s file that under future work!

Recursive text search in project without projectile

If you’re not using projectile, but you would like to be able to perform interactive, search-as-you-type, recursive searches through the current project, this is pretty easy to do if you have counsel and find-file-in-project installed.

Add the following Emacs Lisp to your init:

(defun cpb/counsel-ag-in-project ()
  "Use `ffip' and `counsel-ag' for quick project-wide text searching."
  (let ((project-root (ffip-project-root)))
    ;; if ffip could not find project-root, it will already have
    ;; shown an error message. We only have to check for non-nil.
    (if project-root
        (counsel-ag nil project-root nil
                    (format "Search in PRJ %s" project-root)))))

(global-set-key (kbd "C-c s") 'cpb/counsel-ag-in-project)

This function, which I have bound to ctrl-c s, finds the project root using find-file-in-project's function for this, and then invokes the handy counsel-ag on its result.

From org file with local bibtex to LaTeX and PDF

Screenshot of orgmode source, PDF preview on the right, interactive citation selection in the minibuffer. Click for full resolution.

I have (co-)written a few LaTeX documents in my time.

However, as I have been writing my life and lab notes and many of my technical blog posts in Emacs orgmode for the past few years, I wanted to see how one would go about using BiBTeX references in orgmode files (using John Kitchin’s org-ref package) such that they would render correctly in orgmode’s export LaTeX and PDF.

My use case is of course slightly different than the norm: I maintain my master bibliography using the wonderful Zotero software, and I prefer exporting document-specific BiBTeX files from that main database.

This post shows you how to do it.


  • Make sure that latexmk is installed on your system. This is one of the easiest ways to make sure that pdflatex and bibtex are executed in the correct sequence, and a sufficient number of times.
  • With M-x package-install RET org-ref install the org-ref package. This will enable interactive selection and insertion of bibtex references from your local bib file.
  • You should also have the ox-latex Emacs package installed. This is what gives orgmode its LaTeX output powers.

Emacs setup

In your init.el somewhere, use the following code to ensure that the orgmode LaTeX exporter invokes latexmk in the correct way:

(setq org-latex-pdf-process
    '("latexmk -pdflatex='pdflatex -interaction nonstopmode' -pdf -bibtex -f %f"))

A minimal but complete orgmode example

The usual org-ref workflow is that you use it to manage a single main BiBTeX file, which is configured in your init.el. As mentioned above, I prefer using small document-specific BiBTeX files.

This is what the first three lines sets up using an Emacs file variable. Note that the local bib file is specified without quotes of any kind. Also, because there’s no path specified, it lives together with the orgmode document itself, which I prefer.

# Local Variables:
# org-ref-default-bibliography: local-bibtex-file.bib
# End:

After this, I have two LaTeX-specific lines ensuring that the output PDF looks prettier than default. You can thank me later.

The third LATEX_HEADER line imports the natbib package, which means I have extra cite commands to differentiate between for example textual citing (the first example) and parenthetical citing (the second example).

#+LATEX_CLASS_OPTIONS: [a4paper, 11pt, colorlinks=true, citecolor=., linkcolor=black, urlcolor=black]
#+LATEX_HEADER: \usepackage{fourier}
#+LATEX_HEADER: \usepackage{natbib}

#+TITLE: Example orgmode to LaTeX

* Introduction

  This is that really difficult intro spection, as was also
  demonstrated by cite:huang_identification_2016.

* Conclusions

  This is the even more difficult final section. We will now use a
  parenthetical citation citep:meyer_four-level_2012.

Finally, very importantly, I have the orgmode orgmode markup to select a suitable bibliography style, and again to specify the bib file. This will get automatically translated to the suitable LaTeX or even HTML versions.

Putting it all together

With something like the example above in your orgmode file, and a suitable bib-file, you can do C-c C-e l o to export to LaTeX, execute the latexmk command we specified above, and view the PDF.

In the screenshot at the start of this post, you can see what this looks like on my setup. I am using PDFTools for viewing the PDF inside of Emacs (this gets updated automatically whenever I rebuild the PDF).

You can also see the ivy-based completion as I’m in the process of interactively inserting a new citation into the document.

The default helm-based completion is more attractive, but it takes over the whole display, which is why I made use of ivy for instruction’s sake.

mu4e 0.9.18: E-Mailing with Emacs now even better.

In this post I talk about three great new features in the latest release of mu4e, an email programme that runs in Emacs. I also show my mu4e configuration as an example to others who would like a similar setup.

mu4e 0.9.18 screenshot showing selected context (bottom right of main) and visual-line-mode (long lines) which is activated by the format=flowed support.

After recently discovering that plaintext format=flowed in Thunderbird works only partially, it was time to check back in on mu4e after my previous happy stint using it.

I was pleasantly surprised to see that mu4e development had been quite active, and that this Emacs mail user agent had acquired a number of highly useful new features.

Three features I find particularly interesting are:

Full and deterministic format=flowed support

As I explained in another blog post, format=flowed is a really clever extension to old-school plain text emails that enables any receiving mail app that supports the standard to reflow emails for better display on smaller or larger displays.

mu4e now has first-class and deterministic support of the format=flowed feature.

I do understand that HTML has won the email format war.

However, HTML formatting gives users entirely too much freedom when writing their emails, resulting in misguided typesetting and other unnecessary visual elements, analogous to and just as irritating as chart junk, that regularly damage sensitive eyes.

This is why I prefer writing in plain text. With reliable format=flowed support, mu4e enables me to focus on the content whilst fixing plain-text email’s only real flaw: Reflowing on mobile devices.

Compose emails in a separate window

This, just like anything else, one could previously implement by oneself in emacs-lisp. However, it’s now a welcome core feature activated via the mu4e-compose-in-new-frame variable.

When you press C to compose a new email, a separate frame (this is what Emacs calls a window) is opened. After sending the email with C-c C-c the frame is automatically closed.


In my previous setup, I had made a bunch of emacs-lisp functions to switch between different email identities (from addresses).

This worked, but it was not nearly as convenient and well thought-out as mu4e’s new contexts.

I now have a separate context defined for each email identity. I can switch between them using ; followed by the self-configured shortcut character.

They also activate automatically based on a configurable match function, for example examining the destination address of an email I am replying to. My configuration example below shows this in a little more detail.

My configuration

Because of these new features and a number of other changes, I put together this new configuration from scratch based on mu4e’s excellent documentation and examples.

I am still using `offlineimap` to synchronise my email with the IMAP server (fastmail in my case), but I’m currently sending email using the Emacs smtpmail package.

Here is a slightly shortened and simplified version of my configuration:

;; I installed mu4e from the 0.9.18 tarball
;; this is where make install put the emacs bits
(add-to-list 'load-path "/usr/local/share/emacs/site-lisp/mu4e")

(require 'mu4e)

;; I want my format=flowed thank you very much
;; mu4e sets up visual-line-mode and also fill (M-q) to do the right thing
;; each paragraph is a single long line; at sending, emacs will add the
;; special line continuation characters.
(setq mu4e-compose-format-flowed t)

;; every new email composition gets its own frame! (window)
(setq mu4e-compose-in-new-frame t)

;; give me ISO(ish) format date-time stamps in the header list
(setq mu4e-headers-date-format "%Y-%m-%d %H:%M")

;; show full addresses in view message (instead of just names)
;; toggle per name with M-RET
(setq mu4e-view-show-addresses 't)

;; path to our Maildir directory
(setq mu4e-maildir "~/Maildir")

;; the next are relative to `mu4e-maildir'
;; instead of strings, they can be functions too, see
;; their docstring or the chapter 'Dynamic folders'
(setq mu4e-sent-folder   "/Sent"
      mu4e-drafts-folder "/Drafts"
      mu4e-trash-folder  "/Trash")

;; the maildirs you use frequently; access them with 'j' ('jump')
(setq   mu4e-maildir-shortcuts
    '(("/Archive"     . ?a)
      ("/INBOX"       . ?i)
      ("/Sent"        . ?s)))

;; the list of all of my e-mail addresses
(setq mu4e-user-mail-address-list '(""

;; the headers to show in the headers list -- a pair of a field
;; and its width, with `nil' meaning 'unlimited'
;; (better only use that for the last field.
;; These are the defaults:
(setq mu4e-headers-fields
    '( (:date          .  25)    ;; alternatively, use :human-date
       (:flags         .   6)
       (:from          .  22)
       (:subject       .  nil))) ;; alternatively, use :thread-subject

;; program to get mail; alternatives are 'fetchmail', 'getmail'
;; isync or your own shellscript. called when 'U' is pressed in
;; main view.

;; If you get your mail without an explicit command,
;; use "true" for the command (this is the default)
;; when I press U in the main view, or C-c C-u elsewhere,
;; this command is called, followed by the mu indexer
(setq mu4e-get-mail-command "offlineimap")

;; not using smtp-async yet
;; some of these variables will get overridden by the contexts
 send-mail-function 'smtpmail-send-it
 message-send-mail-function 'smtpmail-send-it
 smtpmail-smtp-server ""
 smtpmail-smtp-service 465
 smtpmail-stream-type 'ssl

;; don't keep message buffers around
(setq message-kill-buffer-on-exit t)

;; here come the contexts
;; I have about 5 of these, chopped down to 2 for demonstration purposes
;; each context can set any number of variables (see :vars)
;; for example below here I'm using two different SMTP servers depending on identity
(setq mu4e-contexts
      `( ,(make-mu4e-context
           :name "c"
           :enter-func (lambda () (mu4e-message "Enter context"))
           :leave-func (lambda () (mu4e-message "Leave context"))
           ;; we match based on the contact-fields of the message (that we are replying to)
           :match-func (lambda (msg)
                         (when msg 
                           (mu4e-message-contact-field-matches msg 
                                                               :to "")))
           :vars '( ( user-mail-address      . ""  )
                    ( user-full-name         . "Charl P. Botha" )
                    ( smtpmail-smtp-server   . "" )
                    ( mu4e-compose-signature .
                                              "dr. charl p. botha\n"

           :name "s"
           :enter-func (lambda () (mu4e-message "Enter context"))
           ;; no leave-func
           ;; we match based on the contact-fields of the message
           :match-func (lambda (msg)
                         (when msg 
                           (mu4e-message-contact-field-matches msg 
                                                               :to "")))
           :vars '( ( user-mail-address       . "" )
                    ( user-full-name          . "Charl P. Botha" )
                    ( smtpmail-smtp-server    . "" )
                    ( mu4e-compose-signature  .
                                               "Dr. Charl P. Botha\n"
                                               "Science Officer :: Stone Three Venture Technology\n"


;; start with the first (default) context; 
(setq mu4e-context-policy 'pick-first)

;; compose with the current context if no context matches;
(setq mu4e-compose-context-policy nil)

;; these are the standard mu4e search bookmarks
;; I've only added the fourth one to pull up flagged emails in my inbox
;; I sometimes use this to shortlist emails I need to get around to ASAP
(setq mu4e-bookmarks
  `( ,(make-mu4e-bookmark
       :name  "Unread messages"
       :query "flag:unread AND NOT flag:trashed"
       :key ?u)
       :name "Today's messages"
       :query ""
       :key ?t)
       :name "Last 7 days"
       :query ""
       :key ?w)
       :name "Flagged in INBOX"
       :query "maildir:\"/INBOX\" and flag:flagged"
       :key ?f)))


This is currently running smoothly on two of my Linux workstations as well as my early 2015 retina MacBook Pro.

The general setup procedure is to install offlineimap using pip2 install --user offlineimap (the offlineimap configuration file hasn’t changed much since the previous post), run offlineimap for the first time, then build and install mu4e from its tarball, and finally to run mu index to initialise the index.

After this, your mail, and plain-text nirvana, is a simple M-x mu4e away.