Fixing crux-open-with on Ubuntu

The crux package for Emacs contains, amongst a list of useful functions (crux does stand for A Collection of Ridiculously Useful eXtensions for Emacs after all), the function crux-open-with. This interactive function opens the file you currently have open, or the file under your cursor in dired, using the system application for that file.

On Ubuntu 16.04, this function unfortunately does not work. It looks like the application tries to start (icon visible in the dash) and then dies.

After some debugging I discovered that on Ubuntu, the system tool xdg-open (which is invoked by crux) starts gvfs-open which on its part does not wait for the app to finish, but returns immediately.

The internal Emacs function start-process used by crux-open-with already runs the child process asynchronously. The early return of gvfs-open results in the sub-process closing, which means your app never gets off the ground.

To fix this, we have to get xdg-open to work in generic (i.e. non-Gnome / GVFS) mode and thus to wait for the app it spawns to return. We do this by setting the environment variable XDG_CURRENT_DESKTOP to X-Generic.

Fortunately, we can do this fairly neatly by wrapping the crux-open-with invocation in a let where we temporarily change the environment.

In my case it looks like this:

(use-package crux
  :config
  (defun cpb-crux-open-with (arg)
    (interactive "P")
    (let ((process-environment (cons "XDG_CURRENT_DESKTOP=X-Generic" process-environment)))
      (crux-open-with arg)))
  (global-set-key (kbd "C-c o") 'cpb-crux-open-with))

If you don’t also use use-package (you really should!) just copy out the part after :config.

This has been tested to work on Ubuntu 16.04 in both Unity and Gnome Shell 3.18.

Updates

  • 2017-04-18: Fixed environment variable + tested in Gnome Shell.

Recursive text search in project without projectile

If you’re not using projectile, but you would like to be able to perform interactive, search-as-you-type, recursive searches through the current project, this is pretty easy to do if you have counsel and find-file-in-project installed.

Add the following Emacs Lisp to your init:

(defun cpb/counsel-ag-in-project ()
  "Use `ffip' and `counsel-ag' for quick project-wide text searching."
  (interactive)
  (let ((project-root (ffip-project-root)))
    ;; if ffip could not find project-root, it will already have
    ;; shown an error message. We only have to check for non-nil.
    (if project-root
        (counsel-ag nil project-root nil
                    (format "Search in PRJ %s" project-root)))))

(global-set-key (kbd "C-c s") 'cpb/counsel-ag-in-project)

This function, which I have bound to ctrl-c s, finds the project root using find-file-in-project's function for this, and then invokes the handy counsel-ag on its result.

From org file with local bibtex to LaTeX and PDF

vxlabs-emacs-org-ref-pdf-example.png
Screenshot of orgmode source, PDF preview on the right, interactive citation selection in the minibuffer. Click for full resolution.

I have (co-)written a few LaTeX documents in my time.

However, as I have been writing my life and lab notes and many of my technical blog posts in Emacs orgmode for the past few years, I wanted to see how one would go about using BiBTeX references in orgmode files (using John Kitchin’s org-ref package) such that they would render correctly in orgmode’s export LaTeX and PDF.

My use case is of course slightly different than the norm: I maintain my master bibliography using the wonderful Zotero software, and I prefer exporting document-specific BiBTeX files from that main database.

This post shows you how to do it.

Pre-requisites

  • Make sure that latexmk is installed on your system. This is one of the easiest ways to make sure that pdflatex and bibtex are executed in the correct sequence, and a sufficient number of times.
  • With M-x package-install RET org-ref install the org-ref package. This will enable interactive selection and insertion of bibtex references from your local bib file.
  • You should also have the ox-latex Emacs package installed. This is what gives orgmode its LaTeX output powers.

Emacs setup

In your init.el somewhere, use the following code to ensure that the orgmode LaTeX exporter invokes latexmk in the correct way:

(setq org-latex-pdf-process
    '("latexmk -pdflatex='pdflatex -interaction nonstopmode' -pdf -bibtex -f %f"))

A minimal but complete orgmode example

The usual org-ref workflow is that you use it to manage a single main BiBTeX file, which is configured in your init.el. As mentioned above, I prefer using small document-specific BiBTeX files.

This is what the first three lines sets up using an Emacs file variable. Note that the local bib file is specified without quotes of any kind. Also, because there’s no path specified, it lives together with the orgmode document itself, which I prefer.

# Local Variables:
# org-ref-default-bibliography: local-bibtex-file.bib
# End:

After this, I have two LaTeX-specific lines ensuring that the output PDF looks prettier than default. You can thank me later.

The third LATEX_HEADER line imports the natbib package, which means I have extra cite commands to differentiate between for example textual citing (the first example) and parenthetical citing (the second example).

#+LATEX_CLASS_OPTIONS: [a4paper, 11pt, colorlinks=true, citecolor=., linkcolor=black, urlcolor=black]
#+LATEX_HEADER: \usepackage{fourier}
#+LATEX_HEADER: \usepackage{natbib}

#+TITLE: Example orgmode to LaTeX

* Introduction

  This is that really difficult intro spection, as was also
  demonstrated by cite:huang_identification_2016.

* Conclusions

  This is the even more difficult final section. We will now use a
  parenthetical citation citep:meyer_four-level_2012.

Finally, very importantly, I have the orgmode orgmode markup to select a suitable bibliography style, and again to specify the bib file. This will get automatically translated to the suitable LaTeX or even HTML versions.

Putting it all together

With something like the example above in your orgmode file, and a suitable bib-file, you can do C-c C-e l o to export to LaTeX, execute the latexmk command we specified above, and view the PDF.

In the screenshot at the start of this post, you can see what this looks like on my setup. I am using PDFTools for viewing the PDF inside of Emacs (this gets updated automatically whenever I rebuild the PDF).

You can also see the ivy-based completion as I’m in the process of interactively inserting a new citation into the document.

The default helm-based completion is more attractive, but it takes over the whole display, which is why I made use of ivy for instruction’s sake.

mu4e 0.9.18: E-Mailing with Emacs now even better.

In this post I talk about three great new features in the latest release of mu4e, an email programme that runs in Emacs. I also show my mu4e configuration as an example to others who would like a similar setup.

mu4e-0.9.18-screenshot.png
mu4e 0.9.18 screenshot showing selected context (bottom right of main) and visual-line-mode (long lines) which is activated by the format=flowed support.

After recently discovering that plaintext format=flowed in Thunderbird works only partially, it was time to check back in on mu4e after my previous happy stint using it.

I was pleasantly surprised to see that mu4e development had been quite active, and that this Emacs mail user agent had acquired a number of highly useful new features.

Three features I find particularly interesting are:

Full and deterministic format=flowed support

As I explained in another blog post, format=flowed is a really clever extension to old-school plain text emails that enables any receiving mail app that supports the standard to reflow emails for better display on smaller or larger displays.

mu4e now has first-class and deterministic support of the format=flowed feature.

I do understand that HTML has won the email format war.

However, HTML formatting gives users entirely too much freedom when writing their emails, resulting in misguided typesetting and other unnecessary visual elements, analogous to and just as irritating as chart junk, that regularly damage sensitive eyes.

This is why I prefer writing in plain text. With reliable format=flowed support, mu4e enables me to focus on the content whilst fixing plain-text email’s only real flaw: Reflowing on mobile devices.

Compose emails in a separate window

This, just like anything else, one could previously implement by oneself in emacs-lisp. However, it’s now a welcome core feature activated via the mu4e-compose-in-new-frame variable.

When you press C to compose a new email, a separate frame (this is what Emacs calls a window) is opened. After sending the email with C-c C-c the frame is automatically closed.

Contexts

In my previous setup, I had made a bunch of emacs-lisp functions to switch between different email identities (from addresses).

This worked, but it was not nearly as convenient and well thought-out as mu4e’s new contexts.

I now have a separate context defined for each email identity. I can switch between them using ; followed by the self-configured shortcut character.

They also activate automatically based on a configurable match function, for example examining the destination address of an email I am replying to. My configuration example below shows this in a little more detail.

My configuration

Because of these new features and a number of other changes, I put together this new configuration from scratch based on mu4e’s excellent documentation and examples.

I am still using `offlineimap` to synchronise my email with the IMAP server (fastmail in my case), but I’m currently sending email using the Emacs smtpmail package.

Here is a slightly shortened and simplified version of my configuration:

;; I installed mu4e from the 0.9.18 tarball
;; this is where make install put the emacs bits
(add-to-list 'load-path "/usr/local/share/emacs/site-lisp/mu4e")

(require 'mu4e)

;; I want my format=flowed thank you very much
;; mu4e sets up visual-line-mode and also fill (M-q) to do the right thing
;; each paragraph is a single long line; at sending, emacs will add the
;; special line continuation characters.
(setq mu4e-compose-format-flowed t)

;; every new email composition gets its own frame! (window)
(setq mu4e-compose-in-new-frame t)

;; give me ISO(ish) format date-time stamps in the header list
(setq mu4e-headers-date-format "%Y-%m-%d %H:%M")

;; show full addresses in view message (instead of just names)
;; toggle per name with M-RET
(setq mu4e-view-show-addresses 't)

;; path to our Maildir directory
(setq mu4e-maildir "~/Maildir")

;; the next are relative to `mu4e-maildir'
;; instead of strings, they can be functions too, see
;; their docstring or the chapter 'Dynamic folders'
(setq mu4e-sent-folder   "/Sent"
      mu4e-drafts-folder "/Drafts"
      mu4e-trash-folder  "/Trash")

;; the maildirs you use frequently; access them with 'j' ('jump')
(setq   mu4e-maildir-shortcuts
    '(("/Archive"     . ?a)
      ("/INBOX"       . ?i)
      ("/Sent"        . ?s)))

;; the list of all of my e-mail addresses
(setq mu4e-user-mail-address-list '("me@home.com"
                                    "me@work.com"
                                    "me@org.org"))

;; the headers to show in the headers list -- a pair of a field
;; and its width, with `nil' meaning 'unlimited'
;; (better only use that for the last field.
;; These are the defaults:
(setq mu4e-headers-fields
    '( (:date          .  25)    ;; alternatively, use :human-date
       (:flags         .   6)
       (:from          .  22)
       (:subject       .  nil))) ;; alternatively, use :thread-subject

;; program to get mail; alternatives are 'fetchmail', 'getmail'
;; isync or your own shellscript. called when 'U' is pressed in
;; main view.

;; If you get your mail without an explicit command,
;; use "true" for the command (this is the default)
;; when I press U in the main view, or C-c C-u elsewhere,
;; this command is called, followed by the mu indexer
(setq mu4e-get-mail-command "offlineimap")

;; not using smtp-async yet
;; some of these variables will get overridden by the contexts
(setq
 send-mail-function 'smtpmail-send-it
 message-send-mail-function 'smtpmail-send-it
 smtpmail-smtp-server "smtp.fastmail.com"
 smtpmail-smtp-service 465
 smtpmail-stream-type 'ssl
 )

;; don't keep message buffers around
(setq message-kill-buffer-on-exit t)

;; here come the contexts
;; I have about 5 of these, chopped down to 2 for demonstration purposes
;; each context can set any number of variables (see :vars)
;; for example below here I'm using two different SMTP servers depending on identity
(setq mu4e-contexts
      `( ,(make-mu4e-context
           :name "c me@home.com"
           :enter-func (lambda () (mu4e-message "Enter me@home.com context"))
           :leave-func (lambda () (mu4e-message "Leave me@home.com context"))
           ;; we match based on the contact-fields of the message (that we are replying to)
           ;; https://www.djcbsoftware.nl/code/mu/mu4e/What-are-contexts.html#What-are-contexts
           :match-func (lambda (msg)
                         (when msg 
                           (mu4e-message-contact-field-matches msg 
                                                               :to "me@home.com")))
           :vars '( ( user-mail-address      . "me@home.com"  )
                    ( user-full-name         . "Charl P. Botha" )
                    ( smtpmail-smtp-server   . "smtp.fastmail.com" )
                    ( mu4e-compose-signature .
                                             (concat
                                              "dr. charl p. botha\n"
                                              "http://charlbotha.com/\n"))))

         ,(make-mu4e-context
           :name "s me@org.org"
           :enter-func (lambda () (mu4e-message "Enter me@org.org context"))
           ;; no leave-func
           ;; we match based on the contact-fields of the message
           :match-func (lambda (msg)
                         (when msg 
                           (mu4e-message-contact-field-matches msg 
                                                               :to "me@org.org")))
           :vars '( ( user-mail-address       . "me@org.org" )
                    ( user-full-name          . "Charl P. Botha" )
                    ( smtpmail-smtp-server    . "smtp.gmail.com" )
                    ( mu4e-compose-signature  .
                                              (concat
                                               "Dr. Charl P. Botha\n"
                                               "Science Officer :: Stone Three Venture Technology\n"
                                               "https://www.stonethree.com\n"))))         

         ))

;; start with the first (default) context; 
(setq mu4e-context-policy 'pick-first)

;; compose with the current context if no context matches;
(setq mu4e-compose-context-policy nil)

;; these are the standard mu4e search bookmarks
;; I've only added the fourth one to pull up flagged emails in my inbox
;; I sometimes use this to shortlist emails I need to get around to ASAP
(setq mu4e-bookmarks
  `( ,(make-mu4e-bookmark
       :name  "Unread messages"
       :query "flag:unread AND NOT flag:trashed"
       :key ?u)
     ,(make-mu4e-bookmark
       :name "Today's messages"
       :query "date:today..now"
       :key ?t)
     ,(make-mu4e-bookmark
       :name "Last 7 days"
       :query "date:7d..now"
       :key ?w)
     ,(make-mu4e-bookmark
       :name "Flagged in INBOX"
       :query "maildir:\"/INBOX\" and flag:flagged"
       :key ?f)))

Conclusion

This is currently running smoothly on two of my Linux workstations as well as my early 2015 retina MacBook Pro.

The general setup procedure is to install offlineimap using pip2 install --user offlineimap (the offlineimap configuration file hasn’t changed much since the previous post), run offlineimap for the first time, then build and install mu4e from its tarball, and finally to run mu index to initialise the index.

After this, your mail, and plain-text nirvana, is a simple M-x mu4e away.

Date-sorted interactive recursive search with ivy, counsel and ag

A core part of my note-taking strategy is a growing collection of Emacs Org mode files containing my monthly lab journals, project summaries and various technical documents.

Usually I rely on Projectile’s Helm integration to perform recursive searches with The Silver Searcher (ag) using helm-projectile-ag (trigged by the completely muscle-memorised C-c p s s) through all of the org files in my notes hierarchy. Recursive regular expression search results are shown interactively as you type. Wonderful!

However, recently I found that I really missed the ability to see my search results sorted by last modified date, so that the most recently touched files would show up first. For a chronological notes database, this sorting makes the most sense.

A few frustrating hours later, interspersed with bouts of seriously pondering the exact nature of my relationship with Emacs, I managed to attain the desired result using the Emacs packages ivy and counsel.

Although I’m sticking with Helm for the rest, in this case the ivy API made it much simpler to do what I needed to do. It is understandable that this is not a straight-forward endeavour, as ag is heavily parallelised and furthermore is handled asynchronously by both helm and ivy / counsel. Sorting the results slows matters down, but for this specific case the slowdown is a reasonable price to pay.

Read on for more detail.

Video demo of end result

If you follow the instruction below, you too will be able to achieve the following feats of recall:

Search results are returned as I type, as is usually the case with counsel-ag and helm-ag. However, they are now sorted by file modified date, last modified file at the top. This usually means I find what I need faster.

Show me the code

The ivy-read function (this is ivy’s single API entrypoint) has two required parameters, one of which is the collection. This is a function returning the list of items that the user wants to narrow down, or the list itself.

Because we need to tell ivy to sort the results by associating a sorting function with a specific collection function, we wrap the supplied counsel-ag-function collection function.

We also supply a new sorting function that will take two of the ag-generated search results, each of the form relative_filename:line_number:line_contents, derive the filenames from that, determine the modified timestamps of both the files and then return t or nil depending on which file is newer.

We finally (for this first snippet) associate the sorting function with our wrapped collection function.

(defun cpb/ag-collection (string)
  "Search for pattern STRING using ag.
We only have this as a separate function so we can assoc with sort function."
  ;; this will use counsel--async-command to run asynchronously
  (counsel-ag-function string counsel-ag-base-command "")) 

;; modified from https://github.com/abo-abo/swiper/wiki/Sort-files-by-mtime
;; * directory to counsel--git-grep-dir: that's where AG is working
;; * removed directory logic: ag returns files
;; * parsed out filename from ag results
(defun eh-ivy-sort-file-by-mtime (x y)
  "Determine if AG sort result X is newer than Y."
  (let* ((x (concat counsel--git-grep-dir (car (split-string x ":"))))
         (y (concat counsel--git-grep-dir (car (split-string y ":"))))
         (x-mtime (nth 5 (file-attributes x)))
         (y-mtime (nth 5 (file-attributes y))))
    (time-less-p y-mtime x-mtime)))

;; ivy uses the ivy-sort-functions-alist to look up suitable sort
;; functions for any given collection function
;; we add a cons cell specifying eh-ivy-sort-file-by-mtime as the sort
;; function to go with our collection function
(add-to-list 'ivy-sort-functions-alist
             '(cpb/ag-collection . eh-ivy-sort-file-by-mtime))

In the second and final snippet, we create a new version of the counsel-ag function where we specify our own collection function, and we supply a truthful :sort argument.

(defun cpb/counsel-ag (&optional initial-input initial-directory extra-ag-args ag-prompt)
  "Grep for a string in the current directory using ag.
INITIAL-INPUT can be given as the initial minibuffer input.
INITIAL-DIRECTORY, if non-nil, is used as the root directory for search.
EXTRA-AG-ARGS string, if non-nil, is appended to `counsel-ag-base-command'.
AG-PROMPT, if non-nil, is passed as `ivy-read' prompt argument.

Modified by cpbotha: Sort results last modified file first."
  (interactive)
  (when current-prefix-arg
    (setq initial-directory
          (or initial-directory
              (read-directory-name (concat
                                    (car (split-string counsel-ag-base-command))
                                    " in directory: "))))
    (setq extra-ag-args
          (or extra-ag-args
              (let* ((pos (cl-position ?  counsel-ag-base-command))
                     (command (substring-no-properties counsel-ag-base-command 0 pos))
                     (ag-args (replace-regexp-in-string
                               "%s" "" (substring-no-properties counsel-ag-base-command pos))))
                (read-string (format "(%s) args:" command) ag-args)))))
  (ivy-set-prompt 'cpb/counsel-ag counsel-prompt-function)
  (setq counsel--git-grep-dir (or initial-directory default-directory))
  (ivy-read (or ag-prompt (car (split-string counsel-ag-base-command)))
            #'cpb/ag-collection
            :initial-input initial-input
            :dynamic-collection t
            ;; yes, we want to sort the results
            :sort t
            :keymap counsel-ag-map
            :history 'counsel-git-grep-history
            :action #'counsel-git-grep-action
            :unwind (lambda ()
                      (counsel-delete-process)
                      (swiper--cleanup))
            :caller 'cpb/counsel-ag))

Closing

You can invoke cpb/counsel-ag interactively to search from the current directory, or you can point it at a directory of your choosing by passing in the initial-directory argument.

In my Emacs initialisation, I have bound another wrapper function, bound to a global shortcut key, that invokes cpb/counsel-ag on my notes directory, so I can instantly search through my notes in reverse chronological order.

It looks like this:

;; wrapper function to invoke chrono-sorted ag search on my notes dir
(defun cpb/counsel-ag-notes ()
  "Search my notes file hierarchy using counsel and ivy."
  (interactive)
  (cpb/counsel-ag nil "~/Dropbox/notes/pkb4000/" nil "PKB4000: "))

;; I invoke search by pressing "Ctrl-C 4"
(global-set-key (kbd "C-c 4") 'cpb/counsel-ag-notes)

I believe that my relationship with Emacs is safe for now.

Updates

  • 2017-02-28: Fixed (interactive) invocation in wrapper function.
  • 2017-02-27: Updated cpb/ag-collection for latest counsel version. Added example of wrapper function and global shortcut.