Date-sorted interactive recursive search with ivy, counsel and ag

A core part of my note-taking strategy is a growing collection of Emacs Org mode files containing my monthly lab journals, project summaries and various technical documents.

Usually I rely on Projectile’s Helm integration to perform recursive searches with The Silver Searcher (ag) using helm-projectile-ag (trigged by the completely muscle-memorised C-c p s s) through all of the org files in my notes hierarchy. Recursive regular expression search results are shown interactively as you type. Wonderful!

However, recently I found that I really missed the ability to see my search results sorted by last modified date, so that the most recently touched files would show up first. For a chronological notes database, this sorting makes the most sense.

A few frustrating hours later, interspersed with bouts of seriously pondering the exact nature of my relationship with Emacs, I managed to attain the desired result using the Emacs packages ivy and counsel.

Although I’m sticking with Helm for the rest, in this case the ivy API made it much simpler to do what I needed to do. It is understandable that this is not a straight-forward endeavour, as ag is heavily parallelised and furthermore is handled asynchronously by both helm and ivy / counsel. Sorting the results slows matters down, but for this specific case the slowdown is a reasonable price to pay.

Read on for more detail.

Video demo of end result

If you follow the instruction below, you too will be able to achieve the following feats of recall:

Search results are returned as I type, as is usually the case with counsel-ag and helm-ag. However, they are now sorted by file modified date, last modified file at the top. This usually means I find what I need faster.

Show me the code

The ivy-read function (this is ivy’s single API entrypoint) has two required parameters, one of which is the collection. This is a function returning the list of items that the user wants to narrow down, or the list itself.

Because we need to tell ivy to sort the results by associating a sorting function with a specific collection function, we wrap the supplied counsel-ag-function collection function.

We also supply a new sorting function that will take two of the ag-generated search results, each of the form relative_filename:line_number:line_contents, derive the filenames from that, determine the modified timestamps of both the files and then return t or nil depending on which file is newer.

We finally (for this first snippet) associate the sorting function with our wrapped collection function.

(defun cpb/ag-collection (string)
  "Search for pattern STRING using ag.
We only have this as a separate function so we can assoc with sort function."
  ;; this will use counsel--async-command to run asynchronously
  (counsel-ag-function string ""))

;; modified from https://github.com/abo-abo/swiper/wiki/Sort-files-by-mtime
;; * directory to counsel--git-grep-dir: that's where AG is working
;; * removed directory logic: ag returns files
;; * parsed out filename from ag results
(defun eh-ivy-sort-file-by-mtime (x y)
  "Determine if AG sort result X is newer than Y."
  (let* ((x (concat counsel--git-grep-dir (car (split-string x ":"))))
         (y (concat counsel--git-grep-dir (car (split-string y ":"))))
         (x-mtime (nth 5 (file-attributes x)))
         (y-mtime (nth 5 (file-attributes y))))
    (time-less-p y-mtime x-mtime)))

(add-to-list 'ivy-sort-functions-alist
             '(cpb/ag-collection . eh-ivy-sort-file-by-mtime))

In the second and final snippet, we create a new version of the counsel-ag function where we specify our own collection function, and we supply a truthful :sort argument.

(defun cpb/counsel-ag (&optional initial-input initial-directory
                                 extra-ag-args ag-prompt)
  "Grep for a string in the current directory using ag.
INITIAL-INPUT can be given as the initial minibuffer input.
INITIAL-DIRECTORY, if non-nil, is used as the root directory for search.
EXTRA-AG-ARGS string, if non-nil, is appended to `counsel-ag-base-command'.
AG-PROMPT, if non-nil, is passed as `ivy-read' prompt argument.

Modified by cpbotha: Sort results last modified file first."
  (interactive
   (list nil
         (when current-prefix-arg
           (read-directory-name (concat
                                 (car (split-string counsel-ag-base-command))
                                 " in directory: ")))))
  (ivy-set-prompt 'counsel-ag counsel-prompt-function)
  (setq counsel--git-grep-dir (or initial-directory default-directory))
  (ivy-read (or ag-prompt (car (split-string counsel-ag-base-command)))
            ;; specify collection function so we can assoc with sort function
            #'cpb/ag-collection
            :initial-input initial-input
            :dynamic-collection t
            ;; yes, we want to sort the results
            :sort t
            :keymap counsel-ag-map
            :history 'counsel-git-grep-history
            :action #'counsel-git-grep-action
            :unwind (lambda ()
                      (counsel-delete-process)
                      (swiper--cleanup))
            :caller 'cpb/counsel-ag))

Closing

You can invoke cpb/counsel-ag interactively to search from the current directory, or you can point it at a directory of your choosing by passing in the initial-directory argument.

In my Emacs initialisation, I have bound another wrapper function that invokes cpb/counsel-ag on my notes directory, so I can instantly search through my notes in reverse chronological order.

I believe that my relationship with Emacs is safe for now.

Step-by-step guide to C++ navigation and completion with Emacs and the Clang-based rtags

If you want C++ completion and navigation (jump to definition, jump to declaration, and so forth), there are several good options for Emacs. For a QtQuick / C++ project I’m working on, I needed the best Emacs has to offer.

This turned out to be the Clang-based rtags system.

Rtags is not the easiest of the options to get going, hence this short tutorial. I initially configured irony-mode, which is also Clang-based and was significantly easier to get going, but it soon started hanging on the completion of for example QStringList methods in my project. Because it also doesn’t support navigation, I decided to try rtags. So far, rtags has been working quite well on a Qt 5.x project of slightly under 200K lines of C++.

Pre-requisites

On Ubuntu 14.04, I tested this with both libclang-3.6-dev from the standard repos and with libclang-3.8-dev from the LLVM repos. CMake should also be installed.

On OSX 10.11 (El Capitan), homebrew took care of the dependencies for me (see next step).

Get and build rtags

From the documentation, simply clone the rtags repo and build it:

git clone --recursive https://github.com/Andersbakken/rtags.git
cd rtags
cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=1 .
make

Make a compilation database for your project

rtags needs a compilation database file, usually called compile_commands.json, to figure out what compile options were used to compile which source files. If your project is using cmake, you only have to add the -DCMAKE_EXPORT_COMPILE_COMMANDS=1 to the cmake invocation to generate that file.

In my case, and perhaps in yours, you have to use something else like Qt’s qmake system. In this case, you can use a tool like Build Ear (BEAR) or the Python-rewrite of scan-build. In both cases, you invoke the utility with your normal build command. In my Qt project, I do:

bear --append make -j16

As your project builds, bear will capture all of the compilation commands and put them in compile_commands.json. With my project, the compile_commands.json finds itself in the out-of-source (shadow) build directory.

(The first time, you have to do this after a clean, or a make clean, so all of the compile commands can be captured.)

BEAR is preferable, but on the latest OSX with its new system integrity protection (SIP) and on some linuxes, it won’t work and you’ll end up with an empty compilation database. In these cases, use the intercept-build command from the scan-build package linked above.

Start the rtags daemon RDM

You can do this from emacs, but due to better system process handling, I invoke this from a normal terminal:

cd /where/you/built/rtags/bin
nohup ./rdm &

When working with rtags, always make sure that RDM is running.

Tell RDM about your project using RC

rc is the rtags command one uses to communicate with the rtags daemon RDM.

Importantly, make sure that rc is in your path.

We only have to tell RDM once about the location of any new project. Do this as follows:

cd /your/project/source
rc -J /dir/containing/compile_commands.json/

Note that the -J parameter is the directory containing the compile_commands.json file.

Configure your Emacs

Install the Emacs rtags package:

M-x package-install RET rtags RET

Make sure that you also have the company package installed.

Next, add the following to your Emacs init.el:

;; ensure that we use only rtags checking
;; https://github.com/Andersbakken/rtags#optional-1
(defun setup-flycheck-rtags ()
  (interactive)
  (flycheck-select-checker 'rtags)
  ;; RTags creates more accurate overlays.
  (setq-local flycheck-highlighting-mode nil)
  (setq-local flycheck-check-syntax-automatically nil))

;; only run this if rtags is installed
(when (require 'rtags nil :noerror)
  ;; make sure you have company-mode installed
  (require 'company)
  (define-key c-mode-base-map (kbd "M-.")
    (function rtags-find-symbol-at-point))
  (define-key c-mode-base-map (kbd "M-,")
    (function rtags-find-references-at-point))
  ;; disable prelude's use of C-c r, as this is the rtags keyboard prefix
  (define-key prelude-mode-map (kbd "C-c r") nil)
  ;; install standard rtags keybindings. Do M-. on the symbol below to
  ;; jump to definition and see the keybindings.
  (rtags-enable-standard-keybindings)
  ;; comment this out if you don't have or don't use helm
  (setq rtags-use-helm t)
  ;; company completion setup
  (setq rtags-autostart-diagnostics t)
  (rtags-diagnostics)
  (setq rtags-completions-enabled t)
  (push 'company-rtags company-backends)
  (global-company-mode)
  (define-key c-mode-base-map (kbd "<C-tab>") (function company-complete))
  ;; use rtags flycheck mode -- clang warnings shown inline
  (require 'flycheck-rtags)
  ;; c-mode-common-hook is also called by c++-mode
  (add-hook 'c-mode-common-hook #'setup-flycheck-rtags))

You can either just eval this new code, or restart your emacs.

Using rtags

I mostly use M-. to jump to any symbol’s definition. Pressing M-. again will jump to and fro between definition and declaration. Use =M-,= to see all references to the symbol under the cursor. This is where Helm comes in especially useful.

Use C-c r [ and C-c r ] to jump backwards and forwards through the position stack that rtags builds up as you jump between symbols. You can use C-c r I to get a list of rtags-extracted symbols in the current file, although semantic-mode with helm (C-c h i) gives you slightly less accurate but more colourful results. I use both.

As you type, company-mode will usually interrupt after you’ve entered the first three characters of a symbol or member. I can’t wait that long, so I press C-tab to get a list of completions immediately.

As an additional bonus, you’ll see Clang warnings indicated in bright red (usually) in your code window!

As you’re working, remember to do builds with bear --append as shown above. I do this via M-x compile. rdm will pick up all changes and re-analyse files as you work!

Conclusions and other rambling

I am quite impressed by how well rtags works on complex C++ projects! It is indeed clang under the hood, but rtags has overcome the considerable challenge of exposing this to Emacs in a robust and usable fashion.

With this setup, I am able to take longer breaks from using QtCreator, which I don’t like very much. QtCreator has great potential, but unfortunately its Emacs keyboard emulation is sorely lacking (this could be forgiven) and as a general C++ IDE it still has a long way to go in the usability department. I remain optimistic that with competition like JetBrains (when I’m not in Emacs, I’m in something by JetBrains; their Emacs emulation is bearable!) out there, QtCreator will improve over time.

Feel free to join the discussion in the comments below or on the reddit topic!

Fixing the Cordova browser platform Access-Control-Allow-Origin error

When developing a mobile app using Cordova or PhoneGap, the browser target platform can really speed up your development. You could serve the HTML files directly using cordova serve, but the browser platform, while being almost as fast, is much closer to the Android / IOS environments your app will eventually find itself in. It also has access to the Cordova APIs.

My general workflow is that I keep cordova run browser running in a terminal window (this will initially start a new instance of chrome), and periodically run cordova prepare browser as I’m developing. This last step packages up the “app”, along with the cordova.js bits, for the browser platform.

However, if your app contacts an external API or server at any point, for example with jQuery.ajax(), you will be greeted by the following error:

XMLHttpRequest cannot load http://some.api.com/api/endpoint. No
'Access-Control-Allow-Origin' header is present on the requested
resource. Origin 'http://localhost:8000' is therefore not allowed
access.

In short, your chrome browser is (wisely) refusing to access that outside resource (the API or server) for security reasons. If your browser did not do this, an attacker could quite easily access web services using your identity!

Reading up about this issue in the context of Cordova / PhoneGap, you’ll find the common solution to be either to modify the server that you’re accessing to allow explicitly this sort of cross-origin access, or to setup a proxy if the former is not possible. Often the former is indeed not possible (in my case, the app is talking to a hardware router API), but the latter is just way too much unnecessary effort.

The easy (but good) fix

Recall that cordova starts up a new instance of chrome which is only used to access your app. The app is completely under your control, so the risk of foul play by a third party is significantly reduced. Instead of setting up proxies, or changing servers, we could just instruct this special instance of chrome to ignore its cross-origin rules!

It took me a while to track this down, so here you go: Edit browser.js in yourapp/platforms/browser/cordova/node_modules/cordova-serve/src and change just the chromeArgs line so it looks like the one in the snippet below (you’re just adding the --disable-web-security argument):

function getBrowser(target, dataDir) {
    dataDir = dataDir || 'temp_chrome_user_data_dir_for_cordova';

    var chromeArgs = ' --user-data-dir=/tmp/' + dataDir + ' --disable-web-security';

Now make 100% sure that all of your Cordova chrome instances are stopped. The next time you do cordova run browser, note that the chrome window that appears has a yellow bar warning you about web security:

chrome-disable-web-security-warning.jpg

Only the special chrome instances started up by Cordova for apps where you have applied the above fix will have web-security disabled. By all means take note of the yellow warning, but also do enjoy all of your Cordova app requests successfully contacting the outside world!

Up and running with ECL and Emacs SLIME in four easy steps

ECL is of course Embeddable Common Lisp, a small but quite complete Common Lisp implementation that can be easily embedded in your C applications to act as dynamic extension language, but it is also a fine stand-alone implementation. It includes an interpreter and a compiler, and can even produce compact binaries. Read this blog post by Chris Kohlhepp for a brief and practical overview.

SLIME is the Superior Lisp Interaction Mode for Emacs, a great Emacs environment within which one can write code for a number of Common Lisp implementations.

I had to consult a number of different sources to get SLIME and ECL working on OSX (El Capitan) using homebrew, so I’ve decided to put it all together in this short howto.

(This post assumes you have homebrew and Emacs installed. For Emacs on OSX, I usually just do brew install --cocoa --srgb emacs. Thanks to user Aidenn0 on reddit for the reminder to mention the emacs install!)

Step 1: Install XCode command-line tools

I use this MacBook for development with XCode, so I would not have thought this step necessary. Without it however, your SLIME will break at startup during the ECL compile of the SWANK slime server due to Clang not being able to find the gmp.h file. The error looks something like this:

.slime/fasl/2016-03-04/ecl-16.0.0-darwin-x86_64/packages.c:5:
In file included from /usr/local/Cellar/ecl/16.0.0/include/ecl/ecl-cmp.h:22:
In file included from /usr/local/Cellar/ecl/16.0.0/include/ecl/ecl.h:35:
/usr/local/Cellar/ecl/16.0.0/include/ecl/config.h:56:10: fatal error: 'gmp.h' file not found
#include "gmp.h"
         ^
1 error generated.

This is all the more infuriating, because homebrew does install the GMP header file in /usr/local/include as a dependency of ECL (see later).

Furthermore, at first start SLIME will hence display an infinite number of polling messages in your minibuffer, as it can’t connect to the SWANK server that ECL should have compiled.

If you install the xcode command-line tools by typing the following in a term and following the prompts:

xcode-select --install

… the clang compiler will henceforth also search in /usr/local/include, avoiding the error above when you startup slime for the first time.

Step 2: Install ECL

Install ECL using homebrew. This will also pull down and install gmp:

brew install ecl

Step 3: Install and configure SLIME

In Emacs, install the SLIME package by doing M-x package-install RET slime RET

After having done this, add the following two lines to your Emacs init.el:

(setq inferior-lisp-program "ecl")
(setq slime-contribs '(slime-fancy))

Without that second line, starting slime will only get you an *inferior-lisp* buffer, and not the *slime-repl ECL* that you deserve!

You can either eval those two lines and then continue with the next step, or restart Emacs if you’re the restarting type. (too much Windows can do that to you)

Step 4: Fire up the SLIME!

In Emacs, do M-x slime RET, at which point you should find yourself in a lovely *slime-repl ECL* buffer, where you can talk to the inferior Lisp process (it’s being modest, because it’s actually superior) with syntax completion, real-time code documentation and other niceties.

You can now enter some Lisp code in a new .lisp buffer, for example something like this:

(defun harmonic_tr (n accum)
  "tail-recursive harmonic function - cpb"
  (if (<= n 1) (+ accum 1)
      (harmonic_tr (- n 1) (+ accum (/ 1.0 n)))))

(time (loop for i from 1 to 10 collect (harmonic_tr 1000000 0)))

You can now put your cursor anywhere on the harmonic_tr function, and press C-c C-c (M-x slime-compile-defun), and then on the time loop (hehe), to have it calculate the millionth harmonic number ten times. As you do this, various outputs will appear in the REPL. You can also switch to the REPL, and continue experimenting with your newly defined functions over there.

Why the harmonic number?

This specific function is due to some completely-for-fun micro-benchmarks I did in 2005 with OCaml, Octave, Python, Pyrex (what later became Cython), and C to compare with Lush. Then I used the harmonic number, because that’s what Lush had on its website as a demonstration of Lush’s looping speed.

Fast forward 10 years, and on this early 2015 MacBook Pro with a 2.7GHz i5 and Embeddable Common Lisp (ECL) 16.0.0 I get 0.0736 seconds per call to harmonic_tr(1000000), averaged over 10 calls. The loop version does marginally better.

For comparison, the straight Python 3.5.1 loop code does 0.1187, a tail-recursive version in Guile 2.0.11 Scheme does 0.0904 seconds (ask me in the comments if you’d like to see this), the clang -O3 loop code in C does 0.0033, and PyPy 2.6.0 on the Python code does 0.0024 seconds.

This micro-benchmark is fun, but do remember that when you’re embedding a dynamic language, you are usually focusing not only on raw performance, but on other matters like the language’s expressivity and, more importantly, on the best impedance match for you extension writer and their domain.

gunicorn as your Django development server

Usually when I’m working on a Django project, I use the built-in runserver command. For deployment, we mostly use nginx and uwsgi in front of Django.

However, for a new side project I needed a local Django setup with a faster debug server. To keep things simple, I was looking for a pure Python solution that could serve static files, and did automatic reloading on source change, just like runserver.

I know that gunicorn is often used for deployment behind a reverse proxy and static file server like nginx, but I was curious whether gunicorn could also act as Django development server, and also serve up static files without having to run collectstatic (i.e. serving the static files directly from their development locations).

The answer is: Yes. With DJ-Static serving static files, gunicorn turns out to be a fantastic development server for Django. Here’s how to get it going:

Install gunicorn and DJ-Static in your Django virtual environment:

workon my_ve # I use virtualenvwrapper, hence workon to activate
pip install gunicorn dj-static

Add dj_static to INSTALLED_APPS in your settings.

In your wsgi.py, wrap your WSGI application in a Cling object. The two relevant lines are:

from dj_static import Cling
application = Cling(get_wsgi_application())

Now you can start your Django development project as follows:

gunicorn -w 6 --access-logfile - --reload your_project.wsgi:application

This will start six worker processes (tune for your setup), logs access to stderr, and reloads when it detects changes to the source. Welcome to your shiny, new and slightly faster Django development server!