Interactive programming with Fennel Lua Lisp, Emacs and Lisp Game Jam winner EXO_encounter 667

Phil Hagelberg recently won the Lisp Game Jam 2018 with his entry EXO_encounter 667.

What I found most interesting however, was his interactive programming setup.

He programmed his game in (and contributed new features to) a Lisp to Lua compiler called Fennel, and used the game programming library Löve.

With Emacs and some Lua thread magic, he was able to perform runtime changes and introspection to his live running game project. (See below for a demo!)

Based on past experience developing visualization and image processing algorithms, I learned how useful this sort of interactive / runtime programming could be.

Hagelberg wrote up his experience in three great blog posts:

… and he has made the full source code to EXO_encounter 667 available on gitlab, so I had to try the interactive programming setup out for myself.

Due to his great write-ups, this was surprisingly easy.

Below you’ll find a short screencast of the setup in action, the steps I took to get everything running, and finally some information on how he put the interactive programming parts of the game together.

Demonstration

Here is a short video demonstrating a live programming session:

Quickstart

Here are the steps I followed to get everything up and running:

Install löve, lua and fennel

brew install caskroom/cask/love
brew install lua
luarocks install --server=http://luarocks.org/dev fennel

Install the Emacs fennel-mode

This important piece of code is also by Hagelberg.

Check out fennel-mode where you usually work with github and gitlab checkouts:

mkdir ~/build && cd ~/build
git clone https://gitlab.com/technomancy/fennel-mode.git

Evaluate the following two lines in Emacs using for example M-x eval-region:

(autoload 'fennel-mode (expand-file-name "~/build/fennel-mode/fennel-mode") nil t)
(add-to-list 'auto-mode-alist '("\\.fnl\\'" . fennel-mode))

Get and start playing with EXO_encounter 667

git clone https://gitlab.com/technomancy/exo-encounter-667.git

Start by opening wrap.fnl in the root directory.

Then, as per the instructions, start the Fennel repl using C-u M-x run-lisp. This will ask you which lisp to use. Replace the default fennel --repl with love . (that’s love followed by space and a period)

At this point you will get a repl via which you can enter fennel commands. You can also edit any of the top-level fennel files, and type C-c C-k to reload the whole file, and watch the game change before your eyes.

More detail on how the interactive programming parts work

When you start love . from within Emacs fennel-mode, this runs the game, but starts an extra Lua thread to listen for input from Emacs.

(see the Interactive Development section in the blog post titled “in which a game jam is recounted further“)

Looking at the source, main.lua bootstraps fennel and loads in wrap.fnl which contains the familiar love.load, love.draw and love.update callbacks.

In love.load, it starts the repl, which is loaded from lib.stdio, which is where the extra listener thread is started up.

Emacs fennel-mode does the rest. Once you’ve done run-lisp with love ., you can use all the dynamic commands described on the fennel-mode gitlab page.

Up and running with ECL and Emacs SLIME in four easy steps

ECL is of course Embeddable Common Lisp, a small but quite complete Common Lisp implementation that can be easily embedded in your C applications to act as dynamic extension language, but it is also a fine stand-alone implementation. It includes an interpreter and a compiler, and can even produce compact binaries. Read this blog post by Chris Kohlhepp for a brief and practical overview.

SLIME is the Superior Lisp Interaction Mode for Emacs, a great Emacs environment within which one can write code for a number of Common Lisp implementations.

I had to consult a number of different sources to get SLIME and ECL working on OSX (El Capitan) using homebrew, so I’ve decided to put it all together in this short howto.

(This post assumes you have homebrew and Emacs installed. For Emacs on OSX, I usually just do brew install --cocoa --srgb emacs. Thanks to user Aidenn0 on reddit for the reminder to mention the emacs install!)

Step 1: Install XCode command-line tools

I use this MacBook for development with XCode, so I would not have thought this step necessary. Without it however, your SLIME will break at startup during the ECL compile of the SWANK slime server due to Clang not being able to find the gmp.h file. The error looks something like this:

.slime/fasl/2016-03-04/ecl-16.0.0-darwin-x86_64/packages.c:5:
In file included from /usr/local/Cellar/ecl/16.0.0/include/ecl/ecl-cmp.h:22:
In file included from /usr/local/Cellar/ecl/16.0.0/include/ecl/ecl.h:35:
/usr/local/Cellar/ecl/16.0.0/include/ecl/config.h:56:10: fatal error: 'gmp.h' file not found
#include "gmp.h"
         ^
1 error generated.

This is all the more infuriating, because homebrew does install the GMP header file in /usr/local/include as a dependency of ECL (see later).

Furthermore, at first start SLIME will hence display an infinite number of polling messages in your minibuffer, as it can’t connect to the SWANK server that ECL should have compiled.

If you install the xcode command-line tools by typing the following in a term and following the prompts:

xcode-select --install

… the clang compiler will henceforth also search in /usr/local/include, avoiding the error above when you startup slime for the first time.

Step 2: Install ECL

Install ECL using homebrew. This will also pull down and install gmp:

brew install ecl

Step 3: Install and configure SLIME

In Emacs, install the SLIME package by doing M-x package-install RET slime RET

After having done this, add the following two lines to your Emacs init.el:

(setq inferior-lisp-program "ecl")
(setq slime-contribs '(slime-fancy))

Without that second line, starting slime will only get you an *inferior-lisp* buffer, and not the *slime-repl ECL* that you deserve!

You can either eval those two lines and then continue with the next step, or restart Emacs if you’re the restarting type. (too much Windows can do that to you)

Step 4: Fire up the SLIME!

In Emacs, do M-x slime RET, at which point you should find yourself in a lovely *slime-repl ECL* buffer, where you can talk to the inferior Lisp process (it’s being modest, because it’s actually superior) with syntax completion, real-time code documentation and other niceties.

You can now enter some Lisp code in a new .lisp buffer, for example something like this:

(defun harmonic_tr (n accum)
  "tail-recursive harmonic function - cpb"
  (if (<= n 1) (+ accum 1)
      (harmonic_tr (- n 1) (+ accum (/ 1.0 n)))))

(time (loop for i from 1 to 10 collect (harmonic_tr 1000000 0)))

You can now put your cursor anywhere on the harmonic_tr function, and press C-c C-c (M-x slime-compile-defun), and then on the time loop (hehe), to have it calculate the millionth harmonic number ten times. As you do this, various outputs will appear in the REPL. You can also switch to the REPL, and continue experimenting with your newly defined functions over there.

Why the harmonic number?

This specific function is due to some completely-for-fun micro-benchmarks I did in 2005 with OCaml, Octave, Python, Pyrex (what later became Cython), and C to compare with Lush. Then I used the harmonic number, because that’s what Lush had on its website as a demonstration of Lush’s looping speed.

Fast forward 10 years, and on this early 2015 MacBook Pro with a 2.7GHz i5 and Embeddable Common Lisp (ECL) 16.0.0 I get 0.0736 seconds per call to harmonic_tr(1000000), averaged over 10 calls. The loop version does marginally better.

For comparison, the straight Python 3.5.1 loop code does 0.1187, a tail-recursive version in Guile 2.0.11 Scheme does 0.0904 seconds (ask me in the comments if you’d like to see this), the clang -O3 loop code in C does 0.0033, and PyPy 2.6.0 on the Python code does 0.0024 seconds.

This micro-benchmark is fun, but do remember that when you’re embedding a dynamic language, you are usually focusing not only on raw performance, but on other matters like the language’s expressivity and, more importantly, on the best impedance match for you extension writer and their domain.

Modify Emacs Deft for recursive directory search

Update 2014-11-18 I’ve forked the original Deft, added this recursive directory listing feature as well as support for multiple different file extensions, and pushed it all to github as deft-turbo!

Deft is a neat Emacs mode for the Notational Velocity-inspired searching, browsing and editing of a directory of text files. In short, this means that simply start typing, and Deft finds the note that you were looking for. It supports straight text searching and regular expression searching, almost like my own baby nvpy.

When your text files are of a structured type that Emacs supports, such as Markdown, orgmode or something else, you can attain note-taking nirvana.

It looks like this:

Emacs Deft on my setup. I had to carefully pick a search string so you wouldn’t see all of my top secret notes! (Yes, you’re seeing notes from two different nested directories!)

However, I keep my personal text notes in a nested directory structure. I have general notes at the top level, daily journals in their own subdirectory, and projects each in their own subdirectory. Deft unfortunately does not support this out of the box; it only searches in the top-level directory you configure it with.

Fortunately I ran into this bit of example code in the Emacs documentation of all places. After a few small tweaks, it looked like this:

(defun deft-find-all-files (directory)
"List the deft-extension files in DIRECTORY and in its sub-directories."
;; cpbotha found this on http://www.gnu.org/software/emacs/manual/html_node/eintr/Files-List.html
;; and adapted for deft
(let (el-files-list
(current-directory-list
(directory-files-and-attributes directory t)))
;; while we are in the current directory
(while current-directory-list
(cond
;; check to see whether filename ends in `.deft-extension’
;; and if so, append its name to a list.
((equal (concat "\." deft-extension) (substring (car (car current-directory-list)) -3))
(setq el-files-list
(cons (car (car current-directory-list)) el-files-list)))
;; check whether filename is that of a directory
((eq t (car (cdr (car current-directory-list))))
;; decide whether to skip or recurse
(if
(equal "."
(substring (car (car current-directory-list)) -1))
;; then do nothing since filename is that of
;; current directory or parent, "." or ".."
()
;; else descend into the directory and repeat the process
(setq el-files-list
(append
(deft-find-all-files
(car (car current-directory-list)))
el-files-list)))))
;; move to the next filename in the list; this also
;; shortens the list so the while loop eventually comes to an end
(setq current-directory-list (cdr current-directory-list)))
;; return the filenames
el-files-list))

Note that deft-find-all-files() takes one argument now, instead of none.

If you copy that function into your deft.l, replacing the deft-find-all-files() that’s already there, and modify deft-cache-initialize() so that the first code line reads:

(setq deft-all-files (deft-find-all-files deft-directory)) ; List all files

… your Deft can support nested note directories too!

(I have sent the author this snippet of code (he’s not on github, so no pull request) – we’ll see what happens)