Creating a Django migration for a GiST / GIN index with a special index operator.

In order to add a GiST index on a Postgres database that could be used to accelerate trigram matches using the pg_trgm module and the special gist_trgm_ops operator, I had to code up a special Django Index

Django will hopefully soon support custom index operators, but if you need the functionality right now, this example will do the trick.

The special GiST index class looks like this:

from django.contrib.postgres.indexes import GistIndex

class GistIndexTrgrmOps(GistIndex):
    def create_sql(self, model, schema_editor):
        # - this Statement is instantiated by the _create_index_sql()
        #   method of django.db.backends.base.schema.BaseDatabaseSchemaEditor.
        #   using sql_create_index template from
        #   django.db.backends.postgresql.schema.DatabaseSchemaEditor
        # - the template has original value:
        #   "CREATE INDEX %(name)s ON %(table)s%(using)s (%(columns)s)%(extra)s"
        statement = super().create_sql(model, schema_editor)
        # - however, we want to use a GIST index to accelerate trigram
        #   matching, so we want to add the gist_trgm_ops index operator
        #   class
        # - so we replace the template with:
        #   "CREATE INDEX %(name)s ON %(table)s%(using)s (%(columns)s gist_trgrm_ops)%(extra)s"
        statement.template =\
            "CREATE INDEX %(name)s ON %(table)s%(using)s (%(columns)s gist_trgm_ops)%(extra)s"

        return statement

Which you can then use in your model class like this:

class YourModel(models.Model):
    some_field = models.TextField(...)

    class Meta:
        indexes = [
            GistIndexTrgrmOps(fields=['some_field'])
        ]

The migration will then generate SQL (use manage.py sqlmigrate to inspect the SQL) which looks like this:

CREATE INDEX "app_somefield_139a28_gist" ON "app_yourmodel" USING gist ("some_field" gist_trgrm_ops);

You can easily modify this pattern for other (Postgres) indices and their operators.

gunicorn as your Django development server

Usually when I’m working on a Django project, I use the built-in runserver command. For deployment, we mostly use nginx and uwsgi in front of Django.

However, for a new side project I needed a local Django setup with a faster debug server. To keep things simple, I was looking for a pure Python solution that could serve static files, and did automatic reloading on source change, just like runserver.

I know that gunicorn is often used for deployment behind a reverse proxy and static file server like nginx, but I was curious whether gunicorn could also act as Django development server, and also serve up static files without having to run collectstatic (i.e. serving the static files directly from their development locations).

The answer is: Yes. With DJ-Static serving static files, gunicorn turns out to be a fantastic development server for Django. Here’s how to get it going:

Install gunicorn and DJ-Static in your Django virtual environment:

workon my_ve # I use virtualenvwrapper, hence workon to activate
pip install gunicorn dj-static

Add dj_static to INSTALLED_APPS in your settings.

In your wsgi.py, wrap your WSGI application in a Cling object. The two relevant lines are:

from dj_static import Cling
application = Cling(get_wsgi_application())

Now you can start your Django development project as follows:

gunicorn -w 6 --access-logfile - --reload your_project.wsgi:application

This will start six worker processes (tune for your setup), logs access to stderr, and reloads when it detects changes to the source. Welcome to your shiny, new and slightly faster Django development server!

Getting Django Rest Framework to parse docstrings as reStructuredText

The Django REST Framework is awesome, for a whole bunch of reasons, one of them being the browsable HTML version of your API that it automatically generates for you.

As a part of this, it extracts any docstrings that you might have defined for the relevant class (ViewSet or CBV) and adds an HTML version of this documentation to the browsable API, like this:

However, as described in the relevant documentation, it expects Markdown syntax by default. I like Markdown, but the rest of my Python docstrings are all in reStructuredText, which is the default documentation format for many Python projects.

In short, I would prefer to type my view docstrings in reStructuredText and not in the DRF-default markdown, because reStructuredText is what all of my other non-DRF code is using.

Fortunately, DRF has made this relatively easily configurable, if you’re not afraid to do some searching and digging (I couldn’t find a single complete worked-out example). Especially the code for transforming RST text into parsed HTML is not as straight-forward as one would have hoped. The DRF part of the deal is clearly described in its documentation.

Here’s how you can do this:

Define a new DRF view description function

Add the following function to your code somewhere. I added it to the views module in a Django app with reusable library code:

from django.utils.safestring import mark_safe
from docutils import core
from rest_framework.compat import smart_text
from rest_framework.utils import formatting

def get_view_description(view_cls, html=False):
    """Alternative view description function to be used as the DRF
    ``VIEW_DESCRIPTION_FUNCTION`` so that RestructuredText can be used
    instead of the DRF-default MarkDown.

    Except for the RST parts, derived by cpbotha@vxlabs.com from the
    DRF default get_view_description function.

    """

    description = view_cls.__doc__ or ''
    description = formatting.dedent(smart_text(description))
    if html:
        # from https://wiki.python.org/moin/ReStructuredText -- we use the
        # third recipe to get just the HTML parts corresponding to the ReST
        # docstring:
        parts = core.publish_parts(source=description, writer_name='html')
        html = parts['body_pre_docinfo']+parts['fragment']
        # have to use mark_safe so our HTML will get explicitly marked as
        # safe and will be correctly rendered
        return mark_safe(html)

    return description

Configure the view description function

In your settings, activate the new view description function:

REST_FRAMEWORK = {
    # other REST_FRAMEWORK config here...
    'VIEW_DESCRIPTION_FUNCTION': 'your_app.views.get_view_description'
}

Enjoy your reStructuredText

If you now reload any of your browsable API pages, you should see your reStructuredText nicely parsed.

The Django Book 2.0 in MobiPocket / Kindle format

I wanted to read the web preview of the Django Book’s second edition on my Kindle. Besides the fact that all image links are broken on that website and have apparently been so for some time, I prefer to have these things in the DRM-free MobiPocket / Kindle format. Of course I couldn’t find this anywhere, so I rolled my own based on the book’s SVN repository.

On this page you can download the MobiPocket version of the book and the HTML source files I generated to make it. You can also read on for the skinny on how you can do this yourself.

This procedure works best on a unix-like machine, as we’re going to use grep and sed along with some Python.

1. We start by doing a checkout of the reStructuredText sources of the book, moving the linked graphics into the same directory as the reStructuredText txt files and then creating a grepindex.txt file that will serve as the basis for our table of contents index.txt:

svn co http://djangobook.com/svn/branches/2.0/ 20svn
cd 20svn
find graphics/ -name *.png -exec mv {} . \;
grep -h "^Chapter [0-9]*:" *.txt > grepindex.txt

2. The grepindex.txt will now be converted to something more reStructuredText-like using this script, called grepindex2index.py:

# first do:
# grep -h "^Chapter [0-9]*:\|^Appendix [A-Z]:" *.txt > grepindex.txt
# then:
# python grepindex2index.py grepindex.txt > index.txt

import re
import sys

rst_header = """
===================
The Django Book 2.0
===================

Copyright 2006, 2007, 2008, 2009 Adrian Holovaty and Jacob Kaplan-Moss.
This work is licensed under the GNU Free Document License.

This ebook version was prepared by Charl Botha <http://charlbotha.com/> from
the SVN at http://djangobook.com/svn/branches/2.0/ on 2011-04-25, and is
hosted by <http://vxlabs.com/>.

"""

def main():
    f1 = open(sys.argv[1])
    print rst_header

    # this will match on "Chapter 10: the title" or "Appendix B: another title"
    # groups 0: chapter 10 or appendix A; 1: 10 (or None), 2: A (or None), 3: title
    pat = re.compile('(^Chapter\s*([0-9]*)|^Appendix\s*([A-Z])):\s*(.*)$')
    chapters = []
    appendices = []
    for l in f1:
        # Chapter 10: Advanced Models -> `Chapter 10: Advanced Models <chapter10.html>`_
        mo = pat.match(l)

        if mo.groups()[1] is not None:
            chapters.append("* `Chapter %s: %s <chapter%02d.html>`_" % (mo.groups()[1],mo.groups()[3],int(mo.groups()[1])))

        else:
            appendices.append("* `Appendix %s: %s <appendix%s.html>`_" % (mo.groups()[2],mo.groups()[3],mo.groups()[2]))

    print "\n".join(chapters)
    print "\n".join(appendices)

if __name__ == "__main__":
    main()

Save this to a script called grepindex2index.py, then invoke it with:

python grepindex2index.py grepindex.txt > index.txt

3. We’ll then proceed to fix all chapter references with the following bit of sed:

sed -i "s/\.\.\/\(chapter[0-9]*\)\//\1.html/g" chapter*txt

(this will change all “../chapter??/” links to just “chapter??.html”)

4. Everything is now ready to be converted to HTML:

for i in *.txt; do rst2html $i `echo $i | cut -f 1 -d .`.html; done

5. After downloading this django desktop background as cover image, I dragged and dropped the top-level index.html file onto the free Calibre software to import it, and then used the edit metadata function to set the cover image. After this, I converted the imported files to MobiPocket remembering to add the word “appendix” to the chapter detection xpath expression in the “structured detection” section of the conversion dialogue.

That’s all there’s to it! Have a good read.