Getting Django Rest Framework to parse docstrings as reStructuredText

The Django REST Framework is awesome, for a whole bunch of reasons, one of them being the browsable HTML version of your API that it automatically generates for you.

As a part of this, it extracts any docstrings that you might have defined for the relevant class (ViewSet or CBV) and adds an HTML version of this documentation to the browsable API, like this:

However, as described in the relevant documentation, it expects Markdown syntax by default. I like Markdown, but the rest of my Python docstrings are all in reStructuredText, which is the default documentation format for many Python projects.

In short, I would prefer to type my view docstrings in reStructuredText and not in the DRF-default markdown, because reStructuredText is what all of my other non-DRF code is using.

Fortunately, DRF has made this relatively easily configurable, if you’re not afraid to do some searching and digging (I couldn’t find a single complete worked-out example). Especially the code for transforming RST text into parsed HTML is not as straight-forward as one would have hoped. The DRF part of the deal is clearly described in its documentation.

Here’s how you can do this:

Define a new DRF view description function

Add the following function to your code somewhere. I added it to the views module in a Django app with reusable library code:

from django.utils.safestring import mark_safe
from docutils import core
from rest_framework.compat import smart_text
from rest_framework.utils import formatting

def get_view_description(view_cls, html=False):
    """Alternative view description function to be used as the DRF
    ``VIEW_DESCRIPTION_FUNCTION`` so that RestructuredText can be used
    instead of the DRF-default MarkDown.

    Except for the RST parts, derived by cpbotha@vxlabs.com from the
    DRF default get_view_description function.

    """

    description = view_cls.__doc__ or ''
    description = formatting.dedent(smart_text(description))
    if html:
        # from https://wiki.python.org/moin/ReStructuredText -- we use the
        # third recipe to get just the HTML parts corresponding to the ReST
        # docstring:
        parts = core.publish_parts(source=description, writer_name='html')
        html = parts['body_pre_docinfo']+parts['fragment']
        # have to use mark_safe so our HTML will get explicitly marked as
        # safe and will be correctly rendered
        return mark_safe(html)

    return description

Configure the view description function

In your settings, activate the new view description function:

REST_FRAMEWORK = {
    # other REST_FRAMEWORK config here...
    'VIEW_DESCRIPTION_FUNCTION': 'your_app.views.get_view_description'
}

Enjoy your reStructuredText

If you now reload any of your browsable API pages, you should see your reStructuredText nicely parsed.

Use ADB to bypass dog-slow MTP transfer of files from Android to Linux

Last night I had to backup 2760 photos and videos, about 6.3G worth, from my Nexus 4 Android phone to my Linux laptop.

The Nexus 4, like many other Android phones, only supports the Media Transfer Protocol, or MTP, for transferring files via USB connection. With Ubuntu 14.04, this is a fortunately a plug and play situation: Connect the phone via USB cable, and start dragging and dropping files to and fro using the built-in file manager on the Linux side.

Unfortunately, this turned out to be dog slow. Stabilising at about 1Mbyte/s, this was going to take more time than I had at my disposal. This looks like it could be a Linux-only problem, but I’d like to see that confirmed. Whatever the case may be, I had to find alternatives.

Accessing the AirDroid web-interface on my Android telephone.

My next stop was the ssh server app on the android side. I confirmed that I could ssh in to my unrooted phone (pretty cool that!), and then I confirmed, using the built in ssh:// filesystem support in the Linux file manager, that file transfer throughput was still dog slow (also about 1Mbit/s). After some minutes, it looked like the whole transfer stalled completely.

btsync was of no help in this case, because I have 1.4 on my phone, and use 1.3 on all my other devices (1.4 was completely useless in its beta phase, so I decided to stick with 1.3 until convinced otherwise).

AirDroid is an extremely well done app that enables one to remote control one’s android phone via a super attractive web interface, over its wifi interface. One of its many functions is file transfer, up and down. Selecting to download a whole directory of files results in a huge ZIP file being streamed. This came down at between 2 and 3 Mbyte/s, staying close to 3 for most of the time.

I was still curious whether we could do better via the USB cable, instead of over wifi, so I fired up the Android Debug Bridge, or ADB. This only works if developer mode has been activated on the android phone, and USB debugging mode has been activated. In other words, this is probably not for novice users.

By using adb pull on the whole directory of files, it started downloading all 6.3G worth of photos and videos. At the end of this, the average throughput was 4.2Mbyte/s, the best of all methods I had tested.

Conclusion

If you need to transfer many files from your MTP-only Android telephone to a Linux host, either use AirDroid if you want this to be as easy (and as pretty!) as possible, or use ADB if you want the maximum throughput and don’t mind getting your hands dirty.

Convert dates to different formats in LibreOffice Calc

I’m helping someone process a collection of research data that has been entered by a third party using Excel. We’re using LibreOffice Calc, because research should be reproducible by anyone, not just those in possession of prioprietary software licenses (this also means that we use R, JGR and DeduceR instead of SPSS and Statistica; perhaps more on that later).

After having to fix hundreds of badly entered dates with basic functions (we highly recommend that ISO 8601 format dates, i.e. YYYY-MM-DD, are used from the very start, instead of ambiguous local formats), we ended up with a stubborn subset of DD-MM-YYYY formatted dates in cells that explicitly had ISO 8601 format configured.

What could we do to convert these DD-MM-YYYY dates to the ISO 8601 standard YYYY-MM-DD dates?

The trick here is to use LibreOffice RIGHT, MID and LEFT functions to pull the faulty date apart, and then to put it back together in a new column using the DATE function.

In this screenshot you can see an example:

RIGHT(A2,4) extracts 4 characters from the right of 13-12-1981, yielding 1981, MID(A2,4,2) yields the 2 characters starting at position 4, and LEFT(A2,2) gives us the two characters at the left, in other words the day.

Putting all of this together with the function =DATE(RIGHT(A2,4), MID(A2,4,2), LEFT(A2,2)) will return the date formatted according to the column format setting (right click column heading, select Format Cells and then the ISO 8601 formatting). Obviously you have to replace A2 with the cell broken date format that you want to fix.

Globally, you would create a new column, use the formula above in its first row to fix the first date in the existing broken date column, then replicate the formula all the way down by clicking on it, and then dragging the rectangle at its lower right all the way down.

(Searching for “LibreOffice convert date formats” was of no help, whereas doing the same for Excel yielded at least two good answers (one and two), on which this post is based. I’m putting this out there so that searching for LibreOffice will also turn up something useful.)

CloudFlare full optimizations break MathJax

I’ve just activated CloudFlare (the free tier) for vxlabs.com, hoping to do even faster page loads. Most of my WordPress installations already use WP Super Cache to serve mostly static pages when you come here, but CloudFlare should speed this up even further via their content and network traffic optimization, and their servers dotted all over the globe.

Configuration was quite painless (you have to configure your domain’s DNS to point to CloudFlare’s servers, then a few settings using their web interface). It somehow missed my mail server’s DNS record when copying everything over, but that was quick to fix. (It also seems that the SSL-protected roundcube I have on another server in this domain now keeps on logging me out after 2 seconds, but I’m not 100% sure that that’s due to CloudFlare.)

In any case, I soon noticed that the MathJax equations in this post about level sets did not show up. Turns out you need to change the CloudFlare Performance Profile from “CDN + Full Optimizations” to “CDN + Basic Optimizations”, and then your equations should appear again.

In other words, do this:

to see this:

Driving the Dell U2713HM at 2650×1440 from the HDMI output of the Acer V3-571G

(TL;DR See the last paragraph for how to get the Dell U2713HM working on the HDMI output of the Acer V3-571G at 2560×1440 @ 50Hz.)

The Dell Ultrasharp U2713HM is a 27″ IPS panel with a resolution of 2560×1440. I recently acquired this monitor and wanted to connect it to my Linux-only Acer V3-571G i7 laptop, which only a VGA (D-SUB; max resolution 2048×1536) and an HDMI 1.4 output.

The monitor has been optimised to show the Dell logo. (image from the Engadget review I linked to above.)

HDMI 1.4 does support 2560×1440, but the HDMI 1.3 input on the Dell U2713HM does not. (The HDMI 1.4 input on the more expensive Dell U2713H does.) This means that we have to use either the DVI or DisplayPort inputs.

For 2560×1440 at 60Hz refresh rate, normal single-link DVI is not sufficient. One either needs dual-link DVI, which I don’t have, or one can use a cheap HDMI to DVI connector and tweak the timings of normal single-link DVI to supply 2560×1440 at a frequency that is as close as possible to 60Hz, but still fits within the available bandwidth.

Part of this tweaking is making use of reduced blanking, an optimization that can be done on LCD panels where there’s no electron beam (as is the case in CRTs) that needs time to be repositioned. In short, we can squeeze out more resolution and refresh from the same bandwidth.

NotebookCheck has a wealth of information on tweaking these timings. Unfortunately, the configuration they supply for 2560×1440 at 55Hz only caused flickering on my setup.

Fortunately, Linus Torvalds (just some guy who seems to know quite a bit about Linux ;) documented on Google+ his adventures getting such a monitor going under Linux, albeit with a 30Hz refresh rate. Fortunately, a commenter named Tim Small posted the timings he had generated with a hacked version of cvt!

Based on his timings, I could get my monitor going stably at 2560×1440 at 50Hz. Enter the following in a terminal:

xrandr --newmode "2560x1440_50.00_rb" 200.25  2560 2608 2640 2720  1440 1443 1448 1474  +HSync -Vsync
xrandr --addmode HDMI1 "2560x1440_50.00_rb"

When you enter after the second line, the monitor should switch to the 2560×1440 mode. After having done this, 2560×1440 appears as a selectable mode in the Ubuntu Displays app.