The Django Book 2.0 in MobiPocket / Kindle format

I wanted to read the web preview of the Django Book’s second edition on my Kindle. Besides the fact that all image links are broken on that website and have apparently been so for some time, I prefer to have these things in the DRM-free MobiPocket / Kindle format. Of course I couldn’t find this anywhere, so I rolled my own based on the book’s SVN repository.

On this page you can download the MobiPocket version of the book and the HTML source files I generated to make it. You can also read on for the skinny on how you can do this yourself.

This procedure works best on a unix-like machine, as we’re going to use grep and sed along with some Python.

1. We start by doing a checkout of the reStructuredText sources of the book, moving the linked graphics into the same directory as the reStructuredText txt files and then creating a grepindex.txt file that will serve as the basis for our table of contents index.txt:

svn co http://djangobook.com/svn/branches/2.0/ 20svn
cd 20svn
find graphics/ -name *.png -exec mv {} . \;
grep -h "^Chapter [0-9]*:" *.txt > grepindex.txt

2. The grepindex.txt will now be converted to something more reStructuredText-like using this script, called grepindex2index.py:

# first do:
# grep -h "^Chapter [0-9]*:\|^Appendix [A-Z]:" *.txt > grepindex.txt
# then:
# python grepindex2index.py grepindex.txt > index.txt

import re
import sys

rst_header = """
===================
The Django Book 2.0
===================

Copyright 2006, 2007, 2008, 2009 Adrian Holovaty and Jacob Kaplan-Moss.
This work is licensed under the GNU Free Document License.

This ebook version was prepared by Charl Botha <http://charlbotha.com/> from
the SVN at http://djangobook.com/svn/branches/2.0/ on 2011-04-25, and is
hosted by <http://vxlabs.com/>.

"""

def main():
    f1 = open(sys.argv[1])
    print rst_header

    # this will match on "Chapter 10: the title" or "Appendix B: another title"
    # groups 0: chapter 10 or appendix A; 1: 10 (or None), 2: A (or None), 3: title
    pat = re.compile('(^Chapter\s*([0-9]*)|^Appendix\s*([A-Z])):\s*(.*)$')
    chapters = []
    appendices = []
    for l in f1:
        # Chapter 10: Advanced Models -> `Chapter 10: Advanced Models <chapter10.html>`_
        mo = pat.match(l)

        if mo.groups()[1] is not None:
            chapters.append("* `Chapter %s: %s <chapter%02d.html>`_" % (mo.groups()[1],mo.groups()[3],int(mo.groups()[1])))

        else:
            appendices.append("* `Appendix %s: %s <appendix%s.html>`_" % (mo.groups()[2],mo.groups()[3],mo.groups()[2]))

    print "\n".join(chapters)
    print "\n".join(appendices)

if __name__ == "__main__":
    main()

Save this to a script called grepindex2index.py, then invoke it with:

python grepindex2index.py grepindex.txt > index.txt

3. We’ll then proceed to fix all chapter references with the following bit of sed:

sed -i "s/\.\.\/\(chapter[0-9]*\)\//\1.html/g" chapter*txt

(this will change all “../chapter??/” links to just “chapter??.html”)

4. Everything is now ready to be converted to HTML:

for i in *.txt; do rst2html $i `echo $i | cut -f 1 -d .`.html; done

5. After downloading this django desktop background as cover image, I dragged and dropped the top-level index.html file onto the free Calibre software to import it, and then used the edit metadata function to set the cover image. After this, I converted the imported files to MobiPocket remembering to add the word “appendix” to the chapter detection xpath expression in the “structured detection” section of the conversion dialogue.

That’s all there’s to it! Have a good read.

Don’t buy HomePlug / Powerline ethernet adapters

(Post summary: The real-world throughput of current generation Homeplug AV 200 Mbit/s powerline ethernet adapters in a modern house is woefully inadequate. Even wireless is much to be preferred, and can be had for cheaper. Read below for why.)

Based on the superb price / performance ratio of the MSI ePower 200AV II kit as extolled by this comparative review (32 powerline adapters were tested), and especially the fact that in the test setup these adapters managed to attain 32 Mbit/s even in the bad case scenario (two different circuits, 100 metres of cable separating the two adapters), I purchased the MSI ePower 200AV+ II kit to replace a wireless link I currently have in my house between the second and third floors. Based on iperf measurements, the wireless link currently manages around 22Mbit/s of throughput. Because the two power sockets I was planning to use are on the same circuit, I thought that I could improve on this existing connection with the two powerline adapters. Little did I know…

The MSI ePower 200AV+ kit still nicely in its box.

After taking delivery of the new kit, I started testing with both adapters in the same room. One thing that can be said for this hardware, is that it’s really plug and play. One adapter in wall-socket and connected to first floor ethernet hub, second adapter in wall-socket and connected to my laptop, and my laptop was online. No mess no fuss.

The first nasty surprise appeared as I installed and started up up the software that came in the box (version 5.0). It claimed not to find any homeplug adapters on the network, although the laptop I was running it on was directly connected to one of the PLCs. So I downloaded version 6.0 of the software, only available in German, from MSI’s website and that did manage to see both adapters. At least now I was able to configure both the PLCs (in German…) and set the network name to something private for security’s sake.

After moving one of the adapters to my study on the third floor and connecting it to a Linux server there, I could start running iperf on the server and my laptop on the second floor for testing. The two PLC adapters were now in the same-circuit sockets I was intending to use. Below is the output of three runs of iperf:

C:\Users\cpbotha\Downloads>iperf -c 192.168.1.126
------------------------------------------------------------
Client connecting to 192.168.1.126, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[148] local 192.168.1.38 port 49921 connected with 192.168.1.126 port 5001
[ ID] Interval       Transfer     Bandwidth
[148]  0.0-10.0 sec  10.7 MBytes  8.95 Mbits/sec

C:\Users\cpbotha\Downloads>iperf -c 192.168.1.126
------------------------------------------------------------
Client connecting to 192.168.1.126, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[148] local 192.168.1.38 port 49987 connected with 192.168.1.126 port 5001
[ ID] Interval       Transfer     Bandwidth
[148]  0.0-10.0 sec  11.2 MBytes  9.41 Mbits/sec

C:\Users\cpbotha\Downloads>iperf -c 192.168.1.126
------------------------------------------------------------
Client connecting to 192.168.1.126, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[148] local 192.168.1.38 port 49988 connected with 192.168.1.126 port 5001
[ ID] Interval       Transfer     Bandwidth
[148]  0.0-10.0 sec  10.4 MBytes  8.70 Mbits/sec

Yes ladies and gentlemen, that’s an absolutely miserable 9 Mbit/s (think 1 Mbyte/s…) between two MSI ePower 200AV+ II adapters on the same circuit, on different floors in a house that was built 5 years ago. For your reference, my wireless link (two Sitecom 300N X2 access points in Wireless Distribution Mode) easily manages 22 Mbit/s effective throughput right through a reinforced concrete floor.

MSI’s own utility claimed the following (I’ve cut out things that you don’t need to see):

Rate 18/39
Vendor Atheros
Firmware INT6400-MAC-4-0-4011-00-3430-20090501-FINAL-C
Übertragungsrate hoch 174.00
Übertragungsrate niedrig 15.00

… with its reported link speed varying between 17 and 35 Mbit/s.

I tried the third floor PLC adapter on various different sockets, the results were similarly depressing.

As if the miserable throughput was not reason enough to avoid powerline adapters, note the following mechanical issue:

When the adapter sits in a double socket, the only thing that’ll fit in the free socket is a really thin plug as the upper part of the adapter is flush with the socket and covers a fair part of the other hole.

As you can see in the caption, although the PLC has a socket of its own, it covers the neighbouring socket in a double-socket setup to such an extent that you can only fit a really thin plug in the neighbouring socket.

To conclude: Homeplugs perform really well in a test environment, even when different circuits are introduced and metres of extra cable are inserted. However, what the tests often fail to take into account, is the fact that people actually make use of other electrical devices besides homeplugs (!!) and that these all introduce noise into the home grid that apparently severely affect powerline ethernet performance. Perhaps in your house things work out differently, but my advice would be to steer clear of powerline adapters (unless you can borrow a pair to test with), opting of course for ethernet cables whenever this is possible and for wireless otherwise.

Sipura / Linksys / Cisco SPA3102 Voice Gateway in The Netherlands

After recently spending some hours configuring my new Cisco SPA3102 Voice Gateway with a Betamax SIP provider (voipbuster / voipstunt / voipcheap / and so forth, see http://www.backsla.sh/betamax for a full list of all Betamax providers) and the Dutch PSTN system, I thought I’d try and make your life easier by documenting the most important of the settings.

With this box you can stick it to the man!

The SPA3102 is connected to an existing router, so I have the Lan Setup set to Bridge, and have configured the Internet port (on the Router | Wan tab) with a suitable static IP. To be clear, in this configuration, one only connects up the Internet port and NOT the Ethernet port.

Voice | Regional

For the Voice | Regional tab I found the following Call Progress Tone settings on a voxilla forum page by users PJH and edokter:

Dial Tone: 425@-10;20(*/0/1)
Second Dial Tone: 425@-10;20(*/0/1)
Outside Dial Tone: 350@-19;440@-19;20(*/0/1+2)
Busy Tone: 425@-10;10(.5/.5/1)
Reorder Tone: 425@-10;10(.25/.25/1)
Off Hook Warning Tone: 1000@0;*(0/9.5/0,.1/.1/1,.1/.1/1,.1/0/1)
Ring Back Tone: 440@-19,480@-19;*(2/4/1+2)
Confirm Tone: 425@-10;1(.1/.1/1)
SIT1 Tone: 950@-19,1400@-19,1800@-19;30(.333/0/1,.333/0/2,.333/1/3)
MWI Dial Tone: 425@-10;1(.1/.1/1);20(*/0/1)
Cfwd Dial Tone: 425@-10;20(.5/.05/1)

On the same tab and from the same forum page, the following Distinctive Ring Patterns and Distinctive Call Waiting Tone Patterns:

Ring1 Cadence: 90(1/4)
Ring2 Cadence: 90(.3/.4,.3/4)
CWT1 Cadence: 60(.5/9.5)
CWT Frequency: 425@-16

To get caller ID working, I made the following changes (still on the Voice | Regional tab):

Caller ID Method: ETSI FSK
Caller ID FSK Standard: bell 202

Finally on this tab, set:

FXS Port Impedance: 270+750||150nF

(according to the Cisco admin manual, this is the standard for The Netherlands)

Voice | Line 1

This is the tab where you get to do most of the SIP configuration.

I have my primary betamax provider (12voip.com) setup under Proxy and Registration and Subscriber Information:

Proxy: name of your sip server e.g. sip.voipbuster.com
Display name: 0031xxxxxxxxx (number that you have verified and configured as callerid with the voipbuster software)
User ID: 0031xxxxxxxxx (same as above)
Password: hard to figure this one out 🙂
Use Auth ID: yes
Auth ID: your betamax (voipbuster etc.) username

My dialplan looks as follows:

(112S0<:@gw0>|0800x.<:@gw0>|090x.<:@gw0>|<#9:>xx.<:@gw0>|<1601:>xx.|<0:0031>[1-7]xxxxxxxxS0|xx.|*xx)

Here is the commented version for your edification:

(112S0<:@gw0>| # emergency number goes through PSTN
0800x.<:@gw0>| # 0800 numbers go through PSTN
090x.<:@gw0>| # 090x numbers go through PSTN
<#9:>xx.<:@gw0>| # prefix number with #9 to force dialing through PSTN
<1601:>xx.| # throw away 1601 preselect still in some handset phonebooks. you won't need this.
<0:0031>[1-7]xxxxxxxxS0| # numbers without 0031 country code will get it added
xx.| # all other numbers
*xx) # linksys codes

You can find more info on dialplans here and here. You can use other betamax providers in the Gateway Accounts and integrate them in your dialplan. The betamax providers don’t require registration.

Voice | PSTN Line

In order for your SPA3102 to realise that the call has been ended, you need to setup PSTN Disconnect Detection correctly:

Detect CPC: yes
Detect Polarity Reversal: no
Detect Disconnect Tone: yes
Disconnect Tone: 425@-30,425@-30; 2(0.5/0.5/1+2)

When I had “Detect Polarity Reversal” set to yes, I’d get immediate disconnects on incoming calls with my new Ziggo cable PSTN connection.

On the same tab, make sure your FXO Port Impedance is also set to 270+750||150nF (the Dutch standard).

Later I discovered that if we took longer than 10 seconds to answer the phone, the call would be lost. To fix this, set the “PSTN Answer Delay” to 60 seconds or something reasonable.

Conclusion

Please let me know in the comments if this worked for you, or if you have any questions on the instructions above!

Updates

  • June 17, 2014: Fixed typesetting of the code examples.
  • September 30, 2012: Removed 13 digit S0 shortcut from dialplan, this could interfere with calls to international numbers with more digits. Added two links explaining dialplans.

International characters on the HTC Desire Z keyboard

I’m typing this up because it took me far too long to find, probably because I wasn’t using the right search terms, or because I was trying stupid key combinations…

Frustration (was: threesixtyfive | day 244)

In any case, if you want to make international characters (that is, characters with accent marks, diacritics, trema (plural tremata), umlauts, eat that google!) such as ë, é, ê or even ö or ï and so forth on the HTC Desire Z hardware keyboard, you simply long press the base character. A menu pops up and you get to choose the accented character that you would prefer to insert at that moment.

I sincerely apologise for interrupting your usual high IQ programming with this. If however you stumble upon this post and it helps you, why don’t you leave a comment just for laughs?

HTC Desire Z: An in-depth and nerdy review.

This review differs from other HTC Desire Z reviews because I’ve actually been using this phone as my only smartphone quite intensively, and because I’m writing up, in great depth, my iopinion as a gadget-toting ex-Linux-zealot computer science nerd. Take that engadget!

As many of you know, I was completely in love with my previous smartphone. Not surprising, seeing as the Nokia e71 is perfect, at least hardware-wise. However, partly due to Symbian (Nokia’s operating system) really dragging its heels (Nokia, does your mobile OS really have to feel like it teleported in from 1970?) and partly due to Android (Google’s mobile operating system) doing the exact opposite, I’d been not-so-patiently waiting for the right Android-running telephone to come along so I could start drinking the Google Kool-Aid.

Enter the HTC Desire Z, known in the US as the T-Mobile G2, a micro-laptop with built-in mobile phone, or a flashlight with built-in laptop and phoning function, whichever you prefer:

Pretty pretty Desire Zs! (image courtesy of HTC)

Acquisition

After having read a gazillion mobile phone reviews and months of intensive pondering on what physical keyboard-toting smartphone I would get after two years with my E71, I had narrowed it down to a Blackberry Bold 9780 or the HTC Desire Z. Blackberry lock-in (if you don’t pay RIM their monthly protection money, half of your telephone stops working) together with my penchant for all things Google led to the purchase of the HTC Desire Z from the smartphoneshop in Rotterdam (highly recommended: It’s a web shop, but I just popped in there on a Sunday to get the phone and sort my contract issues) along with a 1 year T-Mobile iPlan 150 contract (150 minutes, 150 texts and unlimited 2 Mbit/s internet).

The thing has now actually survived three weeks in my hands being super-intensively futzed around with, so it’s finally time to talk honestly about our experiences together.

The Good

Coming from the Blackberry-style Nokia E71 keyboard, a finely tuned mechanism for high-power text entry, it took a while to get used to the Desire Z’s slide-out keyboard. However, I’m now completely convinced that the keyboard is absolutely worth the extra 40 or 50 grams. Many have complained, but the mechanism is beautiful: It slides in and out with a satisfying PLOK. There are no springs or spring-loading in sight, as this business all elegantly gravity-powered. Yes, if you’d try to lie on your back holding it in front of your face, gravity starts working differently and the thing will fall open. My advice: Pick some other activity for performing whilst lying on your back. The keys themselves are nicely separated. Perhaps they could have been a tad less stiff. Importantly, I find that clipped nails greatly enhance my typing experience. To be quite honest, it’s a fine keyboard, but the still wins in this regard.

The phone has a great feel: With its 180 grams and subtle brushed metal finishing, it exudes quality when picked up. However, if you’re a smaller person, you might want to try it first before making the jump. I find the weight and form factor just perfect, but that’s of course all relative. I carry the thing, without a cover of any kind, in my trouser pocket. (Oh who am I kidding, I never wear trousers. They’re jeans. By the way, did you know that someone has a patent on reinforced trouser pockets?)

Finally, the software integration on this telephone is beautiful. So with Android 2.2.1 at the time of writing, and the HTC Sense layer, you receive a mail or check your Friend Stream (twitter, facebook and whatnot), tap on an attachment or link and the relevant application opens. New apps can register as handlers, so after you’ve for example installed Dropbox, long tapping on some files will give you the choice of syncing them with your Dropbox account, and so on. Because this is more a micro-laptop than a telephone, I definitely feel far less constrained when having to work on it than was the case on my E71: I can edit documents, SSH to places, review presentations use all forms of webapps, without anyone having to know that I was trapped on a train or some other odd place for a few hours. Also compared to the Apple 3GS with iOS 4.x, the Desire Z blows it out of the water in this respect, due to both Android and the keyboard of course.

Finally, and this is a good point of any Android device, the apps and the Android Market integration. Further down I will briefly discuss my favourite apps at the moment. For now it suffices to say that there are a gazillion apps, all installable (and the non-free ones purchasable) with a single tap. With a certain amount of searching, you invariably find that one app that you believe will help you be more effective / productive / happy / successful in life, at the very least for the subsequent five minutes. In other words, your toy is able to acquire completely new functions, not always equally useful, at the tap of a button.

The Bad

The 1300 mAh hour battery is a definitely weak point of this telephone. After the initial conditioning discharge-charge cycles, the battery did indeed go from utterly crappy to merely disappointing. It now lasts from about 8:30 in the morning till about midnight and on days when I spend more time than usual futzing with the thing, the hungry beast starts complaining at about 20:00 that I should really consider giving it something to eat. This is in spite of the fact that I mostly explicitly switch off the wireless (I know it times out by itself), keep the GPS switched off, switch off automatic syncing quite often and keep the screen brightness on automatic. If the Battery Usage app can be believed, the screen is the biggest culprit. Usually it consumes 60% or more of the power budget. According to CurrentWidget, the phone eats about 30mA in standby, about 140 mA when I’ve just switched on the screen and about 300 mA when I’m just browsing the interwebs.

I also do have some small gripes with the software:

  • Why in Android GMail 2.3.2, when I want to reply inline to an email (that’s new functionality), does it put my signature ABOVE the email that I’m replying to? The standard is that the signature should go BELOW, as is the case with web GMail. There also doesn’t seem to be any way that I can configure this. I’ve started a thread on the Google Mobile help forums, but apparently no-one finds it interesting enough.
  • Why do I have to root my phone or install the whole SDK and connect the phone to my PC  if I simply want to take a screenshot?
  • Ok sorry that last one was REALLY a minor niggle.
  • Why can’t I change the number of home screens in HTC Sense?
  • This one’s quite serious: Why is can’t I voice chat with Google Talk users from my Android phone? Google Talk is beautifully integrated, but no voice chat to be found, which is verging on the ridiculous.
  • Every so often, the Gallery app just forgets a bunch of photos on the SD card. Either it doesn’t list them, or it shows them as broken images (can’t be loaded). Unmounting and mounting the SD card fixes this, but what an annoyance… (android 2.2.1 update)

The UGLY

Even worse than the battery is the dismally performing optical trackpad. If you’ve ever played with one of the new Blackberrys, you know that an optical trackpad can work just perfectly, almost seeming to read your intentions rather than your finger motion. The trackpad on the Desire Z either reacts very slowly, i.e. vigorous finger motion results in very little cursor motion, or not at all, sometimes requiring 2 or even 3 swipes before the cursor does anything at all. What’s really frustrating, is that if I swipe with a piece of cloth (read “t-shirt”) over my finger, the thing works exactly like it should. Furthermore, swipes over the trackpad often end up on the “menu” or “back” touch keys, resulting in completely unintended consequences.

Now on a touch screen phone, and the Desire Z has a really great touch screen, the trackpad is less relevant, but why did HTC go to the trouble of integrating it there?! Also, why did they pick the crappiest trackpad they could find when the rest of the phone (ok, forget the battery for a moment) is constructed from top notch components? The trackpad could have been a really useful input device, allowing for fine cursor positioning for example.

Let me calm down for a while, and conclude this section with the following disclaimer: In the meantime, I’ve played with the trackpads on two different HTC Desire Zs. One of them was more or less the same as mine, the other seemed to be much better. I’ve also experimented with fingers belonging to different users than myself, that doesn’t seem to be a factor. Whatever the case may be, the trackpad on MY telephone is almost useless, but I’m going to test on more Desire Zs to establish whether mine is a dud, or it was a design error on HTC’s part. smartphoneshop has offered to take my phone back, but I have great difficulties parting with it for more than a few minutes, and it’s not sure whether a repair will even agree that something is wrong with the phone.

My favourite apps

You can see all of the apps I currently have on my telephone by going to my AppBrain user page. I’m only going to mention my favourite ones in the  following list:

  • AppBrain: You create an account on AppBrain with your Google credentials and then install the app on your phone and BAM, you have a much nicer interface for app installation, including personalised recommendations.
  • TweetDeck: Get and give your Twitter, Facebook and Buzz fix in a beautiful and functional column-based interface, plus that you can run variants of the software on your Windows, Mac or Linux running PC.
  • Note Everything: The HTC Desire Z doesn’t come with a default note-taking app, but the free version of Note Everything performs this function perfectly. It’s small and fast, can save to and load from Google Docs, and does search as you type! I bought the Pro version, as it has more types of notes, can backup the your whole archive to SD, and I wanted to support the authors.
  • SipDroid: I tried 3CXPhone and LinPhone, but SipDroid’s perfect integration with the calling function of the telephone won me over. In spite of the integration, it’s easy to exit when you don’t want to do SIP calling anymore.
  • Widgetsoid: Use this to setup pretty buttons on your homescreen for toggling and configuring bluetooth, wifi, syncing, screen brightness, and much much more.
  • QuickOffice Connect: I bought this office suite for its great integration with multiple Google Docs and Dropbox accounts, of course the idea that I can edit MS office format docs on my phone and the really fantastic PDF viewer that’s included. Nice tidbit: When you do a full text search, it searches through all local documents, as well as everything on all linked Google Docs and DropBox accounts. That’s pretty neat!
  • Jorte: The built-in calendar app sucks, wasting valuable screen real-estate on overly large on-screen buttons and huge fonts. In the week view, it’s not even able even to show the appointment label! Jorte solves all these problems, and even syncs with your google tasks. It even shows appointment names in the month view! In the near future, I will also be trying out Pimlical for Android, as I used to be a DateBk6 addict in my Palm days.

Conclusion

In spite of its flaws, namely the disappointing battery life and the hardly functional trackpad, the HTC Desire Z is the information age equivalent of a Swiss Army knife. It has everything you need to function, no, to excel as a data-wrangling nerd. You can go on the road, confident that you can wrestle to the ground anything thrown at you via the internet.  If your battery survives long enough, the built-in flashlight app (the camera flash is very bright) could even help you solve some real-world problems after dark. Accentuating the nature of this power-phone, a number of my more nerdily-inclined but socially adept friends and colleagues have acquired the Desire Z or are at this very moment lusting after it. If you in fact need a micro-laptop with phoning function and you can live with the charge-me-every-day battery and you don’t mind the trackpad, this is just the device for you.

If you have questions or suggestions, please leave me a comment below. I’d like to update and refine this review as new information comes in.

Addendum

  • HTC tech support is good! I mailed concerning my diacritics question, and was answered (in Dutch) within a day.