Use the hardware-based full disk encryption your TCG Opal SSD with msed

My blog post on usable hardware-based SSD encryption has seen a great deal of activity. Although that post dealt primarily with the ATA security based type of hardware-based full drive encryption, readers from all over joined the discussion in the comments to talk about an increasing number of new self-encrypting drives supporting the TCG Opal standard.

Up until recently, configuring these TCG Opal drives was only possible under Windows, or under Linux with a commercial solution that was not available to mere end-users. Fortunately, a programmer named r0m30 stepped up to the challenge and has developed an open source utility called msed and an accompanying pre-boot authorization (PBA) image with which the super fast encryption function on these drives can be fully configured and used also in pure Linux systems.

This post summarises how I built, configured and installed msed and its PBA on my Ubuntu 14.04.1 machine with its Samsung 850 PRO 512G TCG Opal-compliant SSD.

How does TCG Opal drive encryption work?

Many modern SSDs perform transparent AES encryption on all written data in hardware. One advantage of this approach is that the whole drive can be secure erased by simply generating a new set of encryption keys. Another advantage is that users can have all of their data fully encrypted at rest without any performance hit whatsoever. Also, third-party software-based drive encryption negatively affects SSD performance and longevity, for the largest part because this data is basically incompressible when it hits the drive.

TCG Opal is a new standard for communicating with supporting drives concerning their encryption functionality. Furthermore, it includes a really elegant way to have the user supply their authorization credentials.

In its default state, the main disc area is completely locked and inaccessible. However, when the system is booted, the encrypted disc exposes a fake disc from its firmware, called the shadow MBR (master boot record), 128MB in size. Usually this shadow MBR is flashed with the pre-boot authorization (PBA) image, which is in essence a small operating system (including MBR, boot sector, filesystem) that asks the user for their drive password, which it then communicates to the disc via OPAL commands. If the password is valid, the disc unlocks itself, and then the real operating system is loaded up.

This white paper by HP contains an explanation of the provisioning and boot process on page 5. To summarise: Once correctly configured, a system with such an OPAL-compliant disc will request the drive password at boot. The drive will only unlock and decrypt if the correct password is supplied.

Building msed and its PBA image from source

r0m30 programmed a suitable PBA image based on the syslinux open source, and a utility called msed for the provisioning (setting password, writing PBA image) of OPAL drives.

Because this software performs a security critical function, I reviewed as much as possible of the source code in syslinux/com32/msedpba (the Opal-specific part of the PBA) and of the whole msed utility, including the script that builds the PBA image. (I also spent some hours disassembling the binary PBA image.)

After this mini review, it was of course preferable to build and use my own binaries.

To build both the PBA image and msed from source, I did the following:

# I retrieved these sources on Tuesday 2015-02-10
git clone https://github.com/r0m30/msed.git
git clone https://github.com/r0m30/syslinux
cd syslinux
# make clean is going to fail trying to get the EFI submodule. ignore.
make clean
make bios
cd ../msed/image
sudo ./buildbiospba
# remember the location of the resultant .img file!
gunzip biospba-0.20beta.img.gz
# now let's build msed itself
cd ..
# I'm on x86_64, adapt to your own architecture!
make CONF=Release_x86_64
# copy the image to the same location as the msed binary
cp image/biospba-020beta.img .

Stripping the msed binary at top-level, I found an md5sum-identical binary to the 0.20.0 one that I downloaded from r0m30’s site:

cpbotha@meepz97:~/build/msed/msed/dist/Release_x86_64$ md5sum msed 
3a22c344ecbfa15b43ae7764341060ab  msed

Installing the msed PBA

This is very important: I’ve configured my BIOS to boot in legacy mode, i.e. NOT UEFI. The msed documentation also states that this is necessary. It also makes sense, because the PBA image is a legacy boot image!

msed needs libata.allow_tpm to be configured for the running kernel. I edited /etc/default/grub so it looked like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash libata.allow_tpm=1"

… after which I did update-grub and then rebooted. After reboot, msed --scan gave me sensible output.

It was now time to configure the drive for encryption. I found this quite stressful; I’ve had near-bricking experiences with expensive Intel 520 SSDs during some of my previous ATA security experiments with flakey BIOS implementations (Insyde H20, what a mess). In any case, I followed this procedure:

# set the drive password: mine is long, but no spaces, no special chars
./msed --initialsetup mylongpassword /dev/sda
# write the PBA into the shadow MBR
./msed --loadPBAimage mylongpassword biospba-0.20.img /dev/sda
# activate the shadow MBR
./msed --setMBREnable on mylongpassword /dev/sda
# activate drive locking
./msed --enableLockingRange 0 mylongpassword /dev/sda

After this, I switched the machine off, and on again. Lo and behold! I was prompted for my OPAL password at bootup, and could let myself in.

To test, I booted up the machine with a Linux Live USB. In place of the encrypted disk I could only see the shadow MBR.

Conclusion

TCG Opal is a great way of using your SSD’s hardware-based full disc encryption. I am very grateful to r0m30 for creating msed and its PBA image: These are crucially important open source tools for working with Opal discs.

Solving the Ubuntu 14.04 – NVIDIA 346 – nvidia-prime black screen issue

For a project that I’m currently helping with, we needed recent OpenGL features that are only available on NVIDIA drivers with version 340 and later.

Unfortunately, I have one of those NVIDIA Optimus laptops. Up to now, Bumblebee worked a treat (I would recommend this system in most cases), but for this project I needed the whole of X to run on the NVIDIA, so I had to make use of nvidia-prime to switch between Intel and NVIDIA mode.

After upgrading to the nvidia-346* packages from the xorg-edgers PPA, switching to nvidia mode by typing prime-select nvidia and then logging out and in to X, I was greeted by a black screen.

Analysis

Many hours of experimentation, script tracing and web searching later I made the following observations:

  • gpu-manager, part of ubuntu-drivers-common (in my case version 1:0.2.91.7), runs every time you start and stop your display manager (in other words, when you log out and back in) and then rewrites the /etc/X11/xorg.conf based on what it finds in the system.
  • In theory, with prime support in the NVIDIA drivers, xrandr is used to connect the output of NVIDIA adapter to the Intel adapter, which then displays the output. See the NVIDIA driver documentation for more details. The 90-nvidia.conf script in /usr/share/lightdm/lightdm.conf.d/ (part of the nvidia-prime package) calls /sbin/prime-offload, which will automatically take care of the xrandr setup for you.
  • gpu-manager was rewriting my xorg.conf file incorrectly, at least according to NVIDIA’s xrandr documentation. The primary issue was that gpu-manager was using the intel driver for the intel, instead of the modesetting driver.

The solution

All of this lead to the following (working: now tested on two setups) solution:

  • Switch to console (Ctrl-Alt-F1) and stop lightdm: sudo service lightdm stop
  • Disable gpu-manager by commenting out everything in /etc/init/gpu-manager.conf
  • Switch to nvidia mode by doing sudo prime-select nvidia
  • Change your /etc/X11/xorg.conf to look like this, making sure that the nvidia BusId is correct (check with lspci):
    Section "ServerLayout"
     Identifier "layout"
     Screen 0 "nvidia"
     Inactive "intel"
    EndSection
    
    Section "Device"
     Identifier "intel"
     Driver "modesetting"
    EndSection
    
    Section "Screen"
     Identifier "intel"
     Device "intel"
    EndSection
    
    Section "Device"
     Identifier "nvidia"
     Driver "nvidia"
     BusID "PCI:1:0:0"
    EndSection
    
    Section "Screen"
     Identifier "nvidia"
     Device "nvidia"
     Option "UseDisplayDevice" "None"
    EndSection
    
  • In the comments, Christopher May-Townsend made this brilliant suggestion. By doing sudo chattr +i /etc/X11/xorg.conf you can prevent any process from changing the file. We highly recommend that you do this, as users have reported that even after disabling the gpu-manager upstart job, it still manages to change the xorg.conf during reboot.
  • Start X up again with sudo lightdm start

If you are still greeted by a black screen, switch back to the console, and double-check that the xorg.conf has not again been rewritten to its pre-modesetting state. (if you’ve used the chattr trick above, you should be fine)

If you want to switch back to Intel you will have to stop lightdm, re-enable gpu-manager, make xorg.conf editable again with sudo chattr -i /etc/X11/xorg.conf, activate intel mode with sudo prime-select intel and then restart X with sudo service lightdm start.

It’s very possible that later versions of gpu-manager might have fixed this behaviour.

Let me know in the comments if this worked for you!

Sending emails with math and source code

Org mode is great for authoring rich documents with syntax highlighted source code, LaTeX math and images. It even supports evaluating live snippets of code embedded in the text. It does all of this whilst remaining a plain text format.

Imagine how useful it would be to author programming-related or technical emails using this functionality?

Imagine no more! org-mime, part of the org mode contrib, does this for a number of emacs-based mail clients. However, our preference is for mu4e, which is not part of that list.

Fortunately, it seems mu4e has orgmode support integrated. To get this working, configure the following in your ~/.emacs.d/init.el:

;; configure orgmode support in mu4e
(require 'org-mu4e)
;; when mail is sent, automatically convert org body to HTML
(setq org-mu4e-convert-to-html t)

When composing a new email, switch on the special mu4e / orgmode mode with M-x org~mu4e-mime-switch-headers-or-body (it will automatically switch between compose mode and orgmode depending on whether your cursor is on the headers or in the body).

You should add the following options to your org mode body. The first configures LaTeX (math) to be converted using imagemagick (instead of MathJax or dvipng), and the second removes the table of contents.

#+OPTIONS: tex:imagemagick
#+OPTIONS: toc:0

When you want to send the email, move to the headers, and then do C-c C-c. Because org-mu4e-convert-to-html is set to true, the orgmode body will automatically be converted to HTML before being sent.

Until my pull request is merged in, you will have to make the following change to function org~mu4e-mime-convert-to-html() in org-mu4e.el:

(insert (org~mu4e-mime-multipart
                  body html (mapconcat 'identity html-images "\n")))

should be:

(insert (org~mu4e-mime-multipart
                  raw-body html (mapconcat 'identity html-images "\n")))

In other words, body becomes raw-body, else your outgoing mails will have HTML in their text parts, instead of plain text.

In that same function, you might also want to change:

(org-export-preserve-breaks t)

to

(org-export-preserve-breaks nil)

So that the HTML email does not get hard linebreaks.

Following are examples of some of the elements you might include in such a rich text email. They should appear in your rich text capable email client as follows:

Here are the originals:

A section with some math

We can easily include math using LaTeX:

\[ C(p,t) = \lbrace x_0(p,t), x_1(p,t), \cdots, x_{N-1}(p,t) \rbrace\tag{1} \]

Some syntax highlighted C++

Syntax highlighted source code, for all of the languages supported by emacs (a very long list):

int main() {
    // Recently discovered trick, local functions in C++!
    struct Kludge {
        static int doSomething(int x) {
            return x * 2;
        }
    };

    for (int i=0; i < 10; i++) {
        cout << Kludge::doSomething(i) << endl;
    }

    return 0;
}

Inline GraphViz DOT evaluation for graphs using Emacs, org-mode and org-babel

With Emacs, org mode and org-babel, it’s possible to evaluate source code samples embedded in your org files and have the output of said evaluation appear inline. This makes for a beautiful literate programming environment. It also enables one to include graphs in one’s documents (org mode, PDF, HTML presentations or blog posts) by using for example GraphViz.

This blog post (obviously authored using Emacs and Org mode) contains short instructions for doing so.

First follow the org-babel documentation and enable source code evaluation for dot by adding the following to your Emacs init.el:

(org-babel-do-load-languages
 'org-babel-load-languages
 '((dot . t)))

Then create a new document, and add something like the following dot source code sample to it:

#+BEGIN_SRC dot :file dot_success.png :cmdline -Kdot -Tpng
  digraph {
  // graph from left to right
  rankdir=LR;
  splines=true;
  node [shape=box];

  id [label="Install Graphviz"]
  conf [label="Configure org-babel"]
  dot [label="DOT in org-mode"]

  id -> conf
  conf -> dot
  dot -> "Profit"
  dot -> "Success" [style=dotted]
  }
#+END_SRC

Now press C-c C-c to evaluate this code. Emacs will generate the configured output file dot_success.png and then link to it in an automatically created #+RESULTS section right below it.

You can press C-c C-x C-v to toggle display of inline images to see it directly in Emacs. Alternatively, use M-x org-display-inline-images to switch this on. Whenever you change the DOT source code (press C-c ‘ to edit the dot source code in a separate buffer), just press C-c C-c to re-execute the updated source.

In any exported documents (for examlpe this blog post), only the output graph itself will appear, like this:

Exported as a PDF with C-e l o it looks like this.

In your Emacs, it should look like this:

This approach should work for any of the many languages supported by org-babel. Let me know in the comments what you come up with!

Getting Django Rest Framework to parse docstrings as reStructuredText

The Django REST Framework is awesome, for a whole bunch of reasons, one of them being the browsable HTML version of your API that it automatically generates for you.

As a part of this, it extracts any docstrings that you might have defined for the relevant class (ViewSet or CBV) and adds an HTML version of this documentation to the browsable API, like this:

However, as described in the relevant documentation, it expects Markdown syntax by default. I like Markdown, but the rest of my Python docstrings are all in reStructuredText, which is the default documentation format for many Python projects.

In short, I would prefer to type my view docstrings in reStructuredText and not in the DRF-default markdown, because reStructuredText is what all of my other non-DRF code is using.

Fortunately, DRF has made this relatively easily configurable, if you’re not afraid to do some searching and digging (I couldn’t find a single complete worked-out example). Especially the code for transforming RST text into parsed HTML is not as straight-forward as one would have hoped. The DRF part of the deal is clearly described in its documentation.

Here’s how you can do this:

Define a new DRF view description function

Add the following function to your code somewhere. I added it to the views module in a Django app with reusable library code:

from django.utils.safestring import mark_safe
from docutils import core
from rest_framework.compat import smart_text
from rest_framework.utils import formatting

def get_view_description(view_cls, html=False):
    """Alternative view description function to be used as the DRF
    ``VIEW_DESCRIPTION_FUNCTION`` so that RestructuredText can be used
    instead of the DRF-default MarkDown.

    Except for the RST parts, derived by cpbotha@vxlabs.com from the
    DRF default get_view_description function.

    """

    description = view_cls.__doc__ or ''
    description = formatting.dedent(smart_text(description))
    if html:
        # from https://wiki.python.org/moin/ReStructuredText -- we use the
        # third recipe to get just the HTML parts corresponding to the ReST
        # docstring:
        parts = core.publish_parts(source=description, writer_name='html')
        html = parts['body_pre_docinfo']+parts['fragment']
        # have to use mark_safe so our HTML will get explicitly marked as
        # safe and will be correctly rendered
        return mark_safe(html)

    return description

Configure the view description function

In your settings, activate the new view description function:

REST_FRAMEWORK = {
    # other REST_FRAMEWORK config here...
    'VIEW_DESCRIPTION_FUNCTION': 'your_app.views.get_view_description'
}

Enjoy your reStructuredText

If you now reload any of your browsable API pages, you should see your reStructuredText nicely parsed.