How to debug PyInstaller DLL / PYD load failed issues on Windows

TL;DR

When debugging DLL load errors on Windows, use lucasg’s open source and more modern rewrite of the old Dependency Walker software. Very importantly, keep on drilling down through indirect dependencies until you find the missing DLLs.

The Problem

Recently I had to package up a wxPython and VTK-based app for standalone deployment on Windows. Because of great experience with PyInstaller, I opted to use this tool.

With the first try with the freshly built package on the deployment machine, it refused to start up due to an ImportError: DLL load failed: The specified module could not be found., and specifically with the vtk.vtkCommonCorePython.pyd Python extension DLL.

What was frustrating, is that the relevant file was definitely present and in the right place, namely the same folder as the exe file.

A test app that imports only wx (4.0.0rc) and vtk (8.0.1) generated the following traceback:

[3096] LOADER: Running pyiboot01_bootstrap.py
[3096] LOADER: Running t1.py
Traceback (most recent call last):
  File "site-packages\vtk\vtkCommonCore.py", line 5, in <module>
  File "C:\Users\cpbotha\Miniconda3\envs\env1\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 718, in load_module
ImportError: DLL load failed: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "t1.py", line 9, in <module>
  File "C:\Users\cpbotha\Miniconda3\envs\env1\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 631, in exec_module
  File "site-packages\vtk\__init__.py", line 41, in <module>
  File "C:\Users\cpbotha\Miniconda3\envs\env1\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 631, in exec_module
  File "site-packages\vtk\vtkCommonCore.py", line 9, in <module>
  File "C:\Users\cpbotha\Miniconda3\envs\env1\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 718, in load_module
ImportError: DLL load failed: The specified module could not be found.
[3096] Failed to execute script t1
[3096] LOADER: OK.
[3096] LOADER: Cleaning up Python interpreter.

Digging in using pdb and ctypes.WinDLL

By adding import pdb; pdb.set_trace() at a strategic point, I could use the Python debugger to try and investigate why it was not able to load the DLL which was clearly in the right place.

First I confirmed that it could load other namespaced PYDs in the same directory:

(Pdb) import ctypes
(Pdb) ctypes.WinDLL("wx.siplib.pyd")
<PyInstallerWinDLL 'C:\Users\cpbotha\Downloads\t1\wx.siplib.pyd', handle 7fff66320000 at 0x151a6781048>

However, using the same invocation on the offending PYD would still raise an error. By the way, I had to disable PyInstaller’s silly exceptions, because they masked the underlying OSError exception.

Great deal of good that did me, because Windows 10 gets me no additional information other than reporting “error 126” on the top-level DLL. Why can’t such a mature system not offer a little more guidance?

(Pdb) ctypes.WinDLL("vtk.vtkCommonCorePython.pyd")
>>> OSError: [WinError 126] The specified module could not be found
(Pdb) ctypes.WinDLL("C:\\Users\\cpbotha\\Downloads\\t1\\vtk.vtkCommonCorePython.pyd")
>>> OSError: [WinError 126] The specified module could not be found
(Pdb) import os
(Pdb) os.path.exists("C:\\Users\\cpbotha\\Downloads\\t1\\vtk.vtkCommonCorePython.pyd")
True
(Pdb) ctypes.WinDLL("vtkCommonCorePython.pyd")
>>> OSError: [WinError 126] The specified module could not be found
(Pdb) os.path.exists("vtkCommonCorePython.pyd")
True

I did spend more time than I should have in pdb tracing through the whole complicated PyInstaller and Python import code innards. The fact that some PYDs loaded and some did not should have more quickly pushed me in the direction of nested dependencies.

Seeing the light with lucasg’s Dependencies

Although I had at an earlier stage checked DLL loading first with Dependency Walker (this gets very confused by the new Windows api-ms-win-* DLLs) and later with lucasg’s improved utility, I did not drill down far enough into the dependency tree.

It’s important to keep on drilling down until you see missing DLLs, the application won’t automatically traverse the tree.

Anyways, drilling down from the offending vtk.vtkCommonCorePython.pyd soon enough led me to the culprit: The Intel Threading Building Blocks (TBB) DLL conda package I was using was accidentally built in debug mode, and the debug runtimes it relied one were obviously not being deployed:

screenshot_2017-12-06_13-52-48.png

After switching to a TBB conda package from a different channel, the app was finally able to start up and run on the deployment machine.

Fixed position of any Scatter child widget in Kivy

Just in case you were wondering how one could go about having any child widget of a Kivy Scatter widget that would stick in a single window-relative position whilst the scatter itself was being translated and rotated, I thought I’d post this solution.

With a scatter, one can very easily implement a canvas-like object that can be rotated, translated and zoomed, whilst all of the widgets it contains are transformed along with it. This is usually great, unless you’d like, for some or other reason, to display a widget at a fixed position, for example any kind of overlay, such as a label.

The short answer is that this can be done by using a MatrixInstruction that is executed before that widget is drawn (in the canvas.before context) and that effectively reverses the parent Scatter’s transformation.

The slightly more involved answer is the minimal working example below. First we have the Python part of the ScatterOverlayApp. This only defines the MyScatter class which we’ll flesh out in the KV file, and the main app class which instantiates the top-level widget, in our case MyScatter:

from kivy.app import App
from kivy.uix.scatter import Scatter

class MyScatter(Scatter):
    pass

class ScatterOverlayApp(App):
    def build(self):
        return MyScatter()

if __name__ == '__main__':
    ScatterOverlayApp().run()

In the KV Language part of this example, the <MyScatter> rule defines two buttons, and then the Label which we shall configure to be the fixed position overlay. The Label stores the current matrix, and then applies the inverse transformation matrix of its parent, in other words that of the MyScatter.

This will result in the widget drawing in the space of MyScatter‘s parent, in our case the window. So when we specify the label to be in the middle of the root widget, it will actually be drawn right in the middle of the window. Even when we rotate and scale MyScatter (see screenshot below), the label will remain exactly where we put it.

<MyScatter>:
    Button:
        text: "Click me"
        pos: 100, 100

    Button:
        text: "Don't click me"
        pos: 150, 200

    Label:
        pos: root.width / 2.0, root.height / 2.0
        text: "I am an overlay label!"
        font_size: 32

        canvas.before:
            # store current matrix
            PushMatrix
            MatrixInstruction:
                # reverse MyScatter's transformation matrix
                matrix: self.parent.transform_inv

        canvas.after:
            # restore matrix so that other widgets draw normally
            PopMatrix

Here’s a screenshot showing the situation after the containing MyScatter widget has been rotated, translated and zoomed. Both the buttons it contains have also been transformed, but the label is exactly where we placed it initially.

scatteroverlay.jpg

Getting Django Rest Framework to parse docstrings as reStructuredText

The Django REST Framework is awesome, for a whole bunch of reasons, one of them being the browsable HTML version of your API that it automatically generates for you.

As a part of this, it extracts any docstrings that you might have defined for the relevant class (ViewSet or CBV) and adds an HTML version of this documentation to the browsable API, like this:

However, as described in the relevant documentation, it expects Markdown syntax by default. I like Markdown, but the rest of my Python docstrings are all in reStructuredText, which is the default documentation format for many Python projects.

In short, I would prefer to type my view docstrings in reStructuredText and not in the DRF-default markdown, because reStructuredText is what all of my other non-DRF code is using.

Fortunately, DRF has made this relatively easily configurable, if you’re not afraid to do some searching and digging (I couldn’t find a single complete worked-out example). Especially the code for transforming RST text into parsed HTML is not as straight-forward as one would have hoped. The DRF part of the deal is clearly described in its documentation.

Here’s how you can do this:

Define a new DRF view description function

Add the following function to your code somewhere. I added it to the views module in a Django app with reusable library code:

from django.utils.safestring import mark_safe
from docutils import core
from rest_framework.compat import smart_text
from rest_framework.utils import formatting

def get_view_description(view_cls, html=False):
    """Alternative view description function to be used as the DRF
    ``VIEW_DESCRIPTION_FUNCTION`` so that RestructuredText can be used
    instead of the DRF-default MarkDown.

    Except for the RST parts, derived by cpbotha@vxlabs.com from the
    DRF default get_view_description function.

    """

    description = view_cls.__doc__ or ''
    description = formatting.dedent(smart_text(description))
    if html:
        # from https://wiki.python.org/moin/ReStructuredText -- we use the
        # third recipe to get just the HTML parts corresponding to the ReST
        # docstring:
        parts = core.publish_parts(source=description, writer_name='html')
        html = parts['body_pre_docinfo']+parts['fragment']
        # have to use mark_safe so our HTML will get explicitly marked as
        # safe and will be correctly rendered
        return mark_safe(html)

    return description

Configure the view description function

In your settings, activate the new view description function:

REST_FRAMEWORK = {
    # other REST_FRAMEWORK config here...
    'VIEW_DESCRIPTION_FUNCTION': 'your_app.views.get_view_description'
}

Enjoy your reStructuredText

If you now reload any of your browsable API pages, you should see your reStructuredText nicely parsed.