Bootstrap Problems in pex Packages

[ packaging pex python ]

In this post, I laid out an initial attempt at a pex workflow for packaging data analysis code for easy use by others. I still stand by the method in general; however, I ran into some problems with this particular application.

For reasons I don’t fully understand (and which MAY be fixable with, say, the correct pex options, or a more tightly/fully specified requirements-pex.txt, or a precise Python version match, or …), when unpacking matplotlib on a user’s machine, pex fails to populate various critical resource directories (at all? in the right sequence?). This leads to a FileNotFoundError being raised during what appears to be the initial activation of the packaged environment:

Traceback (most recent call last):
  File ".bootstrap\_pex\", line 347, in execute
  File ".bootstrap\_pex\", line 90, in _activate
  File ".bootstrap\_pex\", line 163, in activate
  File ".bootstrap\_pex\", line 217, in _activate
  File ".bootstrap\_pex\", line 152, in update_candidate_distributions
  File ".bootstrap\_pex\", line 122, in load_internal_cache
  File ".bootstrap\_pex\", line 109, in write_zipped_internal_cache
  File ".bootstrap\_pex\", line 178, in cache_distribution
  File ".bootstrap\_pex\", line 141, in safe_open
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\Staff\\.pex\\install\\matplotlib-2.1.2-cp36-cp36m-win_amd64.whl.08a6cc1638139d3fe5ee428cafc4e5e73177805e.835acc1b95bb4add8dcb5d93bf8c585b\\matplotlib-2.1.2-cp36-cp36m-win_amd64.whl\\matplotlib/backends/web_backend/jquery/css/themes/base/images/ui-bg_diagonals-thick_18_b81900_40x40.png'

So, not good. On trying to trace down the error in a bit more detail, I actually had trouble making a simpler .pex work, containing just matplotlib (and its dependencies)… here, it blew up even earlier in the process, during bootstrap, even when run in the same virtualenv within which the .pex was initially created:

Traceback (most recent call last):
  File "C:\Users\Brian\AppData\Local\Programs\Python\Python36\lib\", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\Brian\AppData\Local\Programs\Python\Python36\lib\", line 85, in _run_code
    exec(code, run_globals)
  File "", line 23, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File ".bootstrap\pex\third_party\", line 400, in <module>
  File ".bootstrap\pex\third_party\", line 396, in install
  File ".bootstrap\pex\third_party\", line 207, in install_vendored
  File ".bootstrap\pex\third_party\", line 242, in install
  File ".bootstrap\pex\third_party\", line 155, in _iter_importables
  File ".bootstrap\pex\third_party\", line 110, in iter_root_modules
  File ".bootstrap\pex\third_party\", line 121, in _filter_names
  File "C:\Users\Brian\AppData\Local\Programs\Python\Python36\lib\", line 233, in compile
    return _compile(pattern, flags)
  File "C:\Users\Brian\AppData\Local\Programs\Python\Python36\lib\", line 301, in _compile
    p = sre_compile.compile(pattern, flags)
  File "C:\Users\Brian\AppData\Local\Programs\Python\Python36\lib\", line 562, in compile
    p = sre_parse.parse(p, flags)
  File "C:\Users\Brian\AppData\Local\Programs\Python\Python36\lib\", line 855, in parse
    p = _parse_sub(source, pattern, flags & SRE_FLAG_VERBOSE, 0)
  File "C:\Users\Brian\AppData\Local\Programs\Python\Python36\lib\", line 416, in _parse_sub
    not nested and not items))
  File "C:\Users\Brian\AppData\Local\Programs\Python\Python36\lib\", line 502, in _parse
    code = _escape(source, this, state)
  File "C:\Users\Brian\AppData\Local\Programs\Python\Python36\lib\", line 401, in _escape
    raise source.error("bad escape %s" % escape, len(escape))
sre_constants.error: bad escape \p at position 11

A similar explosion happened with a .pex containing just requests and dependencies. So, seems pex is not as straightforward as I first thought. Too bad, it seemed very promising!

Written on January 10, 2019