Skip to main content
Terraform Utility Modules 0.10.8Last updated in version 0.9.6

Prepare PEX Environment Module

View Source Release Notes

NOTE: This module should not be used directly. Use run-pex-as-data-source or run-pex-as-resource instead.

This module can be used to prepare an runtime environment that can call out to a PEX binary. Specifically, this module:

  • Selects the right PEX binary to use based on the installed python version.
  • Sets up a platform portable PYTHONPATH environment variable for the PEX, as well as the module.
  • Provides an entrypoint script that will parse args in a platform portable manner.

This module uses Python under the hood so, the Python must be installed on the OS.

What is PEX?​

PEX (or Python EXecutable) is an executable python environment in the spirit of virtualenvs. It is generated using the pex library, and is an executable zip file containing:

  • An bootstrap script in python that unpacks the requirements and includes them in the PYTHONPATH (sys.path).
  • Dependencies packaged as wheels for each platform and python version the executable is intended to support.
  • (Optionally) An entrypoint script to run in the context of the unpacked environment.

This provides a convenient way to package python dependencies in a portable manner, allowing execution of the script without the end user having to install all the necessary dependencies.

How to construct a PEX binary​

The PEX binary is generated using the pex tool. To package your script as a PEX for compatibility with this module, you need the following:

  • Python requirements defined as a requirements.txt file using the Pip requirements file syntax.
  • An entrypoint script that should be run in the context of the environment. This should provide the actual logic you wish to provide in terraform (a terraform local-exec provisioner or data source script)

Then, you call out to pex and provide the platform and python versions you wish to support:

pex --python-shebang='/usr/bin/env python' \
    -r requirements.txt \
    --python=python2.7 \
    --platform macosx_10.12-x86_64 \
    --platform macosx_10.13-x86_64 \
    --platform macosx_10.14-x86_64 \
    --platform linux-x86_64 \
    --platform linux-x86_64-cp-27-mu \
    --platform win32 \
    --platform win_amd64 \
    --disable-cache \
    -o ../bin/sample_python_script_py27_env.pex

This will search pypi for the python packages defined in requirements.txt that support the specified platform and python versions, download the wheel/package, inject the bootstrap script and produce an executable zip file.

See the pex/sample-python-script example for an example implementation that you can use as a template.

Known limitations of PEX​

  • For compiled dependencies, PEX relies on pre-built wheel packages to avoid cross compilation. What this means is that the chosen pypi needs to hold a wheel for each compatible platform. For example, compare the available package files for ruamel.yaml with pyyaml. In ruamel.yaml, there is a wheel for each permutation of major platform and major python versions supported. However, for pyyaml, there is only wheels provided for Windows. This means that you need to setup a cross compiler for each compatible platform you wish to support.

    • To avoid this, prefer packages that only rely on python code, or that have prebuilt wheels for the platforms you wish to support.

  • When building a PEX for both python2 with python3, be aware that python2 packages tend to rely on backports of stdlib enhancements (e.g the futures package). These backports cannot be installed in a python3 environment, which creates complications in the PEX binary. Specifically, it will attempt to lookup a dependency that doesn't exist in the packaged zip.

    • You can mitigate this using environment markers. However, for the most robust solution, generate separate PEX binaries for python2 and python3.

  • The PEX binary is directly executable in Unix environments (Linux or Mac OS X). However, for all environments, it can be run using python (e.g python my.pex).

  • There is a known limitation in Windows environments with python 2 that prevents the usage of pex in directories with a long path name. The pex pathing for imports does not seem to support long path names on Windows, hitting the 255 character limitation of path names.

Sample Usage​

main.tf

# ------------------------------------------------------------------------------------------------------
# DEPLOY GRUNTWORK'S PREPARE-PEX-ENVIRONMENT MODULE
# ------------------------------------------------------------------------------------------------------

module "prepare_pex_environment" {

  source = "git::git@github.com:gruntwork-io/terraform-aws-utilities.git//modules/prepare-pex-environment?ref=v0.10.8"

  # ----------------------------------------------------------------------------------------------------
  # REQUIRED VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # Parts of the path (folders and file names) to the python package directory
  # housing the pex file.
  pex_module_path_parts = <list(string)>

  # Parts of the path (folders and files names) to the PEX executable for python
  # as a list of strings.
  python_pex_path_parts = <list(string)>

}


Reference​

Required​

pex_module_path_partslist(string)required

Parts of the path (folders and file names) to the python package directory housing the pex file.

python_pex_path_partslist(string)required

Parts of the path (folders and files names) to the PEX executable for python as a list of strings.