Some recipes for AWS Lambda and Docker Python artefacts (including cross-building from macOS)

[huonw]

Questions often come up about building AWS Lambda and docker artefacts effectively on Slack. I've answered a few based on my experience using pants to deploy AWS Lambda (via zips) and Fargate for a while.

This discussion post collects what is currently working for us to deploy dozens of lambdas and a handful of Fargate containers, to be a reference that I can link to. None of this is perfect, but it's been good enough, and, once we worked through the initial set-up, pants packaging has never been the cause of a deploy failure.

We develop on ARM64 macOS, and deploy to x86-64 Linux. We run pants natively, and try to do as much as work natively too as reasonable because that is noticeably more efficient than emulating the OS, and very much more efficient than emulating the architecture.

We also implemented all this well before Pants 2.15 and its experimental environments feature (https://blog.pantsbuild.org/environments-simpler-multi-platform-workflows/): that can likely be used too, but I cannot report on that.

Basic setup

(UPDATE: With newer Pants, this isn't required. They use a new-enough PEX by default.)

Old requirements for Pants 2.14

We're currently using Pants 2.14 and PEX v2.1.128. Upgrading the latter is required to pick up a few bug fixes that were affecting our Lambda and docker builds. Our pants.toml looks something like:

[GLOBAL]
pants_version = "2.14.0"

backend_packages = [
  "pants.backend.python",
  "pants.backend.awslambda.python",
  "pants.backend.docker",
  ...
]

...

[pex-cli]
version = "v2.1.128"
known_versions = [
    "v2.1.128|macos_arm64|a48a461a9e9f490476aa75976dbe58d3263691a8895a53d11fdaaf8a3bc223a5|4081603",
    "v2.1.128|macos_x86_64|a48a461a9e9f490476aa75976dbe58d3263691a8895a53d11fdaaf8a3bc223a5|4081603",
    "v2.1.128|linux_x86_64|a48a461a9e9f490476aa75976dbe58d3263691a8895a53d11fdaaf8a3bc223a5|4081603",
    "v2.1.128|linux_arm64|a48a461a9e9f490476aa75976dbe58d3263691a8895a53d11fdaaf8a3bc223a5|4081603"
]

...

Python AWS Lambda zip artefacts

UPDATE: for Pants 2.18, this can be simplified fairly significantly, but we haven't applied it locally yet. The instructions in https://www.pantsbuild.org/docs/awslambda-python now see most of the benefits here.

Most important: use the complete_platforms= parameter, NOT runtime=

• We build dozens of lambdas via the built-in python_awslambda target: https://www.pantsbuild.org/docs/awslambda-python
• The most important part of getting this working reliably is using a 'complete platform' JSON file instead of the runtime parameter, to tell PEX about the environment:
  • This allows pants/PEX to select the appropriate dependencies, even when building on a different platform (like macOS vs. Linux). This does require that dependencies are available as a wheel (bdist) and/or are pure Python, because platform-specific dependencies cannot be cross-built. This has worked fine for us in practice, with the dependencies we use.
  • We've generated an appropriate file (many months ago) by running the following in a Lambda (in retrospect, I imagine it could've been run in a one of the AWS-provided docker images, but... this post is just describing what is working for us):

    import subprocess
    
    def lambda_handler(event, context):
        subprocess.run(
            """
            pip install --target=/tmp/subdir pex
            PYTHONPATH=/tmp/subdir /tmp/subdir/bin/pex3 interpreter inspect --markers --tags
            """,
            shell=True
        )
        return {'statusCode': 200, 'body': "{}"}

  • The output is put into 3rdparty/platforms/aws_lambda_python_3_9.json with a BUILD file:

    # 3rdparty/platforms/BUILD
    file(
        name="aws_lambda_python_3_9",
        source="aws_lambda_python_3_9.json",
    )

  • For convenience, here's the literal file we're using (NB. this is Python 3.9 in ap-southeast-2, as of 8 months ago, so your environment may differ): lambda-complete-platform.zip
• To actually build the lambdas we have a macro:

  def your_name_here_python_awslambda(*, **kwargs):
      kwargs.setdefault("dependencies", []).extend(
          [
              # FIXME https://github.com/pantsbuild/pants/issues/15454
              "!!path/to/requirements#mypy",
              "!!path/to/requirements#sqlalchemy-stubs",
          ]
      )
      kwargs["complete_platforms"] = ["3rdparty/platforms:aws_lambda_python_3_9"]  
      python_awslambda(**kwargs)

  • The exclusions are working around Stop including type stubs in pex_binary targets #15454, to reduce the size of the deployed package. You may need to adjust them, based on what packages you use.
• This can be called like the normal python_awslambda target, and generates the same sort of output (a zip in dist/): (As above, note that the runtime= parameter is not used.)

  # some/code/BUILD
  your_name_here_python_awslambda(
      name="some_name",
      handler="path/to/file.py:your_handler",
  )

• We deploy these via CDK, outside of pants, using a construct that takes constructs the appropriate dist/some.code/some_name.zip path to pass to lambda.Code.fromAsset:

  pants package ::
  cdk deploy path/to/stack.js

• Things we'd like to improve, eventually:
  • We'd prefer to deploy dependencies in a layer and then only our first-party source in the direct lambda package (Allow building AWS Lambda Layers #18880)
  • Stop including type stubs in pex_binary targets #15454
  • Using the fact that the target environment is static to layout an appropriate .zip file, and avoid the lambdex/PEX dynamic environment detection at start up (this adds about 1s to the cold start, it seems) (Construct Python serverless artefacts (AWS Lambda etc.) in expected layout, without PEX #18879)
  • Improving build-time/cacheability: the current all-in-one zip files means that a minor change (adding a comment...) to first-party code that affects a bunch of lambdas change might have to generate a fresh 300MB of zips, which both takes time and a lot of cache space. The first point above likely improves this: splitting 1st and 3rd party packages into separate packages.

Docker image artefacts from PEX

• We build docker images inspired by https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/
• We only build them via pants, and do not push them, and instead re-tag them outside pants before pushing, as we haven't had time/strong-need to work through parametrising different registries for prod vs. development images.
• Again, we use complete_platforms to be able to build PEX files natively on macOS:
  • we currently run things in python:3.9-bullseye containers, and thus have added docker_python_3_9_bullseye_arm64.json and docker_python_3_9_bullseye_amd64.json files to 3rdparty/platforms; here's the BUILD file including how we generate them:

    # docker run -it --platform linux/arm64 python:3.9.10-bullseye bash -c 'pip install pex; pex3 interpreter inspect --markers --tags'
    # docker run -it --platform linux/amd64 python:3.9.10-bullseye bash -c 'pip install pex; pex3 interpreter inspect --markers --tags'
    files(
        name="docker_python_3_9_bullseye",
        sources=["docker_python_3_9_bullseye_*.json"],
    )

  • We run Linux/arm64 images on dev machines and Linux/amd64 images on CI/in production, and thus include complete platforms for both (running amd64 images on arm64 dev machines is very slow)
• To actually build images we have a macro:

  BASE_IMAGE = "python:3.9-bullseye"
  COMPLETE_PLATFORMS = ("3rdparty/platforms:docker_python_3_9_bullseye",)
  
  
  def docker_image_from_pex(
      *,
      name,
      pex_entry_point,
      extra_instructions,
      pex_dependencies=None,
      **kwargs,
  ):
      """
      Build a docker image using PEX.
  
      We use ubuntu as a base operating system (as distinct from alpine), because we use large binary
      libraries like pandas and numpy.
  
      We use the pattern of having a single web server process, and let the container orchestration
      system (ALB + ECS) deal with all process/container management.
  
      Based on the micromachines sample dockerfile (MIT licensed), at
      https://github.com/Oovvuu/micromachines/blob/main/gunicorn-service.dockerfile,
      and
      https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/,
      (except we don't bother with separate docker_image()s as they're all
      ephemeral on CI anyway)
      """
      if pex_dependencies is None:
          pex_dependencies = []
  
      deps = pex_binary(
          name=f"_{name}-deps",
          entry_point=pex_entry_point,
          complete_platforms=COMPLETE_PLATFORMS,
          include_sources=False,
          include_requirements=True,
          include_tools=True,
          # significantly faster to splat .whl files around as is, than unzip/rezip ("zipapp",
          # default) or unzip individually ("loose")
          layout="packed",
          dependencies=[
              *pex_dependencies,
              # FIXME https://github.com/pantsbuild/pants/issues/15454
              "!!backend#mypy",
              "!!backend#sqlalchemy-stubs",
              "!!backend#types-pyyaml",
              "!!backend#types-python-dateutil",
              "!!backend#types-pytz",
              "!!backend#types-s3transfer",
          ],
      )
  
      src = pex_binary(
          name=f"_{name}-srcs",
          entry_point=pex_entry_point,
          complete_platforms=COMPLETE_PLATFORMS,
          include_sources=True,
          include_requirements=False,
          include_tools=True,
          layout="packed",
          dependencies=pex_dependencies,
      )
  
      output_directory = ".".join(build_file_dir().parts)
  
      docker_image(
          name=name,
          instructions=[
              # unpack the external dependencies
              f"FROM {BASE_IMAGE} AS deps",
              f"COPY {output_directory}/{deps.name}.pex /deps.pex",
              "RUN PEX_TOOLS=1 PEX_VERBOSE=2 python /deps.pex venv --scope=deps --compile --collisions-ok /opt/venv",
              # unpack our code
              f"FROM {BASE_IMAGE} AS srcs",
              f"COPY {output_directory}/{src.name}.pex /srcs.pex",
              "RUN PEX_TOOLS=1 PEX_VERBOSE=2 python /srcs.pex venv --scope=srcs --compile /opt/venv",
              # set up the actual app
              f"FROM {BASE_IMAGE} AS app",
              # general config
              "ENV PYTHONUNBUFFERED=1",
              # use a non-root user
              "RUN adduser --disabled-login appuser",
              "WORKDIR /home/appuser",
              "USER appuser",
              # dependencies
              'ENV PATH="/opt/venv/bin:$PATH"',
              "COPY --from=deps /opt/venv /opt/venv",
              # our source
              "COPY --from=srcs /opt/venv /opt/venv",
              *extra_instructions,
          ],
          **kwargs,
      )

  • As above, this includes a workaround for Stop including type stubs in pex_binary targets #15454
  • We don't bother with the separate images described in https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/ because we don't really prune things anyway 🤷‍♂️
• This can be invoked like:

  docker_image_from_pex(
      name="some_name",
      pex_entry_point="app.py",
      pex_dependencies=[
          "some/extra/dependency.py",
      ],
      extra_instructions=[
          # how to actually run it
          "COPY path/to/run-service.sh ./run-service.sh",
          'CMD ["./run-service.sh"]',
      ],
      # FIXME https://github.com/pantsbuild/pants/issues/16561
      dependencies=["path/to/run-service.sh"],
  )

• Our pants.toml configuration is:

  [docker]
  env_vars.add = [
      # faster, better output
      "DOCKER_BUILDKIT=1",
  ]
  
  [docker.registries.local]
  # dummy address that we control, so we're not accidentally pulling public images
  address = "local.yourcompany.com"
  default = true
  skip_push = true

• CI runs the equivalent of:

  pants package ::
  docker tag local.yourcompany.com/some_name:latest actual-registry.com/actual-repository:actual-tag
  docker push actual-registry.com/actual-repository:actual-tag

• Limitations/things we'd like to improve:
  • Build speed/cacheability, e.g. similar to a lambda, we could potentially lay out the appropriate code layout natively, and then the docker build could just be some COPYs, without running PEX_TOOLS=1 ... themselves (integrates better with the pants cache), and/or Support for use docker buildx / BuildKit features #15199 (integrates better with the Docker layer cache)
  • Build/pushing the appropriately tagged images directly (I think this just requires taking a bit of time to implement the prod vs. dev conditional behaviour, e.g. via pants.ci.toml)
  • Stop including type stubs in pex_binary targets #15454
  • Generalize docker_image dependency inference to cover source files (i.e. file, shell_source) #16561