pyproject-installer

This tool is intended for build, install, run or management of dependencies sources of Python project in source tree within network-isolated environments.

Description

• Supported platform: Unix.
  Currently, platform-specific parts:

  • pipe is used for calling build backend hooks in subprocess
  • script wrappers are generated only for Unix systems

• OS environment of this project is a network-isolated environment, which implies that a local loopback interface is the only available network interface. Thus, pyproject-installer doesn't perform any network activity (e.g. it doesn't install build dependencies specified via PEP518 configuration or PEP517's get_requires_for_* hooks). This also makes it difficult or impossible to create an isolated Python environment for calling build backend hooks specified in PEP517, therefore, current Python environment is the only available environment.

• Source tree can be either checkout of VCS or unpacked source distribution.

• An installation result will be consumed by external tool like RPM.
  The main usage of pyproject-installer looks like:

  external tool => (pyproject-installer: build => install to destdir) => external tool packages destdir
  

  Therefore, there is no need to build intermediate source distribution for build wheel, only build_wheel backend's hook is actually called.

  Note: an installation into Python virtual environment is also supported, but only the manual uninstallation of such packages is possible (tools should refuse an uninstallation of distribution with missing RECORD file).

• Only stdlib or vendored dependencies can be used in runtime for bootstrapping any Python project.
  Current vendored packages:

  • tomli (used for parsing pyproject.toml configuration file). Note: tomli is the part of stdlib since Python 3.11.
  • packaging (used for parsing PEP508 dependencies)

• Installation of build dependencies is up to the caller.
  These dependencies of Python projects are managed externally with system package managers like apt or dnf. External source of upstream's dependencies may be used for provision of formatted list of dependencies to external tools.

• There is no post-installation bytecompilation.
  PEP427 says that wheel installers should compile any installed .py to .pyc. External tools like RPM already provide Python bytecompilation means, which compile for multiple optimization levels at a time. No point to compile modules twice.

• RECORD file is not installed.
  https://peps.python.org/pep-0627/#optional-record-file:

  Specifically, the RECORD file is unnecessary when projects are installed by a Linux system packaging system, which has its own ways to keep track of files, uninstall them or check their integrity. Having to keep a RECORD file in sync with the disk and the system package database would be unreasonably fragile, and no RECORD file is better than one that does not correspond to reality.

• INSTALLER file is not installed by default(optional).
  https://peps.python.org/pep-0627/#optional-installer-file:

  The INSTALLER file is also made optional, and specified to be used for informational purposes only. It is still a single-line text file containing the name of the installer.

  https://packaging.python.org/en/latest/specifications/recording-installed-packages/#the-installer-file:

  This value should be used for informational purposes only. For example, if a tool is asked to uninstall a project but finds no RECORD file, it may suggest that the tool named in INSTALLER may be able to do the uninstallation.

• Built distribution can be checked within Python virtual environment with the help of run command.

• Project's dependencies sources can be managed (i.e. stored, synced, verified or evaluated) with the help of deps command.

Usage

Build

Build project from source tree in current Python environment according to PEP517. This doesn't trigger installation of project's build dependencies.

python -m pyproject_installer build

Build positional arguments:

description: source directory
default: current working directory
example: python -m pyproject_installer build .

Build options:

name: --outdir OUTDIR, -o OUTDIR
description: output directory for built wheel
default: {srcdir}/dist
example: python -m pyproject_installer build --outdir ~/outdir

Upon successful build pyproject_installer dumps wheel filename into {OUTDIR}/.wheeltracker.

name: --sdist
description: build source distribution(sdist) instead of binary
one(wheel).
note: installer supports only wheel format.
default: build wheel
example: python -m pyproject_installer build --sdist

name: --backend-config-settings BACKEND_CONFIG_SETTINGS
description: ad-hoc configuration for build backend as dumped JSON dictionary
default: None

Example of passing config_settings for setuptools>=64.0.0:
python -m pyproject_installer build --backend-config-settings='{"--build-option": ["--python-tag=sometag", "--build-number=123"]}'

Example of passing config_settings for setuptools<64.0.0:
python -m pyproject_installer build --backend-config-settings='{"--global-option": ["--python-tag=sometag", "--build-number=123"]}'

Example of passing config_settings for pdm backend:
python -m pyproject_installer build --backend-config-settings='{"--python-tag": "sometag"}'

Install

Install project built in wheel format. This doesn't trigger installation of project's runtime dependencies.

python -m pyproject_installer install

Install positional arguments:

description: wheel file to install
default: contructed as directory {cwd}/dist and wheel filename read from
{cwd}/dist/.wheeltracker
example: python -m pyproject_installer install wheel.whl

Install options:

name: --destdir DESTDIR, -d DESTDIR
description: Wheel installation root will be prepended with destdir
default: /
example: python -m pyproject_installer install --destdir ~/destdir

name: --installer INSTALLER
description: Name of installer to be recorded in dist-info/INSTALLER
default: None, INSTALLER will be omitted
example: python -m pyproject_installer install --installer custom_installer

name: --no-strip-dist-info
description: Don't strip dist-info. By default only METADATA
and entry_points.txt files are allowed in dist-info directory.
note: RECORD is unconditionally filtered out.
default: False
example: python -m pyproject_installer install --no-strip-dist-info

Run

Run command within Python virtual environment that has access to system and user site packages, their console scripts and installed built package.

python -m pyproject_installer run

Run positional arguments:

description: command to run within virtual environment
example: python -m pyproject_installer run pytest

Dash note:

https://docs.python.org/3/library/argparse.html#arguments-containing If you have positional arguments that must begin with - and don't look like negative numbers, you can insert the pseudo-argument '--' which tells parse_args() that everything after that is a positional argument:

python -m pyproject_installer run -- pytest -vra

Run options:

name: --wheel WHEEL
description: wheel file to install into virtual environment
default: contructed as directory {cwd}/dist and wheel filename read from
{cwd}/dist/.wheeltracker
example: python -m pyproject_installer run --wheel wheel.whl pytest

Note: venv's directory name is .run_venv.

Management of dependencies sources

Collect PEP508 requirements from different sources, store and evaluate them in Python environment.

python -m pyproject_installer deps --help

Common deps options:

name: --depsconfig
description: configuration file to use
default: {cwd}/pyproject_deps.json
example: python -m pyproject_installer deps --depsconfig foo.json

deps subcommands

name: show
description: show configuration and data of dependencies's sources
example: python -m pyproject_installer deps show --help

Positional arguments:

description: source names
default: all
example: python -m pyproject_installer deps show build

---

name: add
description: configure source of Python dependencies. Supported sources: standardized formats like PEP517, PEP518, PEP735 or core metadata are fully supported, while tool-specific formats like pip, tox, poetry, hatch, pdm or pipenv have limited support.
example: python -m pyproject_installer deps add --help

Positional arguments:

description: source name

description: source type
choice: pep517, pep518, pep735, metadata, pip_reqfile, poetry, tox, hatch, pdm, pipenv

description: specific configuration options for source
default: []

examples:
Configuration of source of PEP518 dependencies:
python -m pyproject_installer deps add build_pep518 pep518

Configuration of source of PEP517 dependencies:
python -m pyproject_installer deps add build_pep517 pep517

Configuration of source of metadata dependencies:
python -m pyproject_installer deps add runtime metadata

Configuration of source of PEP735 dependencies:
python -m pyproject_installer deps add check pep735 test

Configuration of source of pip requirements:
python -m pyproject_installer deps add check pip_reqfile requirements.txt

Configuration of source of tox requirements:
python -m pyproject_installer deps add check tox tox.ini testenv

Configuration of source of poetry requirements:
python -m pyproject_installer deps add check poetry dev

Configuration of source of hatch requirements:
python -m pyproject_installer deps add check hatch hatch.toml test

Configuration of source of pdm requirements:
python -m pyproject_installer deps add check pdm test

Configuration of source of pipenv requirements:
python -m pyproject_installer deps add check pipenv Pipfile packages

---

name: sync
description: sync stored requirements to configured sources
example: python -m pyproject_installer deps sync --help

Positional arguments:

description: source names
default: all
example: python -m pyproject_installer deps sync build

Options:

name: --verify
description: Sync sources, but print diff and exits with code 4 if the sources were unsynced
default: only sync
example: python -m pyproject_installer deps sync --verify build

name: --verify-exclude
description: Regex patterns, exclude from diff requirements PEP503-normalized names of those match one of the patterns. Requires --verify.
default: []
example: python -m pyproject_installer deps sync --verify build --verify-exclude 'foo.*'

---

name: eval
description: evaluate stored requirements according to PEP508 in current Python environment and print them to stdout in PEP508 format (by default) or specified one
example: python -m pyproject_installer deps eval --help

Positional arguments:

description: source names
default: all
example: python -m pyproject_installer deps eval build

Options:

name: --depformat
description: format of dependency to print. Supported substitutions: $name - project's name, $nname - PEP503 normalized project's name, $fextra - project's extras (expanded first with --depformatextra)
default: PEP508 format
example: python -m pyproject_installer deps eval build --depformat='python3-$nn'

name: --depformatextra
description: format of extras to print (one extra of dependencies per line). Result is expanded in format specified by --depformat as $fextra. Supported substitutions: $extra
default: ''
example: python -m pyproject_installer deps eval build --depformat='python3-$nn$fextra' --depformatextra='+$extra'

name: --extra
description: PEP508 'extra' marker to evaluate with
default: None
example: python -m pyproject_installer deps eval build --extra tests

name: --exclude
description: regexes patterns, exclude requirement having PEP503-normalized name that matches one of these patterns
default: []
example: python -m pyproject_installer deps eval build --exclude types- pytest-cov

---

name: delete
description: deconfigure source of Python dependencies
example: python -m pyproject_installer deps delete --help

Positional arguments:

description: source name
example: python -m pyproject_installer deps delete build

Comparison with other tools

pyproject-installer consists of builder and installer, both provide only necessary and sufficient functionality.

builder

Functionally, today's builder is similar to build.
The key differences are:

• highly specialized defaults (see description)
• highly specialized features to drop extra runtime dependencies like pep517. No point to vendor pep517 to call only build_wheel backend hook in subprocess.

installer

Functionally, today's installer is similar to installer.
The key differences are:

• highly specialized defaults and features (see description)

Both can be replaced with pip. But again, no point to use full-featured complex pip to highly specialized task.

Bootstrap

There is a self-hosted build backend to avoid dependency on any other backend.

For example, bootstrap can be done as:

export PYTHONPATH=$(pwd)/src
python -m pyproject_installer build
python -m pyproject_installer install --destdir=/rootdir

Tests

• unit tests can be run as:

  pytest tests/unit
  

• integration tests (require internet connection and git tool) can be run as:

  pytest tests/integration
  

License

Distributed under the terms of the MIT license, pyproject-installer is free and open source software.