Home

pip UX Team work

Testing resolution impossible error messages

A Breakdown of the Packaging Problems

Or in other words, a (likely in-exhaustive) top down view of big problems to solve in Python Packaging.

Disclaimer: This is currently an open/unofficial effort towards grokking the space of packaging problems.

Metadata

Versions:

• 1.0 - key/value
• 1.1 - key/value
• 1.2 - key/value - initial support distribution depedency
• 2.0 - json

Use:

• Setuptools puts METADATA 1.0 in *.egg-info/METADATA's
• Wheel 1.0 puts METADATA "2.0" in *.dist-info/METADATA
• pydist.json supposedly holds METADATA 2.0

Problems:

• Change wheel to indicate it's saving METADATA 1.2?
• Update setuptools to output METADATA 1.2 information, in pure .dist-info form
• Work towards allowing more declarative specification of metadata in packages, than that currently encourages by setuptools
• Figure out best practices for platform and ABI compatibility, requires a lot of communication, but currently going ok

Dependency Resolution

Problems:

• Pip has no ordering of indexes from which to pull packages
• No resolution of conflicting dependencies
• No central database of installations (currently separate .{egg,dist}-info folders)
• Packages must be fully downloaded to discover their dependencies

Solutions:

• Introduce a proper dependency resolution solver to pip
• Ingestion of package metadata for dependency information could be separated from download package contents, to make installs faster and more customizable.
• Allow for more descriptive version and dependency information in metadata (python versions, implementations, platforms, abis, etc).
• A dependency solver needs a reliable database of of what is already installed and all dependency metadata relating to that. Currently metadata 1.2 is minimum viable for this, but 2.0 would be preferred for platform/abi/etc dependencies

Prior art

Packages, Installation and Formats

Wheel:

For future Wheel-Versions

• Add a controlled, sanctioned way of making zip-importable wheels?
• " " " " " " " executable wheels? (python mywheel.whl or python -m mywheel.whl?)
• Fix and version metadata inclusion
• wheel filenames are currently used to tell a lot of compatibility information - is this appropriate? When you start to practically factor in platforms and ABIs, people can get huge filenames. Where / when should the line be drawn? Could/Should this be moved away from? i.e., when all this information comes from metadata associated with the content and available externally to the package, it no longer needs to be in the filename.

Platforming and ABIs

• Discussion on OS X ABI compatibility between its pythons

Setuptools

• Update output information to metadata 1.2, to work towards viable dependency information in local package database

Logistics and Documentation

In progress

• virtualenv docs update
• Completing PPUG - some of our best work here!

Organisation of Mailing Lists & Communication Channels

• Freenode irc: #pypa, #pip, #pypa-dev, #distutils
• Mailing-lists: distutils-sig@python.org, catalog-SIG@python.org?,pypa-dev, python-virtualenv, others?

These could be restructured into two central channels per communication medium - one for users to discuss problems and questions (e.g "pypa-users" or "pypa"), and one for developers to discuss work on projects (e.g "pypa-dev"). If so, such reorganisation should be announced to people!

Furthering Communication

This needs to be improved, a lot. See

• setuptools Features
• Lack of Communication
• Lack of good deprecation policies

I'm sure there are quite a few other examples I've missed that all could be addressed systematically.

PIP project (start: January 2020)

Meeting notes

• Meeting notes