PEP 665 – A file format to list Python dependencies for reproducibility of an application | peps.python.org
PEP 665 – A file format to list Python dependencies for reproducibility of an application PEP 665 – A file format to list Python dependencies for reproducibility of an application Author : Brett Cannon <brett at python.org>, Pradyun Gedam <pradyunsg at gmail.com>, Tzu-ping Chung <uranusjr at gmail.com> PEP-Delegate : Paul Moore <p.f.moore at gmail.com> Discussions-To : Discourse thread Status : Rejected Type : Standards Track Topic : Packaging Created : 29-Jul-2021 Post-History : 29-Jul-2021, 03-Nov-2021, 25-Nov-2021 Superseded-By : 751 Resolution : Discourse message Table of Contents Abstract Terminology Motivation Rationale File Format Secure by Design Cross-Platform Simple Installer Specification Details version created-at [tool] [metadata] metadata.marker metadata.tag metadata.requires metadata.requires-python [[package._name_._version_]] package._name_._version_.filename [package._name_._version_.hashes] package._name_._version_.url package._name_._version_.direct package._name_._version_.requires-python package._name_._version_.requires Example Expectations for Lockers Expectations for Installers (Potential) Tool Support Backwards Compatibility Transition Plan Usability Interoperability Acceptance Security Implications How to Teach This Reference Implementation Rejected Ideas File Formats Other Than TOML Alternative Naming Schemes Supporting a Single Lock File Using a Flat List Instead of a Dependency Graph Use Wheel Tags in the File Name Alternative Names for requires Accepting PEP 650 Specifying Requirements per Package Instead of per File Specify Where Lockers Gather Input Allowing Source Distributions and Source Trees to be an Opt-In, Supported File Format Open Issues Acknowledgments Copyright Note This PEP was rejected due to lukewarm reception from the community from the lack of source distribution support.
Abstract This PEP specifies a file format to specify the list of Python package installation requirements for an application, and the relation between the specified requirements. The list of requirements is considered exhaustive for the installation target, and thus not requiring any information beyond the platform being installed for, and the file itself. The file format is flexible enough to allow installing the requirements across different platforms, which allows for reproducibility on multiple platforms from the same file.
Terminology There are several terms whose definition must be agreed upon in order to facilitate a discussion on the topic of this PEP. A package is something you install as a dependency and use via the import system. The packages on PyPI are an example of this. An application or app is an end product that other external code does not directly rely on via the import system (i.e. they are standalone). Desktop applications, command-line tools, etc. are examples of applications.
A lock file records the packages that are to be installed for an app. Traditionally, the exact version of the package to be installed is specified by a lock file, but specified packages are not always installed on a given platform (according a filtering logic described in a later section), which enables the lock file to describe reproducibility across multiple platforms. Examples of this are package-lock.json from npm , Poetry.lock from Poetry , etc.
Locking is the act of taking the input of the packages an app depends on and producing a lock file from that. A locker is a tool which produces a lock file. An installer consumes a lock file to install what the lock file specifies. Motivation Applications want reproducible installs for a few reasons (we are not worrying about package development, integration into larger systems that would handle locking dependencies external to the Python application, or other situations where flexible installation requirements are desired over strict, reproducible installations).
One, reproducibility eases development. When you and your fellow developers all end up with the same files on a specific platform, you make sure you are all developing towards the same experience for the application. You also want your users to install the same files as you expect to guarantee the experience is the same as you developed for them. Two, you want to be able to reproduce what gets installed across multiple platforms.
Thanks to Python’s portability across operating systems, CPUs, etc., it is very easy and often desirable to create applications that are not restricted to a single platform. Thus, you want to be flexible enough to allow for differences in your package dependencies between platforms, while still having consistency and reproducibility on any one specific platform. Three, reproducibility is more secure. When you control exactly what files are installed, you can make sure no malicious actor is attempting to slip nefarious code into your application (i.e. some supply chain attacks).
By using a lock file which always leads to reproducible installs, we can avoid certain risks entirely. Four, relying on the wheel file format provides reproducibility without requiring build tools to support reproducibility themselves. Thanks to wheels being static and not executing code as part of installation, wheels always lead to a reproducible result. Compare this to source distributions (aka sdists) or source trees which only lead to a reproducible install if their build tool supports reproducibility due to inherent code execution.
Unfortunately the vast majority of build tools do not support reproducible builds, so this PEP helps alleviate that issue by only supporting wheels as a package format. This PEP proposes a standard for a lock file, as the current solutions don’t meet the outlined goals. Today, the closest we come to a lock file standard is the requirements file format from pip. Unfortunately, that format does not lead to inherently reproducible installs (it requires optional features both in the requirements file and the installer itself, to be discussed later).
The community itself has also shown a need for lock files based on the fact that multiple tools have independently created their own lock file formats: PDM pip-tools Pipenv Poetry Pyflow Unfortunately, those tools all use differing lock file formats. This means tooling around these tools must be unique. This impacts tooling such as code editors and hosting providers, which want to be as flexible as possible when it comes to accepting a user’s application code, but also have a limit as to how much development resources they can spend to add support for yet another lock file format.
A standardized format would allow tools to focus their work on a single target, and make sure that workflow decisions made by developers outside of the lock file format are of no concern to e.g. hosting providers. Other programming language communities have also shown the usefulness of lock files by developing their own solution to this problem. Some of those communities include: Dart npm /Node Go Rust The trend in programming languages in the past decade seems to have been toward providing a lock file solution.
Rationale File Format We wanted the file format to be easy to read as a diff when auditing a change to the lock file. As such, and thanks to PEP 518 and pyproject.toml , we decided to go with the TOML file format. Secure by Design Viewing the requirements file format as the closest we have to a lock file standard, there are a few issues with the file format when it comes to security. First is that the file format simply does not require you to specify the exact version of a package.
This is why tools like pip-tools exist to help manage that users of requirements files. Second, you must opt into specifying what files are acceptable to be installed by using the --hash argument for a specific dependency. This is also optional with pip-tools as it requires specifying the --generate-hashes CLI argument. This requires --require-hashes for pip to make sure no dependencies
The rest of this page is on the live site.