PEP 517 – A build-system independent format for source trees | peps.python.org
PEP 517 – A build-system independent format for source trees PEP 517 – A build-system independent format for source trees Author : Nathaniel J. Smith <njs at pobox.com>, Thomas Kluyver <thomas at kluyver.me.uk> BDFL-Delegate : Alyssa Coghlan <ncoghlan at gmail.com> Discussions-To : Distutils-SIG list Status : Final Type : Standards Track Topic : Packaging Created : 30-Sep-2015 Post-History : 01-Oct-2015, 25-Oct-2015, 19-May-2017, 11-Sep-2017 Resolution : Distutils-SIG message Table of Contents Abstract Terminology and goals Source trees Build requirements Build backend interface Mandatory hooks build_wheel build_sdist Optional hooks get_requires_for_build_wheel prepare_metadata_for_build_wheel get_requires_for_build_sdist Config settings Build environment Recommendations for build frontends (non-normative) In-tree build backends Source distributions Evolutionary notes Rejected options Summary of changes to PEP 517 Appendix A: Comparison to PEP 516 Other differences Copyright Abstract While distutils / setuptools have taken us a long way, they suffer from three serious problems: (a) they’re missing important features like usable build-time dependency declaration, autoconfiguration, and even basic ergonomic niceties like DRY -compliant version number management, and (b) extending them is difficult, so while there do exist various solutions to the above problems, they’re often quirky, fragile, and expensive to maintain, and yet (c) it’s very difficult to use anything else, because distutils/setuptools provide the standard interface for installing packages expected by both users and installation tools like pip .
Previous efforts (e.g. distutils2 or setuptools itself) have attempted to solve problems (a) and/or (b). This proposal aims to solve (c). The goal of this PEP is get distutils-sig out of the business of being a gatekeeper for Python build systems. If you want to use distutils, great; if you want to use something else, then that should be easy to do using standardized methods. The difficulty of interfacing with distutils means that there aren’t many such systems right now, but to give a sense of what we’re thinking about see flit or bento .
Fortunately, wheels have now solved many of the hard problems here – e.g. it’s no longer necessary that a build system also know about every possible installation configuration – so pretty much all we really need from a build system is that it have some way to spit out standard-compliant wheels and sdists. We therefore propose a new, relatively minimal interface for installation tools like pip to interact with package source trees and source distributions.
Terminology and goals A source tree is something like a VCS checkout. We need a standard interface for installing from this format, to support usages like pip install some-directory/ . A source distribution is a static snapshot representing a particular release of some source code, like lxml-3.4.4.tar.gz . Source distributions serve many purposes: they form an archival record of releases, they provide a stupid-simple de facto standard for tools that want to ingest and process large corpora of code, possibly written in many languages (e.g. code search), they act as the input to downstream packaging systems like Debian/Fedora/Conda/…, and so forth.
In the Python ecosystem they additionally have a particularly important role to play, because packaging tools like pip are able to use source distributions to fulfill binary dependencies, e.g. if there is a distribution foo.whl which declares a dependency on bar , then we need to support the case where pip install bar or pip install foo automatically locates the sdist for bar , downloads it, builds it, and installs the resulting package.
Source distributions are also known as sdists for short. A build frontend is a tool that users might run that takes arbitrary source trees or source distributions and builds wheels from them. The actual building is done by each source tree’s build backend . In a command like pip wheel some-directory/ , pip is acting as a build frontend. An integration frontend is a tool that users might run that takes a set of package requirements (e.g. a requirements.txt file) and attempts to update a working environment to satisfy those requirements.
This may require locating, building, and installing a combination of wheels and sdists. In a command like pip install lxml==2.4.0 , pip is acting as an integration frontend. Source trees There is an existing, legacy source tree format involving setup.py . We don’t try to specify it further; its de facto specification is encoded in the source code and documentation of distutils , setuptools , pip , and other tools. We’ll refer to it as the setup.py -style.
Here we define a new style of source tree based around the pyproject.toml file defined in PEP 518 , extending the [build-system] table in that file with one additional key, build-backend . Here’s an example of how it would look: [ build - system ] # Defined by PEP 518: requires = [ "flit" ] # Defined by this PEP: build - backend = "flit.api:main" build-backend is a string naming a Python object that will be used to perform the build (see below for details).
This is formatted following the same module:object syntax as a setuptools entry point. For instance, if the string is "flit.api:main" as in the example above, this object would be looked up by executing the equivalent of: import flit.api backend = flit . api . main It’s also legal to leave out the :object part, e.g. build - backend = "flit.api" which acts like: import flit.api backend = flit . api Formally, the string should satisfy this grammar: identifier = (letter | '_') (letter | '_' | digit)* module_path = identifier ('.' identifier)* object_path = identifier ('.' identifier)* entry_point = module_path (':' object_path)?
And we import module_path and then lookup module_path.object_path (or just module_path if object_path is missing). When importing the module path, we do not look in the directory containing the source tree, unless that would be on sys.path anyway (e.g. because it is specified in PYTHONPATH). Although Python automatically adds the working directory to sys.path in some situations, code to resolve the backend should not be affected by this.
If the pyproject.toml file is absent, or the build-backend key is missing, the source tree is not using this specification, and tools should revert to the legacy behaviour of running setup.py (either directly, or by implicitly invoking the setuptools.build_meta:__legacy__ backend). Where the build-backend key exists, this takes precedence and the source tree follows the format and conventions of the specified backend (as such no setup.py is needed unless the backend requires it).
Projects may still wish to include a setup.py for compatibility with tools that do not use this spec. This PEP also defines a backend-path key for use in pyproject.toml , see the “In-Tree Build Backends” section below. This key would be used as follows: [ build - system ] # Defined by PEP 518: requires = [ "flit" ] # Defined by this PEP: build - backend = "local_backend" backend - path = [ "backend" ] Build requirements This PEP places a number of additional requirements on the “build requirements” section of pyproject.toml .
These are intended to ensure that projects do not create impossible to satisfy conditions with their build requirements. Project build requirements will define a directed graph of requirements (project A needs B to build, B needs C and D, etc.) This graph MUST NOT contain cycles. If (due to lack of co-ordination between projects, for example) a cycle is present, front ends MAY refuse to build the project. Where build requirements are available as wheels, front ends SHOULD use these where practical, to avoid deeply nested builds.
However front ends MAY have modes where they do not consider wheels when locating build requirements, and so projects MUST NOT a
The rest of this page is on the live site.