PEP 662 – Editable installs via virtual wheels | peps.python.org
PEP 662 – Editable installs via virtual wheels PEP 662 – Editable installs via virtual wheels Author : Bernát Gábor <gaborjbernat at gmail.com> Sponsor : Brett Cannon <brett at python.org> Discussions-To : Discourse thread Status : Rejected Type : Standards Track Topic : Packaging Created : 28-May-2021 Post-History : Resolution : Discourse thread Table of Contents Abstract Motivation Rationale Terminology and goals The Mechanism get_requires_for_build_editable prepare_metadata_for_build_editable build_editable Build frontend requirements Frontend requirements Example editable implementations Add the source tree as is to the interpreter Using custom importers Rejected ideas Copyright Abstract This document describes extensions to the build backend and frontend communication (as introduced by PEP 517 ) to allow projects to be installed in editable mode by introducing virtual wheels.
Motivation During development, many Python users prefer to install their libraries so that changes to the underlying source code and resources are automatically reflected in subsequent interpreter invocations without an additional installation step. This mode is usually called “development mode” or “editable installs”. Currently, there is no standardized way to accomplish this, as it was explicitly left out of PEP 517 due to the complexity of the actual observed behaviors.
At the moment, users to get this behaviour perform one of the following: For just Python code by adding the relevant source directories to sys.path (configurable from the command line interface via the PYTHONPATH environment variable). Note in this case, the users have to install the project dependencies themselves, and entry points or project metadata are not generated. setuptools provides the setup.py develop mechanism: that installs a pth file that injects the project root onto the sys.path at interpreter startup time, generates the project metadata, and also installs project dependencies. pip exposes calling this mechanism via the pip install -e command-line interface. flit provides the flit install –symlink command that symlinks the project files into the interpreters purelib folder, generates the project metadata, and also installs dependencies.
Note, this allows supporting resource files too. As these examples shows an editable install can be achieved in multiple ways and at the moment there’s no standard way of doing it. Furthermore, it’s not clear whose responsibility it is to achieve and define what an editable installation is: allow the build backend to define and materialize it, allow the build frontend to define and materialize it, explicitly define and standardize one method from the possible options.
The author of this PEP believes there’s no one size fits all solution here, each method of achieving editable effect has its pros and cons. Therefore this PEP rejects option three as it’s unlikely for the community to agree on a single solution. Furthermore, question remains as to whether the frontend or the build backend should own this responsibility. PEP 660 proposes the build backend to own this, while the current PEP proposes primarily the frontend, but still allows the backend to take take control if it wants to do so.
Rationale PEP 517 deferred “Editable installs” because this would have delayed further its adoption, and there wasn’t an agreement on how editable installs should be achieved. Due to the popularity of the setuptools and pip projects, the status quo prevailed, and the backend could achieve editable mode by providing a setup.py develop implementation, which the user could trigger via pip install -e . By defining an editable interface between the build backend and frontend, we can eliminate the setup.py file and their current communication method.
Terminology and goals This PEP aims to delineate the frontend and the backend roles clearly and give the developers of each the maximum ability to provide valuable features to their users. In this proposal, the backend’s role is to prepare the project for an editable installation, and then provide enough information to the frontend so that the frontend can manifest and enforce the editable installation. The information the backend provides to the frontend is a wheel that follows the existing specification within PEP 427 .
The wheel metadata about the archive itself ( {distribution}-{version}.dist-info/WHEEL ) must also contain the key Editable with value of true . However, instead of providing the project files within the wheel, it must provide an editable.json file (at the root level of the wheel) that defines the files to be exposed by the frontend. The content of this file is formulated as a mapping of absolute source tree paths to relative target interpreter destination paths within a scheme mapping.
A wheel that satisfies the previous two paragraphs is a virtual wheel. The frontend’s role is to take the virtual wheel and install the project in editable mode. The way it achieves this is entirely up to the frontend and is considered implementation detail. The editable installation mode implies that the source code of the project being installed is available in a local directory. Once the project is installed in editable mode, some changes to the project code in the local source tree will become effective without the need for a new installation step.
At a minimum, changes to the text of non-generated files that existed at the installation time should be reflected upon the subsequent import of the package. Some kinds of changes, such as adding or modifying entry points or new dependencies, require a new installation step to become effective. These changes are typically made in build backend configuration files (such as pyproject.toml ). This requirement is consistent with the general user expectation that such modifications will only become effective after re-installation.
While users expect editable installations to behave identically to standard installations, this may not always be possible and may be in tension with other user expectations. Depending on how a frontend implements the editable mode, some differences may be visible, such as the presence of additional files (compared to a typical installation), either in the source tree or the interpreter’s installation path. Frontends should seek to minimize differences between the behavior of editable and standard installations and document known differences.
For reference, a non-editable installation works as follows: The developer is using a tool, we’ll call it here the frontend , to drive the project development (e.g., pip ). When the user wants to trigger a package build and installation of a project, they’ll communicate with the frontend . The frontend uses a build frontend to trigger the build of a wheel (e.g., build ). The build frontend uses PEP 517 to communicate with the build backend (e.g. setuptools ) - with the build backend installed into a PEP 518 environment.
Once invoked, the backend returns a wheel. The frontend takes the wheel and feeds it to an installer (e.g., installer ) to install the wheel into the target Python interpreter. The Mechanism This PEP adds two optional hooks to the PEP 517 backend interface. One of the hooks is used to specify the build dependencies of an editable install. The other hook returns the necessary information via the build frontend the frontend needs to create an editable install. get_requires_for_build_editable def get_requires_for_build_editable ( config_settings = None ): ...
This hook MUST return an additional sequence of strings containing PEP 508 dependency specifications, above and beyond those specified in the pyproject.toml file. The frontend must ensure that these dependencies are available in the build environment in which the build_editable hook is called. If not defined, the default implementation is equivalent to returning [] . prepare_metadata_for_build_editable def prepare_me
The rest of this page is on the live site.