PEP 582 – Python local packages directory | peps.python.org
PEP 582 – Python local packages directory PEP 582 – Python local packages directory Author : Kushal Das <mail at kushaldas.in>, Steve Dower <steve.dower at python.org>, Donald Stufft <donald at stufft.io>, Alyssa Coghlan <ncoghlan at gmail.com> Discussions-To : Discourse thread Status : Rejected Type : Standards Track Topic : Packaging Created : 16-May-2018 Python-Version : 3.12 Post-History : 01-Mar-2019 Resolution : Discourse message Table of Contents Abstract Motivation Rationale Specification Example Relationship to virtual environments Security Considerations How to Teach This Impact on Tools Backwards Compatibility Impact on other Python implementations Reference Implementation Rejected Ideas Copyright Abstract This PEP proposes extending the existing mechanism for setting up sys.path to include a new __pypackages__ directory, in addition to the existing locations.
The new directory will be added at the start of sys.path , after the current working directory and just before the system site-packages, to give packages installed there priority over other locations. This is similar to the existing mechanism of adding the current directory (or the directory the script is located in), but by using a subdirectory, additional libraries are kept separate from the user’s work. Motivation New Python programmers can benefit from being taught the value of isolating an individual project’s dependencies from their system environment.
However, the existing mechanism for doing this, virtual environments, is known to be complex and error-prone for beginners to understand. Explaining virtual environments is often a distraction when trying to get a group of beginners set up - differences in platform and shell environments require individual assistance, and the need for activation in every new shell session makes it easy for students to make mistakes when coming back to work after a break.
This proposal offers a lightweight solution that gives isolation without the user needing to understand more advanced concepts. Furthermore, standalone Python applications usually need 3rd party libraries to function. Typically, they are either designed to be run from a virtual environment, where the dependencies are installed into the environment alongside the application, or they bundle their dependencies in a subdirectory, and modify sys.path at application startup.
Virtual environments, while a common and effective solution (used, for example, by the pipx tool), are somewhat awkward to set up and manage, and are not relocatable. On the other hand, manual manipulation of sys.path is boilerplate that developers need to get right, and (being a runtime behaviour) it is not understood by tools like linters and type checkers. The __pypackages__ proposal formalises the idea of a “bundled dependencies” location, avoiding the boilerplate and providing a standard location that development tools can be taught to recognise.
It should be noted that in general, Python libraries cannot be simply copied between machines, platforms, or even necessarily between Python versions. This proposal does nothing to change that fact, and while it is tempting to assume that bundling a script and its __pypackages__ is a mechanism for distributing applications, this is explicitly not a goal of this proposal. Developers remain responsible for the portability of their code.
Rationale While sys.path can be manipulated at runtime, the default value is important, as it establishes a common baseline that users and tools can agree on. The current default does not include a location that could be viewed as “private to the current project”, and yet that is a useful concept. This is similar to the npm node_modules directory, which is popular in the Javascript community, and something that developers familiar with that ecosystem often ask for from Python.
Specification This PEP proposes to add a new step in the process of calculating sys.path at startup. When the interactive interpreter starts, if a __pypackages__ directory is found in the current working directory, then it will be included in sys.path after the entry for current working directory and just before the system site-packages. When the interpreter runs a script, Python will try to find __pypackages__ in the same directory as the script.
If found (along with the current Python version directory inside), then it will be used, otherwise Python will behave as it does currently. The behaviour should work exactly the same as the way the existing mechanism for adding the current working directory or script directory to sys.path works. For example, __pypackages__ will be ignored if the -P option or the PYTHONSAFEPATH environment variable is set. In order to be recognised, the __pypackages__ directory must be laid out according to a new localpackages scheme in the sysconfig module.
Specifically, both of the purelib and platlib directories must be present, using the following code to determine the locations of those directories: scheme = "localpackages" purelib = sysconfig . get_path ( "purelib" , scheme , vars = { "base" : "__pypackages__" , "platbase" : "__pypackages__" }) platlib = sysconfig . get_path ( "platlib" , scheme , vars = { "base" : "__pypackages__" , "platbase" : "__pypackages__" }) These two locations will be added to sys.path , other directories or files in the __pypackages__ directory will be silently ignored.
The paths will be based on Python versions. Note There is a possible option of having a separate new API, it is documented at issue #3013 . Example The following shows an example project directory structure, and different ways the Python executable and any script will behave. The example is for Unix-like systems - on Windows the subdirectories will be different. foo __pypackages__ lib python3 .10 site - packages bottle myscript . py /> python foo / myscript . py sys . path [ 0 ] == 'foo' sys . path [ 1 ] == 'foo/__pypackages__/lib/python3.10/site-packages/' cd foo foo > / usr / bin / ansible #! /usr/bin/env python3 foo > python / usr / bin / ansible foo > python myscript . py foo > python sys . path [ 0 ] == '.' sys . path [ 1 ] == './__pypackages__/lib/python3.10/site-packages' foo > python - m bottle We have a project directory called foo and it has a __pypackages__ inside of it.
We have bottle installed in that __pypackages__/lib/python3.10/site-packages/ , and have a myscript.py file inside of the project directory. We have used whatever tool we generally use to install bottle in that location. For invoking a script, Python will try to find a __pypackages__ inside of the directory that the script resides [1] , /usr/bin . The same will happen in case of the last example, where we are executing /usr/bin/ansible from inside of the foo directory.
In both cases, it will not use the __pypackages__ in the current working directory. Similarly, if we invoke myscript.py from the first example, it will use the __pypackages__ directory that was in the foo directory. If we go inside of the foo directory and start the Python executable (the interpreter), it will find the __pypackages__ directory inside of the current working directory and use it in the sys.path . The same happens if we try to use the -m and use a module.
In our example, bottle module will be found inside of the __pypackages__ directory. The above two examples are only cases where __pypackages__ from current working directory is used. In another example scenario, a trainer of a Python class can say “Today we are going to learn how to use Twisted! To start, please checkout our example project, go to that directory, and then run a given command to install Twisted.” That will install Twisted into a directory separate from python3 .
There’s no need to discuss virtual environments, global versus user installs, etc. as the install will be local by default. The trainer can then just keep telling them to use python3 without any activation step, etc. Relationship to virt
The rest of this page is on the live site.