PEP 3147 – PYC Repository Directories | peps.python.org
PEP 3147 – PYC Repository Directories PEP 3147 – PYC Repository Directories Author : Barry Warsaw <barry at python.org> Status : Final Type : Standards Track Created : 16-Dec-2009 Python-Version : 3.2 Post-History : 30-Jan-2010, 25-Feb-2010, 03-Mar-2010, 12-Apr-2010 Resolution : Python-Dev message Table of Contents Abstract Background Rationale Proposal Examples Python behavior Case 0: The steady state Case 1: The first import Case 2: The second import Case 3: __pycache__/foo.<magic>.pyc with no source Case 4: legacy pyc files and source-less imports Case 5: read-only file systems Flow chart Alternative Python implementations Implementation strategy Effects on existing code Detecting PEP 3147 availability __file__ py_compile and compileall bdist_wininst and the Windows installer File extension checks Backports Makefiles and other dependency tools Alternatives Hexadecimal magic tags PEP 304 Fat byte compilation files Multiple file extensions .pyc Reference implementation References ACKNOWLEDGMENTS Copyright Abstract This PEP describes an extension to Python’s import mechanism which improves sharing of Python source code files among multiple installed different versions of the Python interpreter.
It does this by allowing more than one byte compilation file (.pyc files) to be co-located with the Python source file (.py file). The extension described here can also be used to support different Python compilation caches, such as JIT output that may be produced by an Unladen Swallow ( PEP 3146 ) enabled C Python. Background CPython compiles its source code into “byte code”, and for performance reasons, it caches this byte code on the file system whenever the source file has changes.
This makes loading of Python modules much faster because the compilation phase can be bypassed. When your source file is foo.py , CPython caches the byte code in a foo.pyc file right next to the source. Byte code files contain two 32-bit big-endian numbers followed by the marshaled [2] code object. The 32-bit numbers represent a magic number and a timestamp. The magic number changes whenever Python changes the byte code format, e.g. by adding new byte codes to its virtual machine.
This ensures that pyc files built for previous versions of the VM won’t cause problems. The timestamp is used to make sure that the pyc file match the py file that was used to create it. When either the magic number or timestamp do not match, the py file is recompiled and a new pyc file is written. In practice, it is well known that pyc files are not compatible across Python major releases. A reading of import.c [3] in the Python source code proves that within recent memory, every new CPython major release has bumped the pyc magic number.
Rationale Linux distributions such as Ubuntu [4] and Debian [5] provide more than one Python version at the same time to their users. For example, Ubuntu 9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1, with Python 2.6 being the default. This causes a conflict for third party Python source files installed by the system, because you cannot compile a single Python source file for more than one Python version at a time.
When Python finds a pyc file with a non-matching magic number, it falls back to the slower process of recompiling the source. Thus if your system installed a /usr/share/python/foo.py , two different versions of Python would fight over the pyc file and rewrite it each time the source is compiled. (The standard library is unaffected by this, since multiple versions of the stdlib are installed on such distributions..) Furthermore, in order to ease the burden on operating system packagers for these distributions, the distribution packages do not contain Python version numbers [6] ; they are shared across all Python versions installed on the system.
Putting Python version numbers in the packages would be a maintenance nightmare, since all the packages - and their dependencies - would have to be updated every time a new Python release was added or removed from the distribution. Because of the sheer number of packages available, this amount of work is infeasible. ( PEP 384 has been proposed to address binary compatibility issues of third party extension modules across different versions of Python.) Because these distributions cannot share pyc files, elaborate mechanisms have been developed to put the resulting pyc files in non-shared locations while the source code is still shared.
Examples include the symlink-based Debian regimes python-support [8] and python-central [9] . These approaches make for much more complicated, fragile, inscrutable, and fragmented policies for delivering Python applications to a wide range of users. Arguably more users get Python from their operating system vendor than from upstream tarballs. Thus, solving this pyc sharing problem for CPython is a high priority for such vendors.
This PEP proposes a solution to this problem. Proposal Python’s import machinery is extended to write and search for byte code cache files in a single directory inside every Python package directory. This directory will be called __pycache__ . Further, pyc file names will contain a magic string (called a “tag”) that differentiates the Python version they were compiled for. This allows multiple byte compiled cache files to co-exist for a single Python source file.
The magic tag is implementation defined, but should contain the implementation name and a version number shorthand, e.g. cpython-32 . It must be unique among all versions of Python, and whenever the magic number is bumped, a new magic tag must be defined. An example pyc file for Python 3.2 is thus foo.cpython-32.pyc . The magic tag is available in the imp module via the get_tag() function. This is parallel to the imp.get_magic() function.
This scheme has the added benefit of reducing the clutter in a Python package directory. When a Python source file is imported for the first time, a __pycache__ directory will be created in the package directory, if one does not already exist. The pyc file for the imported source will be written to the __pycache__ directory, using the magic-tag formatted name. If either the creation of the __pycache__ directory or the pyc file inside that fails, the import will still succeed, just as it does in a pre- PEP 3147 world.
If the py source file is missing, the pyc file inside __pycache__ will be ignored. This eliminates the problem of accidental stale pyc file imports. For backward compatibility, Python will still support pyc-only distributions, however it will only do so when the pyc file lives in the directory where the py file would have been, i.e. not in the __pycache__ directory. pyc file outside of __pycache__ will only be imported if the py source file is missing.
Tools such as py_compile [15] and compileall [16] will be extended to create PEP 3147 formatted layouts automatically, but will have an option to create pyc-only distribution layouts. Examples What would this look like in practice? Let’s say we have a Python package named alpha which contains a sub-package name beta . The source directory layout before byte compilation might look like this: alpha / __init__ . py one . py two . py beta / __init__ . py three . py four . py After byte compiling this package with Python 3.2, you would see the following layout: alpha / __pycache__ / __init__ . cpython - 32. pyc one . cpython - 32. pyc two . cpython - 32. pyc __init__ . py one . py two . py beta / __pycache__ / __init__ . cpython - 32. pyc three . cpython - 32. pyc four . cpython - 32. pyc __init__ . py three . py four . py Note: listing order may differ depending on the platform.
Let’s say that two new versions of Python are installed, one is Python 3.3 and another is Unladen Swallow. After byte compilation, the file system would look like this: alpha / __pycache__ / __init__ . cpython - 32. pyc __init__ . cpython - 33. pyc __init__ . unladen - 10. pyc one . cpython - 32. py
The rest of this page is on the live site.