PEP 489 – Multi-phase extension module initialization | peps.python.org
PEP 489 – Multi-phase extension module initialization PEP 489 – Multi-phase extension module initialization Author : Petr Viktorin <encukou at gmail.com>, Stefan Behnel <stefan_ml at behnel.de>, Alyssa Coghlan <ncoghlan at gmail.com> BDFL-Delegate : Eric Snow <ericsnowcurrently at gmail.com> Discussions-To : Import-SIG list Status : Final Type : Standards Track Created : 11-Aug-2013 Python-Version : 3.5 Post-History : 23-Aug-2013, 20-Feb-2015, 16-Apr-2015, 07-May-2015, 18-May-2015 Resolution : Python-Dev message Table of Contents Abstract Motivation The current process The proposal Pseudo-code Overview Module Creation Phase The Py_mod_create slot Post-creation steps Module Execution Phase Pre-Execution steps The Py_mod_exec slot Legacy Init Built-In modules Subinterpreters and Interpreter Reloading Functions incompatible with multi-phase initialization Module state and C-level callbacks New Functions Export Hook Name Module Reloading Multiple modules in one library Testing and initial implementations Summary of API Changes and Additions Backwards Compatibility Possible Future Extensions Implementation Previous Approaches References Copyright Important This PEP is a historical document.
The up-to-date, canonical documentation can now be found at Initializing C modules . For Python 3.14+, see Defining extension modules and Module definition struct . × See PEP 1 for how to propose changes. Abstract This PEP proposes a redesign of the way in which built-in and extension modules interact with the import machinery. This was last revised for Python 3.0 in PEP 3121, but did not solve all problems at the time.
The goal is to solve import-related problems by bringing extension modules closer to the way Python modules behave; specifically to hook into the ModuleSpec-based loading mechanism introduced in PEP 451 . This proposal draws inspiration from PyType_Spec of PEP 384 to allow extension authors to only define features they need, and to allow future additions to extension module declarations. Extensions modules are created in a two-step process, fitting better into the ModuleSpec architecture, with parallels to __new__ and __init__ of classes.
Extension modules can safely store arbitrary C-level per-module state in the module that is covered by normal garbage collection and supports reloading and sub-interpreters. Extension authors are encouraged to take these issues into account when using the new API. The proposal also allows extension modules with non-ASCII names. Not all problems tackled in PEP 3121 are solved in this proposal. In particular, problems with run-time module lookup ( PyState_FindModule ) are left to a future PEP.
Motivation Python modules and extension modules are not being set up in the same way. For Python modules, the module object is created and set up first, then the module code is being executed ( PEP 302 ). A ModuleSpec object ( PEP 451 ) is used to hold information about the module, and passed to the relevant hooks. For extensions (i.e. shared libraries) and built-in modules, the module init function is executed straight away and does both the creation and initialization.
The initialization function is not passed the ModuleSpec, or any information it contains, such as the __file__ or fully-qualified name. This hinders relative imports and resource loading. In Py3, modules are also not being added to sys.modules, which means that a (potentially transitive) re-import of the module will really try to re-import it and thus run into an infinite loop when it executes the module init function again.
Without access to the fully-qualified module name, it is not trivial to correctly add the module to sys.modules either. This is specifically a problem for Cython generated modules, for which it’s not uncommon that the module init code has the same level of complexity as that of any ‘regular’ Python module. Also, the lack of __file__ and __name__ information hinders the compilation of “ __init__.py ” modules, i.e. packages, especially when relative imports are being used at module init time.
Furthermore, the majority of currently existing extension modules has problems with sub-interpreter support and/or interpreter reloading, and, while it is possible with the current infrastructure to support these features, it is neither easy nor efficient. Addressing these issues was the goal of PEP 3121 , but many extensions, including some in the standard library, took the least-effort approach to porting to Python 3, leaving these issues unresolved.
This PEP keeps backwards compatibility, which should reduce pressure and give extension authors adequate time to consider these issues when porting. The current process Currently, extension and built-in modules export an initialization function named “ PyInit_modulename ”, named after the file name of the shared library. This function is executed by the import machinery and must return a fully initialized module object.
The function receives no arguments, so it has no way of knowing about its import context. During its execution, the module init function creates a module object based on a PyModuleDef object. It then continues to initialize it by adding attributes to the module dict, creating types, etc. In the back, the shared library loader keeps a note of the fully qualified module name of the last module that it loaded, and when a module gets created that has a matching name, this global variable is used to determine the fully qualified name of the module object.
This is not entirely safe as it relies on the module init function creating its own module object first, but this assumption usually holds in practice. The proposal The initialization function ( PyInit_modulename ) will be allowed to return a pointer to a PyModuleDef object. The import machinery will be in charge of constructing the module object, calling hooks provided in the PyModuleDef in the relevant phases of initialization (as described below).
This multi-phase initialization is an additional possibility. Single-phase initialization, the current practice of returning a fully initialized module object, will still be accepted, so existing code will work unchanged, including binary compatibility. The PyModuleDef structure will be changed to contain a list of slots, similarly to PEP 384 ’s PyType_Spec for types. To keep binary compatibility, and avoid needing to introduce a new structure (which would introduce additional supporting functions and per-module storage), the currently unused m_reload pointer of PyModuleDef will be changed to hold the slots.
The structures are defined as: typedef struct { int slot ; void * value ; } PyModuleDef_Slot ; typedef struct PyModuleDef { PyModuleDef_Base m_base ; const char * m_name ; const char * m_doc ; Py_ssize_t m_size ; PyMethodDef * m_methods ; PyModuleDef_Slot * m_slots ; /* changed from `inquiry m_reload;` */ traverseproc m_traverse ; inquiry m_clear ; freefunc m_free ; } PyModuleDef ; The m_slots member must be either NULL, or point to an array of PyModuleDef_Slot structures, terminated by a slot with id set to 0 (i.e. {0, NULL} ).
To specify a slot, a unique slot ID must be provided. New Python versions may introduce new slot IDs, but slot IDs will never be recycled. Slots may get deprecated, but will continue to be supported throughout Python 3.x. A slot’s value pointer may not be NULL, unless specified otherwise in the slot’s documentation. The following slots are currently available, and described later: Py_mod_create Py_mod_exec Unknown slot IDs will cause the import to fail with SystemError.
When using multi-phase initialization, the m_name field of PyModuleDef will not be used during importing; the module name will be taken from the ModuleSpec. Before it is returned from PyInit_*, the PyModuleDef object must be initialized using the newly added PyModuleDef_Init function. This sets the object type (which cannot be done statically on certain compilers), refcount, and in
The rest of this page is on the live site.