PEP 793 – PyModExport: A new entry point for C extension modules | peps.python.org
PEP 793 – PyModExport: A new entry point for C extension modules PEP 793 – PyModExport: A new entry point for C extension modules Author : Petr Viktorin <encukou at gmail.com> Discussions-To : Discourse thread Status : Final Type : Standards Track Created : 23-May-2025 Python-Version : 3.15 Post-History : 14-Mar-2025 , 27-May-2025 Resolution : 23-Oct-2025 Table of Contents Abstract Background & Motivation The interpreter switch Rationale Using slots without a wrapper struct Tokens Soft-deprecating the existing export hook Specification The export hook Dynamic creation Tokens New slots Bits & Pieces Soft-deprecating the existing export hook New API summary Backwards Compatibility Forward compatibility Porting guide Backwards compatibility shim Security Implications How to Teach This Example Reference Implementation Rejected Ideas Exporting a data pointer rather than a function Changing PyModuleDef to not be PyObject Possible Future Directions Improving slots in general Updating defaults The inittab Footnotes Copyright Important This PEP is a historical document.
The up-to-date, canonical documentation can now be found at Defining extension modules . × See PEP 1 for how to propose changes. Abstract In this PEP, we propose a new entry point for C extension modules, by which one can define a module using an array of PyModuleDef_Slot structures without an enclosing PyModuleDef structure. This allows extension authors to avoid using a statically allocated PyObject , lifting the most common obstacle to making one compiled library file usable with both regular and free-threaded builds of CPython.
To make this viable, we also specify new module slot IDs to replace PyModuleDef ’s fields, and to allow adding a token similar to the Py_tp_token used for type objects. We also add an API for defining modules from slots dynamically. The existing API ( PyInit_* ) is soft-deprecated. (That is: it will continue to work without warnings, and it’ll be fully documented and supported, but we plan to not add any new features to it.) Background & Motivation The memory layout of Python objects differs between regular and free-threading builds.
So, an ABI that supports both regular and free-threading builds cannot include the current PyObject memory layout. To stay compatible with existing ABI (and API), it cannot support statically allocated Python objects. There is one type of object that is needed in most extension modules and is allocated statically in virtually all cases: the PyModuleDef returned from the module export hooks (that is, PyInit_* functions).
Module export hooks ( PyInit_* functions) can return two kinds of objects: A fully initialized module object (for so-called single-phase initialization ). This was the only option in 3.4 and below. Modules created this way have surprising (but backwards-compatible) behaviour around multiple interpreters or repeated loading. (Specifically, the contents of such a module’s __dict__ are shared across all instances of the module object.) The module returned is typically created using the PyModule_Create function, which requires a statically allocated (or at least long-lived) PyModuleDef struct.
It is possible to bypass this using the lower-level PyModule_New* API. This avoids the need for PyModuleDef , but offers much less functionality. A PyModuleDef object containing a description of how to create a module object. This option, multi-phase initialization , was introduced in PEP 489 ; see its motivation for why it exists. The interpreter cannot distinguish between these cases before the export hook is called.
The interpreter switch Python 3.12 added a way for modules to mark whether they may be loaded in a subinterpreter: the Py_mod_multiple_interpreters slot. Setting it to the “not supported” value signals that an extension can only be loaded in the main interpreter. Unfortunately, Python can only get this information by calling the module export hook. For single-phase modules, that creates the module object and runs arbitrary initialization code.
For modules that set Py_mod_multiple_interpreters to “not supported”, this initialization needs to happen in the main interpreter. To make this work, if a new module is loaded in a sub-interpreter, Python temporarily switches to the main interpreter, calls the export hook there, and then either switches back and redoes the import, or fails. This unnecessary and fragile extra work highlights the underlying design issue: Python has no way to get information about an extension before the extension can, potentially, fully initialize itself.
Rationale For avoiding the module export hook requiring a statically allocated PyObject* , two options come to mind: Returning a dynamically allocated object, whose ownership is transferred to the interpreter. This stucture could be very similar to the existing PyModuleDef , since it needs to contain the same data. Unlike the existing PyModuleDef , this one would need to be reference-counted so that it both outlives “its” module and does not leak.
Adding a new export hook, which does not return a PyObject* . This was considered already for Python 3.5 in PEP 489 , but rejected: Keeping only the PyInit hook name, even if it’s not entirely appropriate for exporting a definition, yielded a much simpler solution. Alas, after a decade of fixing the implications of this choice, the solution is no longer simple. A new hook will also allow Python to avoid the second issue mentioned in Motivation – the interpreter switch.
Effectivelly, it will add a new phase to multi-phase initialization, in which Python can check whether the module is compatible. Using slots without a wrapper struct The existing PyModuleDef is a struct with some fixed fields and a “slots” array. Unlike slots, the fixed fields cannot be individually deprecated and replaced. This proposal does away with fixed fields and proposes using a slots array directly, without a wrapper struct.
The PyModuleDef_Slot struct does have some downsides compared to fixed fields. We believe these are fixable, but leave that out of scope of this PEP. (Note: this was done in PEP 820 , still in Python 3.15.) Tokens A static PyModuleDef has another purpose besides describing how a module should be created. As a statically allocated singleton that remains attached to the module object, it allows extension authors to check whether a given Python module is “theirs”: if a module object has a known PyModuleDef , its module state will have a known memory layout.
An analogous issue was solved for types by adding Py_tp_token . This proposal adds the same mechanism to modules. Unlike types, the import mechanism often has a pointer that’s known to be suitable as a token value; in these cases it can provide a default token. Thus, module tokens do not need a variant of the inelegant Py_TP_USE_SPEC . To help extensions that straddle Python versions, PyModuleDef addresses are used as default tokens, and where it’s reasonable, they are made interchangeable with tokens.
Soft-deprecating the existing export hook The only reason for authors of existing extensions to switch to the API proposed here is that it allows a single module for both free-threaded and non-free-threaded builds. It is important that Python allows that, but for many existing modules, it is nowhere near worth losing compatibility with 3.14 and lower versions. It is much too early to plan deprecation of the old API. Instead, this PEP proposes to stop adding new features to the PyInit_* scheme.
After all, the perfect time for extension authors to switch is when they want to modify module initialization anyway. Specification The export hook When importing an extension module, Python will now first look for an export hook like this: PyModuleDef_Slot * PyModExport_ < NAME > ( void ); Note PEP 820 changed the return type to PySlot * . where <NAME> is the name of the module. For non-ASCII names, it will instead look for PyModExpor
The rest of this page is on the live site.