PEP 573 – Module State Access from C Extension Methods | peps.python.org
PEP 573 – Module State Access from C Extension Methods PEP 573 – Module State Access from C Extension Methods Author : Petr Viktorin <encukou at gmail.com>, Alyssa Coghlan <ncoghlan at gmail.com>, Eric Snow <ericsnowcurrently at gmail.com>, Marcel Plch <gmarcel.plch at gmail.com> BDFL-Delegate : Stefan Behnel Discussions-To : Import-SIG list Status : Final Type : Standards Track Created : 02-Jun-2016 Python-Version : 3.9 Post-History : Table of Contents Abstract Terminology Process-Global State Per-module State Static Type Heap Type Defining Class C-API Rationale Background Proposal Slot methods Specification Adding module references to heap types Passing the defining class to extension methods Argument Clinic Helpers Modules Converted in the Initial Implementation Summary of API Changes and Additions Backwards Compatibility Implementation Possible Future Extensions Slot methods Easy creation of types with module references Optimization References Copyright Abstract This PEP proposes to add a way for CPython extension methods to access context, such as the state of the modules they are defined in.
This will allow extension methods to use direct pointer dereferences rather than PyState_FindModule for looking up module state, reducing or eliminating the performance cost of using module-scoped state over process global state. This fixes one of the remaining roadblocks for adoption of PEP 3121 (Extension module initialization and finalization) and PEP 489 (Multi-phase extension module initialization). While this PEP takes an additional step towards fully solving the problems that PEP 3121 and PEP 489 started tackling, it does not attempt to resolve all remaining concerns.
In particular, access to the module state from slot methods ( nb_add , etc) is not solved. Terminology Process-Global State C-level static variables. Since this is very low-level memory storage, it must be managed carefully. Per-module State State local to a module object, allocated dynamically as part of a module object’s initialization. This isolates the state from other instances of the module (including those in other subinterpreters).
Accessed by PyModule_GetState() . Static Type A type object defined as a C-level static variable, i.e. a compiled-in type object. A static type needs to be shared between module instances and has no information of what module it belongs to. Static types do not have __dict__ (although their instances might). Heap Type A type object created at run time. Defining Class The defining class of a method (either bound or unbound) is the class on which the method was defined.
A class that merely inherits the method from its base is not the defining class. For example, int is the defining class of True.to_bytes , True.__floor__ and int.__repr__ . In C, the defining class is the one defined with the corresponding tp_methods or “tp slots” [1] entry. For methods defined in Python, the defining class is saved in the __class__ closure cell. C-API The “Python/C API” as described in Python documentation.
CPython implements the C-API, but other implementations exist. Rationale PEP 489 introduced a new way to initialize extension modules, which brings several advantages to extensions that implement it: The extension modules behave more like their Python counterparts. The extension modules can easily support loading into pre-existing module objects, which paves the way for extension module support for runpy or for systems that enable extension module reloading.
Loading multiple modules from the same extension is possible, which makes it possible to test module isolation (a key feature for proper sub-interpreter support) from a single interpreter. The biggest hurdle for adoption of PEP 489 is allowing access to module state from methods of extension types. Currently, the way to access this state from extension methods is by looking up the module via PyState_FindModule (in contrast to module level functions in extension modules, which receive a module reference as an argument).
However, PyState_FindModule queries the thread-local state, making it relatively costly compared to C level process global access and consequently deterring module authors from using it. Also, PyState_FindModule relies on the assumption that in each subinterpreter, there is at most one module corresponding to a given PyModuleDef . This assumption does not hold for modules that use PEP 489 ’s multi-phase initialization, so PyState_FindModule is unavailable for these modules.
A faster, safer way of accessing module-level state from extension methods is needed. Background The implementation of a Python method may need access to one or more of the following pieces of information: The instance it is called on ( self ) The underlying function The defining class , i. e. the class the method was defined in The corresponding module The module state In Python code, the Python-level equivalents may be retrieved as: import sys class Foo : def meth ( self ): instance = self module_globals = globals () module_object = sys . modules [ __name__ ] # (1) underlying_function = Foo . meth # (1) defining_class = Foo # (1) defining_class = __class__ # (2) Note The defining class is not type(self) , since type(self) might be a subclass of Foo .
The statements marked (1) implicitly rely on name-based lookup via the function’s __globals__ : either the Foo attribute to access the defining class and Python function object, or __name__ to find the module object in sys.modules . In Python code, this is feasible, as __globals__ is set appropriately when the function definition is executed, and even if the namespace has been manipulated to return a different object, at worst an exception will be raised.
The __class__ closure, (2), is a safer way to get the defining class, but it still relies on __closure__ being set appropriately. By contrast, extension methods are typically implemented as normal C functions. This means that they only have access to their arguments and C level thread-local and process-global states. Traditionally, many extension modules have stored their shared state in C-level process globals, causing problems when: running multiple initialize/finalize cycles in the same process reloading modules (e.g. to test conditional imports) loading extension modules in subinterpreters PEP 3121 attempted to resolve this by offering the PyState_FindModule API, but this still has significant problems when it comes to extension methods (rather than module level functions): it is markedly slower than directly accessing C-level process-global state there is still some inherent reliance on process global state that means it still doesn’t reliably handle module reloading It’s also the case that when looking up a C-level struct such as module state, supplying an unexpected object layout can crash the interpreter, so it’s significantly more important to ensure that extension methods receive the kind of object they expect.
Proposal Currently, a bound extension method ( PyCFunction or PyCFunctionWithKeywords ) receives only self , and (if applicable) the supplied positional and keyword arguments. While module-level extension functions already receive access to the defining module object via their self argument, methods of extension types don’t have that luxury: they receive the bound instance via self , and hence have no direct access to the defining class or the module level state.
The additional module level context described above can be made available with two changes. Both additions are optional; extension authors need to opt in to start using them: Add a pointer to the module to heap type objects. Pass the defining class to the underlying C function. In CPython, the defining class is readily available at the time the built-in method object ( PyCFunctionObject ) is created, so it can be stored in a new struct that extends PyCFunctionObject .
The module state can then be retrieved from the module objec
The rest of this page is on the live site.