PEP 697 – Limited C API for Extending Opaque Types | peps.python.org
PEP 697 – Limited C API for Extending Opaque Types PEP 697 – Limited C API for Extending Opaque Types Author : Petr Viktorin <encukou at gmail.com> Discussions-To : Discourse thread Status : Final Type : Standards Track Created : 23-Aug-2022 Python-Version : 3.12 Post-History : 24-May-2022 , 06-Oct-2022 Resolution : Discourse message Table of Contents Abstract Motivation Rationale Extending opaque types Extending variable-size objects Variable-size layouts Extending classes with the PyHeapTypeObject -like layout Big picture Relative member offsets Specification Relative basicsize Inheriting itemsize Relative member offsets List of new API Backwards Compatibility Assumptions Security Implications Endorsements How to Teach This Reference Implementation Possible Future Enhancements Alignment & Performance Other layouts for variable-size types Rejected Ideas Footnotes Copyright Important This PEP is a historical document.
The up-to-date, canonical documentation can now be found at PyType_Spec.basicsize , PyObject_GetTypeData() , Py_TPFLAGS_ITEMS_AT_END , Py_RELATIVE_OFFSET , PyObject_GetItemData() . × See PEP 1 for how to propose changes. Abstract Add Limited C API support for extending some types with opaque data by allowing code to only deal with data specific to a particular (sub)class. This mechanism is required to be usable with PyHeapTypeObject .
This PEP does not propose allowing to extend non-dynamically sized variable sized objects such as tuple or int due to their different memory layout and perceived lack of demand for doing so. This PEP leaves room to do so in the future via the same mechanism if ever desired. Motivation The motivating problem this PEP solves is attaching C-level state to custom types — i.e. metaclasses (subclasses of type ). This is often needed in “wrappers” that expose another type system (e.g.
C++, Java, Rust) as Python classes. These typically need to attach information about the “wrapped” non-Python class to the Python type object. This should be possible to do in the Limited API, so that the language wrappers or code generators can be used to create Stable ABI extensions. (See PEP 652 for the benefits of providing a stable ABI.) Extending type is an instance of a more general problem: extending a class while maintaining loose coupling – that is, not depending on the memory layout used by the superclass. (That’s a lot of jargon; see Rationale for a concrete example of extending list .) Rationale Extending opaque types In the Limited API, most struct s are opaque: their size and memory layout are not exposed, so they can be changed in new versions of CPython (or alternate implementations of the C API).
This means that the usual subclassing pattern – making the struct used for instances of the base type be the first element of the struct used for instances of the derived type – does not work. To illustrate with code, the example from the tutorial extends PyListObject ( list ) using the following struct : typedef struct { PyListObject list ; int state ; } SubListObject ; This won’t compile in the Limited API, since PyListObject is opaque (to allow changes as features and optimizations are implemented).
Instead, this PEP proposes using a struct with only the state needed in the subclass, that is: typedef struct { int state ; } SubListState ; // (or just `typedef int SubListState;` in this case) The subclass can now be completely decoupled from the memory layout (and size) of the superclass. This is possible today. To use such a struct: when creating the class, use PyListObject->tp_basicsize + sizeof(SubListState) as PyType_Spec.basicsize ; when accessing the data, use PyListObject->tp_basicsize as the offset into the instance ( PyObject* ).
However, this has disadvantages: The base’s basicsize may not be properly aligned, causing issues on some architectures if not mitigated. (These issues can be particularly nasty if alignment changes in a new release.) PyTypeObject.tp_basicsize is not exposed in the Limited API, so extensions that support Limited API need to use PyObject_GetAttrString(obj, "__basicsize__") . This is cumbersome, and unsafe in edge cases (the Python attribute can be overridden).
Variable-size objects are not handled (see Extending variable-size objects below). To make this easy (and even best practice for projects that choose loose coupling over maximum performance), this PEP proposes an API to: During class creation, specify that SubListState should be “appended” to PyListObject , without passing any additional details about list . (The interpreter itself gets all necessary info, like tp_basicsize , from the base).
This will be specified by a negative PyType_Spec.basicsize : -sizeof(SubListState) . Given an instance, and the subclass PyTypeObject* , get a pointer to the SubListState . A new function, PyObject_GetTypeData , will be added for this. The base class is not limited to PyListObject , of course: it can be used to extend any base class whose instance struct is opaque, unstable across releases, or not exposed at all – including type ( PyHeapTypeObject ) or third-party extensions (for example, NumPy arrays [1] ).
For cases where no additional state is needed, a zero basicsize will be allowed: in that case, the base’s tp_basicsize will be inherited. (This currently works, but lacks explicit documentation and tests.) The tp_basicsize of the new class will be set to the computed total size, so code that inspects classes will continue working as before. Extending variable-size objects Additional considerations are needed to subclass variable-sized objects while maintaining loose coupling: the variable-sized data can collide with subclass data ( SubListState in the example above).
Currently, CPython doesn’t provide a way to prevent such collisions. So, the proposed mechanism of extending opaque classes (negative base->tp_itemsize ) will fail by default. We could stop there, but since the motivating type — PyHeapTypeObject — is variable sized, we need a safe way to allow subclassing it. A bit of background first: Variable-size layouts There are two main memory layouts for variable-sized objects.
In types such as int or tuple , the variable data is stored at a fixed offset. If subclasses need additional space, it must be added after any variable-sized data: PyTupleObject: ┌───────────────────┬───┬───┬╌╌╌╌┐ │ PyObject_VAR_HEAD │var. data │ └───────────────────┴───┴───┴╌╌╌╌┘ tuple subclass: ┌───────────────────┬───┬───┬╌╌╌╌┬─────────────┐ │ PyObject_VAR_HEAD │var. data │subclass data│ └───────────────────┴───┴───┴╌╌╌╌┴─────────────┘ In other types, like PyHeapTypeObject , variable-sized data always lives at the end of the instance’s memory area: heap type: ┌───────────────────┬──────────────┬───┬───┬╌╌╌╌┐ │ PyObject_VAR_HEAD │Heap type data│var. data │ └───────────────────┴──────────────┴───┴───┴╌╌╌╌┘ type subclass: ┌───────────────────┬──────────────┬─────────────┬───┬───┬╌╌╌╌┐ │ PyObject_VAR_HEAD │Heap type data│subclass data│var. data │ └───────────────────┴──────────────┴─────────────┴───┴───┴╌╌╌╌┘ The first layout enables fast access to the items array.
The second all
The rest of this page is on the live site.