PEP 733 – An Evaluation of Python’s Public C API | peps.python.org
PEP 733 – An Evaluation of Python’s Public C API PEP 733 – An Evaluation of Python’s Public C API Author : Erlend Egeberg Aasland <erlend at python.org>, Domenico Andreoli <domenico.andreoli at linux.com>, Stefan Behnel <stefan_ml at behnel.de>, Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>, Simon Cross <hodgestar at gmail.com>, Steve Dower <steve.dower at python.org>, Tim Felgentreff <tim.felgentreff at oracle.com>, David Hewitt <1939362+davidhewitt at users.noreply.github.com>, Shantanu Jain <hauntsaninja at gmail.com>, Wenzel Jakob <wenzel.jakob at epfl.ch>, Irit Katriel <irit at python.org>, Marc-Andre Lemburg <mal at lemburg.com>, Donghee Na <donghee.na at python.org>, Karl Nelson <nelson85 at llnl.gov>, Ronald Oussoren <ronaldoussoren at mac.com>, Antoine Pitrou <solipsis at pitrou.net>, Neil Schemenauer <nas at arctrix.com>, Mark Shannon <mark at hotpy.org>, Stepan Sindelar <stepan.sindelar at oracle.com>, Gregory P.
Smith <greg at krypto.org>, Eric Snow <ericsnowcurrently at gmail.com>, Victor Stinner <vstinner at python.org>, Guido van Rossum <guido at python.org>, Petr Viktorin <encukou at gmail.com>, Carol Willing <willingc at gmail.com>, William Woodruff <william at yossarian.net>, David Woods <dw-git at d-woods.co.uk>, Jelle Zijlstra <jelle.zijlstra at gmail.com> Status : Final Type : Informational Created : 16-Oct-2023 Post-History : 01-Nov-2023 Table of Contents Abstract Introduction C API Stakeholders Common Actions for All Stakeholders Extension Writers Authors of Embedded Python Applications Python Implementations Alternative APIs and Binding Generators Strengths of the C API C API problems API Evolution and Maintenance API Specification and Abstraction Object Reference Management Type Definition and Object Creation Error Handling API Tiers and Stability Guarantees Use of the C Language Implementation Flaws Missing Functionality Debug Mode Introspection Improved Interaction with Other Languages References Copyright Abstract This informational PEP describes our shared view of the public C API.
The document defines: purposes of the C API stakeholders and their particular use cases and requirements strengths of the C API problems of the C API categorized into nine areas of weakness This document does not propose solutions to any of the identified problems. By creating a shared list of C API issues, this document will help to guide continuing discussion about change proposals and to identify evaluation criteria.
Introduction Python’s C API was not designed for the different purposes it currently fulfills. It evolved from what was initially the internal API between the C code of the interpreter and the Python language and libraries. In its first incarnation, it was exposed to make it possible to embed Python into C/C++ applications and to write extension modules in C/C++. These capabilities were instrumental to the growth of Python’s ecosystem.
Over the decades, the C API grew to provide different tiers of stability, conventions changed, and new usage patterns have emerged, such as bindings to languages other than C/C++. In the next few years, new developments are expected to further test the C API, such as the removal of the GIL and the development of a JIT compiler. However, this growth was not supported by clearly documented guidelines, resulting in inconsistent approaches to API design in different subsystems of CPython.
In addition, CPython is no longer the only implementation of Python, and some of the design decisions made when it was, are difficult for alternative implementations to work with [ Issue 64 ]. In the meantime, lessons were learned and mistakes in both the design and the implementation of the C API were identified. Evolving the C API is hard due to the combination of backwards compatibility constraints and its inherent complexity, both technical and social.
Different types of users bring different, sometimes conflicting, requirements. The tradeoff between stability and progress is an ongoing, highly contentious topic of discussion when suggestions are made for incremental improvements. Several proposals have been put forward for improvement, redesign or replacement of the C API, each representing a deep analysis of the problems. At the 2023 Language Summit, three back-to-back sessions were devoted to different aspects of the C API.
There is general agreement that a new design can remedy the problems that the C API has accumulated over the last 30 years, while at the same time updating it for use cases that it was not originally designed for. However, there was also a sense at the Language Summit that we are trying to discuss solutions without a clear common understanding of the problems that we are trying to solve. We decided that we need to agree on the current problems with the C API, before we are able to evaluate any of the proposed solutions.
We therefore created the capi-workgroup repository on GitHub in order to collect everyone’s ideas on that question. Over 60 different issues were created on that repository, each describing a problem with the C API. We categorized them and identified a number of recurring themes. The sections below mostly correspond to these themes, and each contains a combined description of the issues raised in that category, along with links to the individual issues.
In addition, we included a section that aims to identify the different stakeholders of the C API, and the particular requirements that each of them has. C API Stakeholders As mentioned in the introduction, the C API was originally created as the internal interface between CPython’s interpreter and the Python layer. It was later exposed as a way for third-party developers to extend and embed Python programs. Over the years, new types of stakeholders emerged, with different requirements and areas of focus.
This section describes this complex state of affairs in terms of the actions that different stakeholders need to perform through the C API. Common Actions for All Stakeholders There are actions which are generic, and required by all types of API users: Define functions and call them Define new types Create instances of builtin and user-defined types Perform operations on object instances Introspect objects, including types, instances, and functions Raise and handle exceptions Import modules Access to Python’s OS interface The following sections look at the unique requirements of various stakeholders.
Extension Writers Extension writers are the traditional users of the C API. Their requirements are the common actions listed above. They also commonly need to: Create new modules Efficiently interface between modules at the C level Authors of Embedded Python Applications Applications with an embedded Python interpreter. Examples are Blender and OBS . They need to be able to: Configure the interpreter (import paths, inittab, sys.argv , memory allocator, etc.).
Interact with the execution model and program lifetime, including clean interpreter shutdown and restart. Represent complex data models in a way Python can use without having to create deep copies. Provide and import frozen modules. Run and manage multiple independent interpreters (in particular, when embedded in a library that wants to avoid global effects). Python Implementations Python implementations such as CPython , PyPy , GraalPy , IronPython , RustPython , MicroPython , and Jython ), may take very different approaches for the implementation of different subsystems.
They need: The API to be abstract and hide implementation details. A specification of the API, ideally with a test suite that ensures compatibility. It would be nice to have an ABI that can be shared across Python implementations. Alternative APIs and Binding Generators There are several projects that implement alternatives to the C API, which offer extension users advantanges over programming directly with the C API.
These APIs are implemented with the C API, and in some cases by usin
The rest of this page is on the live site.