Text of discuss.python.org › t › adding-c-api-for-use-in-tp-traverse › 105331 as Styx crawled it on Nov 19, 2026. The page may have changed since. Only its words are kept, without pictures or layout, and only the first part of a long page.

Open the live page

Adding C API for use in tp_traverse - C API - Discussions on Python.org

Adding C API for use in tp_traverse Core Development C API encukou (Petr Viktorin) December 17, 2025, 1:48pm 1 There are newly documented restrictions on tp_traverse : gh-123241: Document restrictions for `tp_traverse` implementations by colesbury · Pull Request #142272 · python/cpython · GitHub The traversal function must not have any side effects. It must not modify the reference counts of any Python objects nor create or destroy any Python objects.

This primarily means you can’t use functions that return new references, or ones that raise (new) exceptions. (Basically, the kind of functions we’re trying to add nowadays.) But you also can’t use any C API function, unless it guarantees that it has no side effects. I don’t think we guarantee that for anything. You typically don’t need to do much in tp_traverse . But, both current PEPs for a free-threading stable ABI (PEP 803 & PEP 809) require some newer APIs for module state and PEP 697 layouts (instance structs that don’t include PyObject ).

So, I’d like to add variants for existing functions that are usable in this context – they’d: never set an exception (on failure, only return NULL ) return borrowed references do not have “internal” side effects (for example, call a “normal C API function and then DECREF the result) For the first two I’d normally name the functions with _NoError and _Borrow suffixes, but, perhaps that’s included in “safe to use in GC callbacks”.

Would _GCSafe work? PyObject_GetTypeData_GCSafe (doesn’t set exceptions) PyObject_GetItemData_GCSafe (ditto) PyType_GetModuleState_GCSafe (ditto) PyModule_GetState_GCSafe (ditto) PyModule_GetToken_GCSafe (ditto) PyType_GetBaseByToken_GCSafe (doesn’t set exceptions & returns borrowed ref) PyType_GetModule_GCSafe (ditto) PyType_GetModuleByToken_GCSafe (ditto) Additionally, document Py_TYPE & Py_SIZE [edit: and PyObject_VisitManagedDict ] as GC-safe.

Does that make sense? 1 Like davidhewitt (David Hewitt) December 19, 2025, 2:49pm 2 I guess that these new variants should be discouraged outside of use during GC because the borrowed references (and lack of exceptions) are generally worse for UX? Should the name be even stronger, maybe a TpTraverse suffix or prefix, e.g. PyObject_GetTypeData_TpTraverse , or PyTpTraverse_Object_GetTypeData ? This way it’s more clear (to me) these functions are only for that use case, because they’re so specialized it probably doesn’t matter if their names are long. da-woods (Da Woods) December 19, 2025, 4:26pm 3 At the risk of pointing out the something that’s absolutely obvious and trivial: PyObject_VisitManagedDict is hopefully GC-safe (and looks to be documented as such).

1 Like encukou (Petr Viktorin) December 19, 2025, 4:27pm 4 Good question. As for UX: I expect users to define their own helpers on top of these (most importantly, to cast the result PyModule_GetState ). Defining two variants is, unfortunately, worse for UX than borrowed refs & lack of exceptions. What I’m more worried about is borrowed references themselves. Without GIL, if another thread changes a class’s MRO, the PyType_GetBase* could become invalid.

This currently can’t happen: both GC and MRO changes are stop-the-world operations. But that doesn’t seem like something we want to guarantee in the long term. ngoldbaum (Nathan Goldbaum) December 19, 2025, 8:39pm 5 Maybe the stop-the-world part is the key aspect and the part that should be highlighted in the name? I’m not sure if the restrictions you’re thinking of are exactly the same as the ones for code happening during stop-the-world events, but it sounds similar to what @nas described to me when he was doing work to make mutating type objects thread-safe.

This is a teeny bit off-topic for this thread, but I want to bring up here that I bet that a future version of Python will expose a C API to trigger a stop-the-world pause. Memray needs it, for one: memray/src/memray/_memray/compat.h at 64c0e81806ba01aaa11d8d98e05c479c30982fae · bloomberg/memray · GitHub I suspect other profilers will end up using the private API for stopping the world since it’s needed to do prevent races while observing the Python stack.

Better to come up with defined and documented semantics for doing this stuff. 1 Like emmatyping (Emma Smith) December 19, 2025, 9:01pm 6 As @encukou says at the end there, we probably don’t want to guarantee stop-the-world in the long-term as it’s really an implementation detail that could change. I’d like to keep the door open for a different design for the GC that may not be stop-the-world. encukou (Petr Viktorin) December 22, 2025, 3:16pm 7 In #142376 we have a request for allowing PyObject_GetTypeData without an attached thread state .

It looks like the use case would apply for all the API listed here, which suggests that it might make sense to include that in the design. Let’s say that for the new _GCSafe [1] API: CPython guarantee that the APIs are strictly read-only; they can’t change refcounts or create new objects. This makes them safe to use in tp_traverse . CPython requires callers to ensure that the data the API “touches” isn’t concurrently modified.

That data being: the type of any PyObject* argument the MRO of PyTypeObject* argument for completeness: type layout data – flags, basicsize , itemsize , and similar. (Ideally we’d tell people to always treat those as immutable, but I don’t think we currently do…) And we can let users ensure that by not changing it themselves and: Call the functions from tp_traverse , or Have an attached thread state (which locks CPython into changing type/bases being a stop-the-world operation), or (to address #142376 :‍) Use some new API to acquire a “reader lock on types & memory layouts” – instead of an attached thread state, you’d get a lock that prevents stop-the-world operations [2] , but while you hold it, you’re only allowed to use the “ _GCSafe ” API. (I’m assuming that this could be significantly cheaper than attaching a thread state.) We also need to outlaw directly setting ob_type , tp_bases , tp_basicsize etc., but I assume that’s already part of making an extension free-threading-compatible.

Does that sound like a reasonable direction? it needs a better name; let me stick with this for now ↩︎ Or only type/MRO reassignments? ↩︎ colesbury (Sam Gross) December 22, 2025, 3:47pm 8 I’m not convinced these are worth the complexity of adding new functions, but I don’t feel super strongly. I guess the idea is that they’re not safe if they are called incorrectly and set an exception, but: Just don’t call them incorrectly.

These exceptions are all of the “programmer error” type, not runtime error. Pretty much nobody checks the exception anyways. They’re useless. We’ve made our lives more difficult by making these simple functions capable of failing. I think the idea is that validating arguments and raising a Python error is somehow nicer for C API users than a C assertion or fatal error, but that totally ignores how people actually use these APIs.

6 Likes encukou (Petr Viktorin) December 22, 2025, 4:24pm 9 Fair, I partially agree. There was a long discussion on fallible APIs; I don’t think we need to rehash it at this point. If fallibility was the only issue, I think we could make things work with the current API. But the main issue is the borrowed references. In “normal” code, another thread can invalidate borrowed refs; in tp_traverse we can’t create strong refs. da-woods (Da Woods) December 22, 2025, 9:16pm 10 As the requester of the thread-state-less PyObject_GetTypeData this is going to sound a bit hypocritical.

However: What I’m a bit worried about is that the rules are very tied to “these things are done with stop the world” (which @emmatyping says, does feel like a real implementation detail). The opaque type data is mainly useful for the Limited API, where ideally we want to stay away f

The rest of this page is on the live site.