About `defaultdict.__missing__` - Core Development - Discussions on Python.org
About `defaultdict.__missing__` Core Development storchaka (Serhiy Storchaka) April 10, 2026, 9:58am 1 The __missing__() method (if exists) is called in dict.__getitem__() if the key was not found in the dict, and its result is returned. It is not defined in the base dict , but defaultdict defines defaultdict.__missing__() which call default_factory and also sets the result as the value for the key, so following subscribing operations will not call __missing__() , but will return the same value for the same key.
It is commonly used for grouping values: d = defaultdict(list) ... d[key].append(value) All is fine until we try to use d[key] concurently. Then it is possible that different d[key] will return different values for the same key. See defaultdict.__missing__ has data races · Issue #142495 · python/cpython · GitHub . We can fix the race condition by using setdefault() instead of __setitem__() . This will guarantee that d[key] will always return the same value for the same key.
This was considered a nice bugfix, so this change was backported to 3.13 and 3.14. But it turned out, that the new behavior does not comform the documentation in some details. We do not know real examples of breaking user code, but this can affect some code trying to do low-level things with __missing__() literally following the documentation. See Behaviour change in defaultdict.__missing__ between 3.13.11 and 3.13.12 · Issue #148242 · python/cpython · GitHub .
What should we do? Update the documentation to match the new behavior in 3.13+. Revert the change in 3.13-3.14 (note that it was already released in bugfix releases), document the new behavior as new in 3.15, and maybe add some hints about concurrent unsafety in the 3.13-3.14 documentation. Revert the change in all branches. Let the user use external means to ensure atomicity. 1 Like Wombat (Zeke W) April 10, 2026, 10:50am 2 I vote for: “Revert the change in all branches.
Let the user use external means to ensure atomicity.” In the bug report, you note, “It seems that no one thought that __missing__() could be used directly.” However, there is real code that calls __missing__ directly. ChainMap is one example. And the person who filed with bug report has another example. Both rely on long standing, documented behavior that __missing__ can be called directly. Presumably there are also defaultdict` subclasses that do this well.
The defaultdict implementation should match the pure python versions of it found on the web: class DefaultDict(dict): def __init__(self, default_factory=None, *args, **kwargs): super().__init__(*args, **kwargs) # default_factory must be a callable or None self.default_factory = default_factory def __missing__(self, key): if self.default_factory is None: raise KeyError(key) # Call the factory, store the result, and return it value = self.default_factory() self[key] = value return value def __repr__(self): return f"DefaultDict({self.default_factory}, {super().__repr__()})" Note that this pure python version that people expect is also what pypy does: % pypy3 -c 'from collections import defaultdict; print(defaultdict(lambda: "A", {None: "B"}).__missing__(None))' A So, I think the edit needs to be reverted.
The old code is what people expect. It Is what was documented. It is what pypy does. It wasn’t a bug. It, like many multistep operations, was just inconveniently non-atomic. 3 Likes Changaco (Charly Coste) April 10, 2026, 11:26am 3 Since the change could silently break code that depends on the old behavior, it would be prudent to revert it from all branches. defaultdict can be made fully thread-safe without changing the behavior of __missing__ , by explicitly acquiring a lock instead of calling setdefault .
1 Like barry (Barry Warsaw) April 10, 2026, 2:37pm 4 I lean toward reverting also, although it’s unfortunate that the change has made it to bug fix releases. Definitely seek RM opinion/approval before making a change. BrenBarn (Brendan Barnwell) April 10, 2026, 8:11pm 5 How do you see that documentation as saying that __missing__ can be called directly? To me it just says that the key lookup operation on dict calls it.
It doesn’t say it can be called directly, nor that it cannot. Despite that I agree that the change should be reverted in all branches, since I think in general a method that is documented should be callable. (Maybe that’s what was meant in the quote above by saying it’s documented?) The documentation is silent on whether __missing__ is threadsafe. Given ambiguity on whether it can be called directly and ambiguity on whether it is threadsafe, I think the right move is to have a PEP to fully specify the desired behavior and only then implement any change (to docs or to behavior).
Allowing “backdoor” changes like this, where we update documentation to reflect an unannounced behavior change, damages users’ confidence in Python’s stability. Explicit behavior changes are better than implicit ones. oscarbenjamin (Oscar Benjamin) April 10, 2026, 8:34pm 6 Does anyone have any actual use that is broken by this change? Atomicity of normal usage seems more useful to me than making any promises about what happens calling __missing__ explicitly.
You don’t have to call __missing__ when you can just call the factory function directly if that is what you want. 2 Likes pf_moore (Paul Moore) April 10, 2026, 8:59pm 7 I think it’s huge overkill to require a PEP for something this obscure. Do you genuinely imagine that any participants in a thread for such a PEP would actually have real-world experience of using this feature? It would be a clear case of “design by committee”, and wouldn’t (IMO) improve the outcome in any measurable way.
Using the term “backdoor” for a change made by the normal PR process seems very extreme as well - it’s quite literally a core dev’s job to make decisions like this as part of implementing a PR. In this case, the decision seems to have caused some problems. It’s a shame these didn’t get caught before release, and even more so that it got into bugfix releases, but mistakes happen. I don’t have a strong opinion as to whether it’s worth changing the documented behaviour to allow atomicity.
Depending on the decision about that I think one of “revert the change in 3.13-3.14” or “revert in all branches” should apply. Whichever happens, the choice should be made deliberately and documented properly, but I don’t think we should over-react and require anything more than a PR to do so. While I said I don’t have a strong opinion, this argument does make sense to me, so count me as weakly in favour of this option.
4 Likes BrenBarn (Brendan Barnwell) April 11, 2026, 5:58am 8 I could be persuaded that we don’t need a PEP, but we at least need this kind of public discussion to see if we need a PEP. That is, we need the kind of discussion we’re having now, and I appreciate that we’re having it. We won’t know unless we have the discussion somewhere more public than on the PR. Maybe, but we won’t know unless we have the discussion somewhere more public than on the PR.
What I meant by “backdoor” is no so much the PR process as the sequence of events that goes: seemingly small and/or beneficial change made unexpected consequence discovered (change in behavior, divergence from documentation, etc,) the change in step 1 is considered “status quo” because it has already been made instead of undoing the change, we retcon the docs to specify the behavior that was accidentally implemented In short, changes in behavior, and especially changes that tread close to lines of ambiguity in docs, should always be made with full awareness, not just left in because we didn’t notice the problem at the time the change was made.
If something happens and we later realize it contradicts the docs, the default response should be to roll it back, and then maybe later we can think if we want to change the docs to clarify, a
The rest of this page is on the live site.