Generator memory leak · Issue #61670 · python/cpython · GitHub
Skip to content You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert Generator memory leak #61670 New issue Copy link New issue Copy link Closed Closed Generator memory leak #61670 Copy link Labels interpreter-core (Objects, Python, Grammar, and Parser dirs) (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error An unexpected behavior, bug, or error Description AnssiKriinen mannequin opened on Mar 18, 2013 Issue body actions BPO 17468 Nosy @amauryfa , @ncoghlan , @pitrou , @vstinner , @benjaminp , @phmc Superseder bpo-17807 : Generator cleanup without tp_del Files test.py Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details GitHub fields: assignee = None closed_at = < Date 2013 - 05 - 10.17 : 39 : 50.874 > created_at = < Date 2013 - 03 - 18.20 : 34 : 25.229 > labels = [ 'interpreter-core' , 'type-bug' ] title = 'Generator memory leak' updated_at = < Date 2015 - 10 - 22.14 : 18 : 14.858 > user = 'https://bugs.python.org/AnssiKriinen' bugs.python.org fields: activity = < Date 2015 - 10 - 22.14 : 18 : 14.858 > actor = 'Vikas' assignee = 'none' closed = True closed_date = < Date 2013 - 05 - 10.17 : 39 : 50.874 > closer = 'pitrou' components = [ 'Interpreter Core' ] creation = < Date 2013 - 03 - 18.20 : 34 : 25.229 > creator = 'Anssi.K \xc3 \xa4 \xc3 \xa4 ri \xc3 \xa4 inen' dependencies = [] files = [ '29455' ] hgrepos = [] issue_num = 17468 keywords = [] message_count = 26.0 messages = [ '184515' , '184635' , '184671' , '187354' , '187359' , '187360' , '187367' , '187369' , '187372' , '187374' , '187375' , '187403' , '187408' , '187410' , '187413' , '187415' , '187419' , '187423' , '187424' , '187425' , '187431' , '187434' , '187437' , '187483' , '188854' , '253331' ] nosy_count = 10.0 nosy_names = [ 'amaury.forgeotdarc' , 'ncoghlan' , 'pitrou' , 'vstinner' , 'benjamin.peterson' , 'docs@python' , 'Anssi.K \xc3 \xa4 \xc3 \xa4 ri \xc3 \xa4 inen' , 'pconnell' , 'isoschiz' , 'Vikas' ] pr_nums = [] priority = 'normal' resolution = 'duplicate' stage = None status = 'closed' superseder = '17807' type = 'behavior' url = 'https://bugs.python.org/issue17468' versions = [ 'Python 2.7' , 'Python 3.3' , 'Python 3.4' ] Activity Next AnssiKriinen commented on Mar 18, 2013 AnssiKriinen mannequin on Mar 18, 2013 Mannequin Author More actions A generator is leaked to gc.garbage in a situation where __del__ isn't defined.
See the attached file for as-minimalistic test case as I could make. Tested on Python 3.3.0, 3.2.3 and 2.7.3. Note that if the try-except is removed from iterator(), then there is no leak. This is related to Django bug bpo-19895 ( https://code.djangoproject.com/ticket/19895 ). AnssiKriinen added performance Performance or resource usage Performance or resource usage on Mar 18, 2013 amauryfa commented on Mar 19, 2013 amauryfa on Mar 19, 2013 Contributor More actions Well, there is a __del__: the one from the generator function.
And it's part of the cycle. AnssiKriinen commented on Mar 19, 2013 AnssiKriinen mannequin on Mar 19, 2013 Mannequin Author More actions I am trying to read the code, and it seems objects of type generator are uncollectable if the code of the generator has a block of other type than SETUP_LOOP. I can see how try-finally for example in the generator would cause problems - the finally might reference objects which are already collected.
But plain try-except? What is the problem with that? (Not saying there isn't one, I am new to the code...). I tried to change the code to also allow collecting generators with plain try-except (add in a check for SETUP_EXCEPTION in addition to SETUP_LOOP in PyGen_NeedsFinalizing()). This resolves the leak, and all tests pass. I can't tell if it is actually safe to return 0 in SETUP_EXCEPTION case. After the change try-finally will still leak but that seems correct.
Maybe this limitation should be documented? I think I understand now why finally or with statement inside a cyclic referenced generator will end up in gc.garbage. However the docs tell that only objects with __del__ defined will leak. From user perspective generator doesn't have __del__, at least it is not to be found with pdb. Generators have potentially unsafe finalizer code but that isn't technically equivalent to having __del__ method defined as far as I can tell. benjaminp commented on Apr 19, 2013 benjaminp on Apr 19, 2013 Contributor More actions Generators exit using GeneratorExit, which you could possible catch.
AnssiKriinen commented on Apr 19, 2013 AnssiKriinen mannequin on Apr 19, 2013 Mannequin Author More actions True, except GeneratorExit will run at garbage collection time and this will result in reference cycle problems. Checking if there is no except GeneratorExit clause might be too complicated. I still think this is worth a brief note in the gc docs. The gc docs talk only about __del__ methods, and there isn't one defined for the example generator.
The generator does have something technically equivalent, but I don't spot any actual __del__ methods in the reference loop. A docs mention will not help much in avoiding this problem, but it will at least make the code behaviour equivalent to documentation. A mention in the gc docs would have helped me when trying to debug the leak in Django. benjaminp commented on Apr 19, 2013 benjaminp on Apr 19, 2013 Contributor More actions A documentation patch sounds good to me. benjaminp added docs Documentation in the Doc dir Documentation in the Doc dir on Apr 19, 2013 benjaminp assigned docspython on Apr 19, 2013 benjaminp added type-bug An unexpected behavior, bug, or error An unexpected behavior, bug, or error and removed performance Performance or resource usage Performance or resource usage on Apr 19, 2013 ncoghlan commented on Apr 19, 2013 ncoghlan on Apr 19, 2013 Contributor More actions Adding Guido because this appears to be due to a longstanding difference between the handling of tp_del and most other slots Specifically, the reason for the odd behaviour appears to be that generator objects define tp_del [1] (which is what the cyclic gc *really* looks for), but while defining __del__ in Python will populate tp_del in the type slots, defining tp_del directly in a builtin or extension type doesn't create an exposed __del__ at the Python level (there's no wrapper function identified in the slot definition).
So, at the very least, the fact that builtin and extension types can define tp_del without creating a visible __del__ method needs to be documented as a CPython implementation detail. However, I'm not sure we actually have a good *reason* for tp_del failing to generate __del__ automatically - it's been that way since Guido implemented it [2], but Guido's comment in that commit message about not needing a wrapper because types written in C can't define tp_del is clearly no longer accurate. [1] http://hg.python.org/cpython/file/2.7/Objects/genobject.c#l328 [2] http://hg.python.org/cpython/annotate/71dca7c76fa2/Objects/typeobject.c#3955 pitrou commented on Apr 19, 2013 pitrou on Apr 19, 2013 Member More actions With the advent of yield-based asynchronous programming, it is going to be problematic if a generator caught in a reference cycle can create a memory leak. benjaminp commented on Apr 19, 2013 benjaminp on Apr 19, 2013 Contributor More actions I think the creation of __del__ wrappers for extension types is separate from this issue of generator finalization. benjaminp commented on Apr 19, 2013 benjaminp on Apr 19, 2013 Contributor More actions Yes.
Hopefully, the async framework using generators can properly can close() on them, though. 6 remaining items AnssiKriinen commented on Apr 20, 2013 AnssiKriinen mannequin on Apr 20, 2013 Mannequin Author More actions I was imag
The rest of this page is on the live site.