PEP 454 – Add a new tracemalloc module to trace Python memory allocations | peps.python.org
PEP 454 – Add a new tracemalloc module to trace Python memory allocations PEP 454 – Add a new tracemalloc module to trace Python memory allocations Author : Victor Stinner <vstinner at python.org> BDFL-Delegate : Charles-François Natali <cf.natali at gmail.com> Status : Final Type : Standards Track Created : 03-Sep-2013 Python-Version : 3.4 Resolution : Python-Dev message Table of Contents Abstract Rationale Proposal API Functions Filter Frame Snapshot Statistic StatisticDiff Trace Traceback Rejected Alternatives Log calls to the memory allocator Prior Work Links Copyright Abstract This PEP proposes to add a new tracemalloc module to trace memory blocks allocated by Python.
Rationale Classic generic tools like Valgrind can get the C traceback where a memory block was allocated. Using such tools to analyze Python memory allocations does not help because most memory blocks are allocated in the same C function, in PyMem_Malloc() for example. Moreover, Python has an allocator for small objects called “pymalloc” which keeps free blocks for efficiency. This is not well handled by these tools.
There are debug tools dedicated to the Python language like Heapy Pympler and Meliae which lists all alive objects using the garbage collector module (functions like gc.get_objects() , gc.get_referrers() and gc.get_referents() ), compute their size (ex: using sys.getsizeof() ) and group objects by type. These tools provide a better estimation of the memory usage of an application. They are useful when most memory leaks are instances of the same type and this type is only instantiated in a few functions.
Problems arise when the object type is very common like str or tuple , and it is hard to identify where these objects are instantiated. Finding reference cycles is also a difficult problem. There are different tools to draw a diagram of all references. These tools cannot be used on large applications with thousands of objects because the diagram is too huge to be analyzed manually. Proposal Using the customized allocation API from PEP 445 , it becomes easy to set up a hook on Python memory allocators.
A hook can inspect Python internals to retrieve Python tracebacks. The idea of getting the current traceback comes from the faulthandler module. The faulthandler dumps the traceback of all Python threads on a crash, here is the idea is to get the traceback of the current Python thread when a memory block is allocated by Python. This PEP proposes to add a new tracemalloc module, a debug tool to trace memory blocks allocated by Python.
The module provides the following information: Traceback where an object was allocated Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks Computed differences between two snapshots to detect memory leaks The API of the tracemalloc module is similar to the API of the faulthandler module: enable() / start() , disable() / stop() and is_enabled() / is_tracing() functions, an environment variable ( PYTHONFAULTHANDLER and PYTHONTRACEMALLOC ), and a -X command line option ( -X faulthandler and -X tracemalloc ).
See the documentation of the faulthandler module . The idea of tracing memory allocations is not new. It was first implemented in the PySizer project in 2005. PySizer was implemented differently: the traceback was stored in frame objects and some Python types were linked the trace with the name of object type. PySizer patch on CPython adds an overhead on performances and memory footprint, even if the PySizer was not used. tracemalloc attaches a traceback to the underlying layer, to memory blocks, and has no overhead when the module is not tracing memory allocations.
The tracemalloc module has been written for CPython. Other implementations of Python may not be able to provide it. API To trace most memory blocks allocated by Python, the module should be started as early as possible by setting the PYTHONTRACEMALLOC environment variable to 1 , or by using -X tracemalloc command line option. The tracemalloc.start() function can be called at runtime to start tracing Python memory allocations.
By default, a trace of an allocated memory block only stores the most recent frame (1 frame). To store 25 frames at startup: set the PYTHONTRACEMALLOC environment variable to 25 , or use the -X tracemalloc=25 command line option. The set_traceback_limit() function can be used at runtime to set the limit. Functions clear_traces() function: Clear traces of memory blocks allocated by Python. See also stop() . get_object_traceback(obj) function: Get the traceback where the Python object obj was allocated.
Return a Traceback instance, or None if the tracemalloc module is not tracing memory allocations or did not trace the allocation of the object. See also gc.get_referrers() and sys.getsizeof() functions. get_traceback_limit() function: Get the maximum number of frames stored in the traceback of a trace. The tracemalloc module must be tracing memory allocations to get the limit, otherwise an exception is raised. The limit is set by the start() function. get_traced_memory() function: Get the current size and maximum size of memory blocks traced by the tracemalloc module as a tuple: (size: int, max_size: int) . get_tracemalloc_memory() function: Get the memory usage in bytes of the tracemalloc module used to store traces of memory blocks.
Return an int . is_tracing() function: True if the tracemalloc module is tracing Python memory allocations, False otherwise. See also start() and stop() functions. start(nframe: int=1) function: Start tracing Python memory allocations: install hooks on Python memory allocators. Collected tracebacks of traces will be limited to nframe frames. By default, a trace of a memory block only stores the most recent frame: the limit is 1 . nframe must be greater or equal to 1 .
Storing more than 1 frame is only useful to compute statistics grouped by 'traceback' or to compute cumulative statistics: see the Snapshot.compare_to() and Snapshot.statistics() methods. Storing more frames increases the memory and CPU overhead of the tracemalloc module. Use the get_tracemalloc_memory() function to measure how much memory is used by the tracemalloc module. The PYTHONTRACEMALLOC environment variable ( PYTHONTRACEMALLOC=NFRAME ) and the -X tracemalloc=NFRAME command line option can be used to start tracing at startup.
See also stop() , is_tracing() and get_traceback_limit() functions. stop() function: Stop tracing Python memory allocations: uninstall hooks on Python memory allocators. Clear also traces of memory blocks allocated by Python Call take_snapshot() function to take a snapshot of traces before clearing them. See also start() and is_tracing() functions. take_snapshot() function: Take a snapshot of traces of memory blocks allocated by Python.
Return a new Snapshot instance. The snapshot does not include memory blocks allocated before the tracemalloc module started to trace memory allocations. Tracebacks of traces are limited to get_traceback_limit() frames. Use the nframe parameter of the start() function to store more frames. The tracemalloc module must be tracing memory allocations to take a snapshot, see the start() function. See also the get_object_traceback() function.
Filter Filter(inclusive: bool, filename_pattern: str, lineno: int=None, all_frames: bool=False) class: Filter on traces of memory blocks. See the fnmatch.fnmatch() function for the syntax of filename_pattern . The '.pyc' and '.pyo' file extensions are replaced with '.py' . Examples: Filter(True, subprocess.__file__) only includes traces of the subprocess module Filter(False, tracemalloc.__file__) excludes traces of the tracemalloc module Filter(False, "<unknown>") excludes empty tracebacks inclusive attribute: If inclusive is True (include), only trace memory blocks allocated in a file with a name matching filename_pattern at line number lineno .
If inclusive is False (exclude), ig
The rest of this page is on the live site.