PEP 471 – os.scandir() function – a better and faster directory iterator | peps.python.org
PEP 471 – os.scandir() function – a better and faster directory iterator PEP 471 – os.scandir() function – a better and faster directory iterator Author : Ben Hoyt <benhoyt at gmail.com> BDFL-Delegate : Victor Stinner <vstinner at python.org> Status : Final Type : Standards Track Created : 30-May-2014 Python-Version : 3.5 Post-History : 27-Jun-2014, 08-Jul-2014, 14-Jul-2014 Table of Contents Abstract Rationale Implementation Specifics of proposal os.scandir() os.walk() Examples Notes on caching Notes on exception handling Support Use in the wild Rejected ideas Naming Wildcard support Methods not following symlinks by default DirEntry attributes being properties DirEntry fields being “static” attribute-only objects DirEntry fields being static with an ensure_lstat option Return values being (name, stat_result) two-tuples Return values being overloaded stat_result objects Return values being pathlib.Path objects Possible improvements Previous discussion Copyright Abstract This PEP proposes including a new directory iteration function, os.scandir() , in the standard library.
This new function adds useful functionality and increases the speed of os.walk() by 2-20 times (depending on the platform and file system) by avoiding calls to os.stat() in most cases. Rationale Python’s built-in os.walk() is significantly slower than it needs to be, because – in addition to calling os.listdir() on each directory – it executes the stat() system call or GetFileAttributes() on each file to determine whether the entry is a directory or not.
But the underlying system calls – FindFirstFile / FindNextFile on Windows and readdir on POSIX systems – already tell you whether the files returned are directories or not, so no further system calls are needed. Further, the Windows system calls return all the information for a stat_result object on the directory entry, such as file size and last modification time. In short, you can reduce the number of system calls required for a tree function like os.walk() from approximately 2N to N, where N is the total number of files and directories in the tree. (And because directory trees are usually wider than they are deep, it’s often much better than this.) In practice, removing all those extra system calls makes os.walk() about 8-9 times as fast on Windows , and about 2-3 times as fast on POSIX systems .
So we’re not talking about micro-optimizations. See more benchmarks here . Somewhat relatedly, many people (see Python Issue 11406 ) are also keen on a version of os.listdir() that yields filenames as it iterates instead of returning them as one big list. This improves memory efficiency for iterating very large directories. So, as well as providing a scandir() iterator function for calling directly, Python’s existing os.walk() function can be sped up a huge amount.
Implementation The implementation of this proposal was written by Ben Hoyt (initial version) and Tim Golden (who helped a lot with the C extension module). It lives on GitHub at benhoyt/scandir . (The implementation may lag behind the updates to this PEP a little.) Note that this module has been used and tested (see “Use in the wild” section in this PEP), so it’s more than a proof-of-concept. However, it is marked as beta software and is not extensively battle-tested.
It will need some cleanup and more thorough testing before going into the standard library, as well as integration into posixmodule.c . Specifics of proposal os.scandir() Specifically, this PEP proposes adding a single function to the os module in the standard library, scandir , that takes a single, optional string as its argument: scandir ( path = '.' ) -> generator of DirEntry objects Like listdir , scandir calls the operating system’s directory iteration system calls to get the names of the files in the given path , but it’s different from listdir in two ways: Instead of returning bare filename strings, it returns lightweight DirEntry objects that hold the filename string and provide simple methods that allow access to the additional data the operating system may have returned.
It returns a generator instead of a list, so that scandir acts as a true iterator instead of returning the full list immediately. scandir() yields a DirEntry object for each file and sub-directory in path . Just like listdir , the '.' and '..' pseudo-directories are skipped, and the entries are yielded in system-dependent order. Each DirEntry object has the following attributes and methods: name : the entry’s filename, relative to the scandir path argument (corresponds to the return values of os.listdir ) path : the entry’s full path name (not necessarily an absolute path) – the equivalent of os.path.join(scandir_path, entry.name) inode() : return the inode number of the entry.
The result is cached on the DirEntry object, use os.stat(entry.path, follow_symlinks=False).st_ino to fetch up-to-date information. On Unix, no system call is required. is_dir(*, follow_symlinks=True) : similar to pathlib.Path.is_dir() , but the return value is cached on the DirEntry object; doesn’t require a system call in most cases; don’t follow symbolic links if follow_symlinks is False is_file(*, follow_symlinks=True) : similar to pathlib.Path.is_file() , but the return value is cached on the DirEntry object; doesn’t require a system call in most cases; don’t follow symbolic links if follow_symlinks is False is_symlink() : similar to pathlib.Path.is_symlink() , but the return value is cached on the DirEntry object; doesn’t require a system call in most cases stat(*, follow_symlinks=True) : like os.stat() , but the return value is cached on the DirEntry object; does not require a system call on Windows (except for symlinks); don’t follow symbolic links (like os.lstat() ) if follow_symlinks is False All methods may perform system calls in some cases and therefore possibly raise OSError – see the “Notes on exception handling” section for more details.
The DirEntry attribute and method names were chosen to be the same as those in the new pathlib module where possible, for consistency. The only difference in functionality is that the DirEntry methods cache their values on the entry object after the first call. Like the other functions in the os module, scandir() accepts either a bytes or str object for the path parameter, and returns the DirEntry.name and DirEntry.path attributes with the same type as path .
However, it is strongly recommended to use the str type, as this ensures cross-platform support for Unicode filenames. (On Windows, bytes filenames have been deprecated since Python 3.3). os.walk() As part of this proposal, os.walk() will also be modified to use scandir() rather than listdir() and os.path.isdir() . This will increase the speed of os.walk() very significantly (as mentioned above, by 2-20 times, depending on the system).
Examples First, a very simple example of scandir() showing use of the DirEntry.name attribute and the DirEntry.is_dir() method: def subdirs ( path ): """Yield directory names not starting with '.' under given path.""" for entry in os . scandir ( path ): if not entry . name . startswith ( '.' ) and entry . is_dir (): yield entry . name This subdirs() function will be significantly faster with scandir than os.listdir() and os.path.isdir() on both Windows and POSIX systems, especially on medium-sized or large directories.
Or, for getting the total size of files in a directory tree, showing use of the DirEntry.stat() method and DirEntry.path attribute: def get_tree_size ( path ): """Return total size of files in given path and subdirs.""" total = 0 for entry in os . scandir ( path ): if entry . is_dir ( follow_symlinks = False ): total += get_tree_size ( entry . path ) else : total += entry . stat ( follow_symlinks = False ) . st_size return total This also shows the use of the follow_symlinks parameter to is_dir() – in a recursive function like this, we probably don
The rest of this page is on the live site.