PEP 529 – Change Windows filesystem encoding to UTF-8 | peps.python.org
PEP 529 – Change Windows filesystem encoding to UTF-8 PEP 529 – Change Windows filesystem encoding to UTF-8 Author : Steve Dower <steve.dower at python.org> Status : Final Type : Standards Track Created : 27-Aug-2016 Python-Version : 3.6 Post-History : 01-Sep-2016, 04-Sep-2016 Resolution : Python-Dev message Table of Contents Abstract Background Proposal Specific Changes Update sys.getfilesystemencoding Add sys.getfilesystemencodeerrors Update path_converter Remove unused ANSI code Add legacy mode Undeprecate bytes paths on Windows Beta experiment Affected Modules Rejected Alternatives Use strict mbcs decoding Make bytes paths an error on Windows Make bytes paths an error on all platforms Code that may break Not managing encodings across boundaries Explicitly using ‘mbcs’ Copyright Abstract Historically, Python uses the ANSI APIs for interacting with the Windows operating system, often via C Runtime functions.
However, these have been long discouraged in favor of the UTF-16 APIs. Within the operating system, all text is represented as UTF-16, and the ANSI APIs perform encoding and decoding using the active code page. See Naming Files, Paths, and Namespaces for more details. This PEP proposes changing the default filesystem encoding on Windows to utf-8, and changing all filesystem functions to use the Unicode APIs for filesystem paths.
This will not affect code that uses strings to represent paths, however those that use bytes for paths will now be able to correctly round-trip all valid paths in Windows filesystems. Currently, the conversions between Unicode (in the OS) and bytes (in Python) were lossy and would fail to round-trip characters outside of the user’s active code page. Notably, this does not impact the encoding of the contents of files.
These will continue to default to locale.getpreferredencoding() (for text files) or plain bytes (for binary files). This only affects the encoding used when users pass a bytes object to Python where it is then passed to the operating system as a path name. Background File system paths are almost universally represented as text with an encoding determined by the file system. In Python, we expose these paths via a number of interfaces, such as the os and io modules.
Paths may be passed either direction across these interfaces, that is, from the filesystem to the application (for example, os.listdir() ), or from the application to the filesystem (for example, os.unlink() ). When paths are passed between the filesystem and the application, they are either passed through as a bytes blob or converted to/from str using os.fsencode() and os.fsdecode() or explicit encoding using sys.getfilesystemencoding() .
The result of encoding a string with sys.getfilesystemencoding() is a blob of bytes in the native format for the default file system. On Windows, the native format for the filesystem is utf-16-le. The recommended platform APIs for accessing the filesystem all accept and return text encoded in this format. However, prior to Windows NT (and possibly further back), the native format was a configurable machine option and a separate set of APIs existed to accept this format.
The option (the “active code page”) and these APIs (the “*A functions”) still exist in recent versions of Windows for backwards compatibility, though new functionality often only has a utf-16-le API (the “*W functions”). In Python, str is recommended because it can correctly round-trip all characters used in paths (on POSIX with surrogateescape handling; on Windows because str maps to the native representation).
On Windows bytes cannot round-trip all characters used in paths, as Python internally uses the *A functions and hence the encoding is “whatever the active code page is”. Since the active code page cannot represent all Unicode characters, the conversion of a path into bytes can lose information without warning or any available indication. As a demonstration of this: >>> open ( 'test \uAB00 .txt' , 'wb' ) . close () >>> import glob >>> glob . glob ( 'test*' ) ['test\uab00.txt'] >>> glob . glob ( b 'test*' ) [b'test?.txt'] The Unicode character in the second call to glob has been replaced by a ‘?’, which means passing the path back into the filesystem will result in a FileNotFoundError .
The same results may be observed with os.listdir() or any function that matches the return type to the parameter type. While one user-accessible fix is to use str everywhere, POSIX systems generally do not suffer from data loss when using bytes exclusively as the bytes are the canonical representation. Even if the encoding is “incorrect” by some standard, the file system will still map the bytes back to the file. Making use of this avoids the cost of decoding and reencoding, such that (theoretically, and only on POSIX), code such as this may be faster because of the use of b'.' compared to using '.' : >>> for f in os . listdir ( b '.' ): ... os . stat ( f ) ...
As a result, POSIX-focused library authors prefer to use bytes to represent paths. For some authors it is also a convenience, as their code may receive bytes already known to be encoded correctly, while others are attempting to simplify porting their code from Python 2. However, the correctness assumptions do not carry over to Windows where Unicode is the canonical representation, and errors may result. This potential data loss is why the use of bytes paths on Windows was deprecated in Python 3.3 - all of the above code snippets produce deprecation warnings on Windows.
Proposal Currently the default filesystem encoding is ‘mbcs’, which is a meta-encoder that uses the active code page. However, when bytes are passed to the filesystem they go through the *A APIs and the operating system handles encoding. In this case, paths are always encoded using the equivalent of ‘mbcs:replace’ with no opportunity for Python to override or change this. This proposal would remove all use of the *A APIs and only ever call the *W APIs.
When Windows returns paths to Python as str , they will be decoded from utf-16-le and returned as text (in whatever the minimal representation is). When Python code requests paths as bytes , the paths will be transcoded from utf-16-le into utf-8 using surrogatepass (Windows does not validate surrogate pairs, so it is possible to have invalid surrogates in filenames). Equally, when paths are provided as bytes , they are transcoded from utf-8 into utf-16-le and passed to the *W APIs.
The use of utf-8 will not be configurable, except for the provision of a “legacy mode” flag to revert to the previous behaviour. The surrogateescape error mode does not apply here, as the concern is not about retaining nonsensical bytes. Any path returned from the operating system will be valid Unicode, while invalid paths created by the user should raise a decoding error (currently these would raise OSError or a subclass).
The choice of utf-8 bytes (as opposed to utf-16-le bytes) is to ensure the ability to round-trip path names and allow basic manipulation (for example, using the os.path module) when assuming an ASCII-compatible encoding. Using utf-16-le as the encoding is more pure, but will cause more issues than are resolved. This change would also undeprecate the use of bytes paths on Windows. No change to the semantics of using bytes as a path is required - as before, they must be encoded with the encoding specified by sys.getfilesystemencoding() .
Specific Changes Update sys.getfilesystemencoding Remove the default value for Py_FileSystemDefaultEncoding and set it in initfsencoding() to utf-8, or if the legacy-mode switch is enabled to mbcs. Update the implementations of PyUnicode_DecodeFSDefaultAndSize() and PyUnicode_EncodeFSDefault() to use the utf-8 codec, or if the legacy-mode switch is enabled the existing mbcs codec. Add sys.getfilesystemencodeerrors As the error mode may now change between surrogatepass and replace , Python code that manually perfo
The rest of this page is on the live site.