Text of peps.python.org › pep-0538 as Styx crawled it on Nov 19, 2026. The page may have changed since. Only its words are kept, without pictures or layout, and only the first part of a long page.

Open the live page

PEP 538 – Coercing the legacy C locale to a UTF-8 based locale | peps.python.org

PEP 538 – Coercing the legacy C locale to a UTF-8 based locale PEP 538 – Coercing the legacy C locale to a UTF-8 based locale Author : Alyssa Coghlan <ncoghlan at gmail.com> BDFL-Delegate : INADA Naoki Status : Final Type : Standards Track Created : 28-Dec-2016 Python-Version : 3.7 Post-History : 03-Jan-2017, 07-Jan-2017, 05-Mar-2017, 09-May-2017 Resolution : Python-Dev message Table of Contents Abstract Implementation Notes Background Relationship with other PEPs Motivation Design Principles Specification Legacy C locale coercion in the standalone Python interpreter binary Legacy C locale warning during runtime initialization New build-time configuration options Changes to the default error handling on the standard streams Changes to locale settings on Android Platform Support Changes Rationale Improving the handling of the C locale Defaulting to “surrogateescape” error handling on the standard IO streams Avoiding setting PYTHONIOENCODING during UTF-8 locale coercion Dropping official support for ASCII based text handling in the legacy C locale Providing implicit locale coercion only when running standalone Allowing restoration of the legacy behaviour Querying LC_CTYPE for C locale detection Explicitly setting LC_CTYPE for UTF-8 locale coercion Avoiding setting LANG for UTF-8 locale coercion Avoiding setting LC_ALL for UTF-8 locale coercion Skipping locale coercion if LC_ALL is set in the current environment Considering locale coercion independently of “UTF-8 mode” Enabling C locale coercion and warnings on Mac OS X, iOS and Android Implementation Backporting to earlier Python 3 releases Backporting to Python 3.6.x Backporting to other 3.x releases Acknowledgements References Copyright Abstract An ongoing challenge with Python 3 on *nix systems is the conflict between needing to use the configured locale encoding by default for consistency with other locale-aware components in the same process or subprocesses, and the fact that the standard C locale (as defined in POSIX:2001) typically implies a default text encoding of ASCII, which is entirely inadequate for the development of networked services and client applications in a multilingual world.

PEP 540 proposes a change to CPython’s handling of the legacy C locale such that CPython will assume the use of UTF-8 in such environments, rather than persisting with the demonstrably problematic assumption of ASCII as an appropriate encoding for communicating with operating system interfaces. This is a good approach for cases where network encoding interoperability is a more important concern than local encoding interoperability.

However, it comes at the cost of making CPython’s encoding assumptions diverge from those of other locale-aware components in the same process, as well as those of components running in subprocesses that share the same environment. This can cause interoperability problems with some extension modules (such as GNU readline’s command line history editing), as well as with components running in subprocesses (such as older Python runtimes).

It also requires non-trivial changes to the internals of how CPython itself works, rather than relying primarily on existing configuration settings that are supported by Python versions prior to Python 3.7. Accordingly, this PEP proposes that independently of the UTF-8 mode proposed in PEP 540 , the way the CPython implementation handles the default C locale be changed to be roughly equivalent to the following existing configuration settings (supported since Python 3.1): LC_CTYPE = C .

UTF - 8 PYTHONIOENCODING = utf - 8 : surrogateescape The exact target locale for coercion will be chosen from a predefined list at runtime based on the actually available locales. The reinterpreted locale settings will be written back to the environment so they’re visible to other components in the same process and in subprocesses, but the changed PYTHONIOENCODING default will be made implicit in order to avoid causing compatibility problems with Python 2 subprocesses that don’t provide the surrogateescape error handler.

The new legacy locale coercion behavior can be disabled either by setting LC_ALL (which may still lead to a Unicode compatibility warning) or by setting the new PYTHONCOERCECLOCALE environment variable to 0 . With this change, any *nix platform that does not offer at least one of the C.UTF-8 , C.utf8 or UTF-8 locales as part of its standard configuration would only be considered a fully supported platform for CPython 3.7+ deployments when a suitable locale other than the default C locale is configured explicitly (e.g. en_AU.UTF-8 , zh_CN.gb18030 ).

If PEP 540 is accepted in addition to this PEP, then pure Python modules would also be supported when using the proposed PYTHONUTF8 mode, but expectations for full Unicode compatibility in extension modules would continue to be limited to the platforms covered by this PEP. As it only reflects a change in default settings rather than a fundamentally new capability, redistributors (such as Linux distributions) with a narrower target audience than the upstream CPython development team may also choose to opt in to this locale coercion behaviour for the Python 3.6.x series by applying the necessary changes as a downstream patch.

Implementation Notes Attempting to implement the PEP as originally accepted showed that the proposal to emit locale coercion and compatibility warnings by default simply wasn’t practical (there were too many cases where previously working code failed because of the warnings , rather than because of latent locale handling defects in the affected code). As a result, the PY_WARN_ON_C_LOCALE config flag was removed, and replaced with a runtime PYTHONCOERCECLOCALE=warn environment variable setting that allows developers and system integrators to opt-in to receiving locale coercion and compatibility warnings, without emitting them by default.

The output examples in the PEP itself have also been updated to remove the warnings and make them easier to read. Background While the CPython interpreter is starting up, it may need to convert from the char * format to the wchar_t * format, or from one of those formats to PyUnicodeObject * , in a way that’s consistent with the locale settings of the overall system. It handles these cases by relying on the operating system to do the conversion and then ensuring that the text encoding name reported by sys.getfilesystemencoding() matches the encoding used during this early bootstrapping process.

On Windows, the limitations of the mbcs format used by default in these conversions proved sufficiently problematic that PEP 528 and PEP 529 were implemented to bypass the operating system supplied interfaces for binary data handling and force the use of UTF-8 instead. On Mac OS X, iOS, and Android, many components, including CPython, already assume the use of UTF-8 as the system encoding, regardless of the locale setting.

However, this isn’t the case for all components, and the discrepancy can cause problems in some situations (for example, when using the GNU readline module [ 16 ]). On non-Apple and non-Android *nix systems, these operations are handled using the C locale system in glibc, which has the following characteristics [4] : by default, all processes start in the C locale, which uses ASCII for these conversions. This is almost never what anyone doing multilingual text processing actually wants (including CPython and C/C++ GUI frameworks). calling setlocale(LC_ALL, "") reconfigures the active locale based on the locale categories configured in the current process environment if the locale requested by the current environment is unknown, or no specific locale is configured, then the default C locale will remain active The specific locale category that covers the APIs that CPython depends on is LC_CTYPE , which applies to “classification and conversion of characters, and to multibyte and wide characters” [5] .

Accordingly, CP

The rest of this page is on the live site.