PEP 3138 – String representation in Python 3000 | peps.python.org
PEP 3138 – String representation in Python 3000 PEP 3138 – String representation in Python 3000 Author : Atsuo Ishimoto <ishimoto at gembook.org> Status : Final Type : Standards Track Created : 05-May-2008 Python-Version : 3.0 Post-History : 05-May-2008, 05-Jun-2008 Table of Contents Abstract Motivation Specification Rationale Alternate Solutions Backwards Compatibility Rejected Proposals Implementation References Copyright Abstract This PEP proposes a new string representation form for Python 3000.
In Python prior to Python 3000, the repr() built-in function converted arbitrary objects to printable ASCII strings for debugging and logging. For Python 3000, a wider range of characters, based on the Unicode standard, should be considered ‘printable’. Motivation The current repr() converts 8-bit strings to ASCII using following algorithm. Convert CR, LF, TAB and ‘\’ to ‘\r’, ‘\n’, ‘\t’, ‘\\’.
Convert other non-printable characters(0x00-0x1f, 0x7f) and non-ASCII characters (>= 0x80) to ‘\xXX’. Backslash-escape quote characters (apostrophe, ‘) and add the quote character at the beginning and the end. For Unicode strings, the following additional conversions are done. Convert leading surrogate pair characters without trailing character (0xd800-0xdbff, but not followed by 0xdc00-0xdfff) to ‘\uXXXX’. Convert 16-bit characters (>= 0x100) to ‘\uXXXX’.
Convert 21-bit characters (>= 0x10000) and surrogate pair characters to ‘\U00xxxxxx’. This algorithm converts any string to printable ASCII, and repr() is used as a handy and safe way to print strings for debugging or for logging. Although all non-ASCII characters are escaped, this does not matter when most of the string’s characters are ASCII. But for other languages, such as Japanese where most characters in a string are not ASCII, this is very inconvenient.
We can use print(aJapaneseString) to get a readable string, but we don’t have a similar workaround for printing strings from collections such as lists or tuples. print(listOfJapaneseStrings) uses repr() to build the string to be printed, so the resulting strings are always hex-escaped. Or when open(japaneseFilename) raises an exception, the error message is something like IOError: [Errno 2] No such file or directory: '\u65e5\u672c\u8a9e' , which isn’t helpful.
Python 3000 has a lot of nice features for non-Latin users such as non-ASCII identifiers, so it would be helpful if Python could also progress in a similar way for printable output. Some users might be concerned that such output will mess up their console if they print binary data like images. But this is unlikely to happen in practice because bytes and strings are different types in Python 3000, so printing an image to the console won’t mess it up.
This issue was once discussed by Hye-Shik Chang [1] , but was rejected. Specification Add a new function to the Python C API int Py_UNICODE_ISPRINTABLE (Py_UNICODE ch) . This function returns 0 if repr() should escape the Unicode character ch ; otherwise it returns 1. Characters that should be escaped are defined in the Unicode character database as: Cc (Other, Control) Cf (Other, Format) Cs (Other, Surrogate) Co (Other, Private Use) Cn (Other, Not Assigned) Zl (Separator, Line), refers to LINE SEPARATOR (’\u2028’).
Zp (Separator, Paragraph), refers to PARAGRAPH SEPARATOR (’\u2029’). Zs (Separator, Space) other than ASCII space (’\x20’). Characters in this category should be escaped to avoid ambiguity. The algorithm to build repr() strings should be changed to: Convert CR, LF, TAB and ‘\’ to ‘\r’, ‘\n’, ‘\t’, ‘\\’. Convert non-printable ASCII characters (0x00-0x1f, 0x7f) to ‘\xXX’. Convert leading surrogate pair characters without trailing character (0xd800-0xdbff, but not followed by 0xdc00-0xdfff) to ‘\uXXXX’.
Convert non-printable characters (Py_UNICODE_ISPRINTABLE() returns 0) to ‘\xXX’, ‘\uXXXX’ or ‘\U00xxxxxx’. Backslash-escape quote characters (apostrophe, 0x27) and add a quote character at the beginning and the end. Set the Unicode error-handler for sys.stderr to ‘backslashreplace’ by default. Add a new function to the Python C API PyObject *PyObject_ASCII (PyObject *o) . This function converts any python object to a string using PyObject_Repr() and then hex-escapes all non-ASCII characters.
PyObject_ASCII() generates the same string as PyObject_Repr() in Python 2. Add a new built-in function, ascii() . This function converts any python object to a string using repr() and then hex-escapes all non-ASCII characters. ascii() generates the same string as repr() in Python 2. Add a '%a' string format operator. '%a' converts any python object to a string using repr() and then hex-escapes all non-ASCII characters.
The '%a' format operator generates the same string as '%r' in Python 2. Also, add '!a' conversion flags to the string.format() method and add '%A' operator to the PyUnicode_FromFormat(). They convert any object to an ASCII string as '%a' string format operator. Add an isprintable() method to the string type. str.isprintable() returns False if repr() would escape any character in the string; otherwise returns True. The isprintable() method calls the Py_UNICODE_ISPRINTABLE() function internally.
Rationale The repr() in Python 3000 should be Unicode, not ASCII based, just like Python 3000 strings. Also, conversion should not be affected by the locale setting, because the locale is not necessarily the same as the output device’s locale. For example, it is common for a daemon process to be invoked in an ASCII setting, but writes UTF-8 to its log files. Also, web applications might want to report the error information in more readable form based on the HTML page’s encoding.
Characters not supported by the user’s console could be hex-escaped on printing, by the Unicode encoder’s error-handler. If the error-handler of the output file is ‘backslashreplace’, such characters are hex-escaped without raising UnicodeEncodeError. For example, if the default encoding is ASCII, print('Hello ¢') will print ‘Hello \xa2’. If the encoding is ISO-8859-1, ‘Hello ¢’ will be printed. The default error-handler for sys.stdout is ‘strict’.
Other applications reading the output might not understand hex-escaped characters, so unsupported characters should be trapped when writing. If unsupported characters must be escaped, the error-handler should be changed explicitly. Unlike sys.stdout, sys.stderr doesn’t raise UnicodeEncodingError by default, because the default error-handler is ‘backslashreplace’. So printing error messages containing non-ASCII characters to sys.stderr will not raise an exception.
Also, information about uncaught exceptions (exception object, traceback) is printed by the interpreter without raising exceptions. Alternate Solutions To help debugging in non-Latin languages without changing repr(), other suggestions were made. Supply a tool to print lists or dicts. Strings to be printed for debugging are not only contained by lists or dicts, but also in many other types of object. File objects contain a file name in Unicode, exception objects contain a message in Unicode, etc.
These strings should be printed in readable form when repr()ed. It is unlikely to be possible to implement a tool to print all possible object types. Use sys.displayhook and sys.excepthook. For interactive sessions, we can write hooks to restore hex escaped characters to the original characters. But these hooks are called only when printing the result of evaluating an expression entered in an interactive Python session, and don’t work for the print() function, for non-interactive sessions or for logging.debug("%r", ...) , etc.
Subclass sys.stdout and sys.stderr. It is difficult to implement a subclass to restore hex-escaped characters since there isn’t enough information left by the time it’s a string to undo the escaping correctly in all c
The rest of this page is on the live site.