PEP 100 – Python Unicode Integration | peps.python.org
PEP 100 – Python Unicode Integration PEP 100 – Python Unicode Integration Author : Marc-André Lemburg <mal at lemburg.com> Status : Final Type : Standards Track Created : 10-Mar-2000 Python-Version : 2.0 Post-History : Table of Contents Historical Note Introduction Conventions General Remarks Unicode Default Encoding Unicode Constructors Unicode Type Object Unicode Output Unicode Ordinals Comparison & Hash Value Coercion Exceptions Codecs (Coder/Decoders) Lookup Standard Codecs Codecs Interface Definition Whitespace Case Conversion Line Breaks Unicode Character Properties Private Code Point Areas Internal Format Buffer Interface Pickle/Marshalling Regular Expressions Formatting Markers Internal Argument Parsing File/Stream Output File/Stream Input Unicode Methods & Attributes Code Base Test Cases References History of this Proposal 1.7 1.6 1.5 1.4 1.3 1.2 1.1 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 Historical Note This document was first written by Marc-Andre in the pre-PEP days, and was originally distributed as Misc/unicode.txt in Python distributions up to and included Python 2.1.
The last revision of the proposal in that location was labeled version 1.7 (CVS revision 3.10). Because the document clearly serves the purpose of an informational PEP in the post-PEP era, it has been moved here and reformatted to comply with PEP guidelines. Future revisions will be made to this document, while Misc/unicode.txt will contain a pointer to this PEP. -Barry Warsaw, PEP editor Introduction The idea of this proposal is to add native Unicode 3.0 support to Python in a way that makes use of Unicode strings as simple as possible without introducing too many pitfalls along the way.
Since this goal is not easy to achieve – strings being one of the most fundamental objects in Python – we expect this proposal to undergo some significant refinements. Note that the current version of this proposal is still a bit unsorted due to the many different aspects of the Unicode-Python integration. The latest version of this document is always available at: http://starship.python.net/~lemburg/unicode-proposal.txt Older versions are available as: http://starship.python.net/~lemburg/unicode-proposal-X.X.txt [ed. note: new revisions should be made to this PEP document, while the historical record previous to version 1.7 should be retrieved from MAL’s url, or Misc/unicode.txt] Conventions In examples we use u = Unicode object and s = Python string ‘XXX’ markings indicate points of discussion (PODs) General Remarks Unicode encoding names should be lower case on output and case-insensitive on input (they will be converted to lower case by all APIs taking an encoding name as input).
Encoding names should follow the name conventions as used by the Unicode Consortium: spaces are converted to hyphens, e.g. ‘utf 16’ is written as ‘utf-16’. Codec modules should use the same names, but with hyphens converted to underscores, e.g. utf_8 , utf_16 , iso_8859_1 . Unicode Default Encoding The Unicode implementation has to make some assumption about the encoding of 8-bit strings passed to it for coercion and about the encoding to as default for conversion of Unicode to strings when no specific encoding is given.
This encoding is called <default encoding> throughout this text. For this, the implementation maintains a global which can be set in the site.py Python startup script. Subsequent changes are not possible. The <default encoding> can be set and queried using the two sys module APIs: sys.setdefaultencoding(encoding) Sets the <default encoding> used by the Unicode implementation. encoding has to be an encoding which is supported by the Python installation, otherwise, a LookupError is raised.
Note: This API is only available in site.py! It is removed from the sys module by site.py after usage. sys.getdefaultencoding() Returns the current <default encoding>. If not otherwise defined or set, the <default encoding> defaults to ‘ascii’. This encoding is also the startup default of Python (and in effect before site.py is executed). Note that the default site.py startup module contains disabled optional code which can set the <default encoding> according to the encoding defined by the current locale.
The locale module is used to extract the encoding from the locale default settings defined by the OS environment (see locale.py). If the encoding cannot be determined, is unknown or unsupported, the code defaults to setting the <default encoding> to ‘ascii’. To enable this code, edit the site.py file or place the appropriate code into the sitecustomize.py module of your Python installation. Unicode Constructors Python should provide a built-in constructor for Unicode strings which is available through __builtins__ : u = unicode ( encoded_string [, encoding =< default encoding > ][, errors = "strict" ]) u = u '<unicode-escape encoded Python string>' u = ur '<raw-unicode-escape encoded Python string>' With the ‘unicode-escape’ encoding being defined as: all non-escape characters represent themselves as Unicode ordinal (e.g. ‘a’ -> U+0061). all existing defined Python escape sequences are interpreted as Unicode ordinals; note that \xXXXX can represent all Unicode ordinals, and \OOO (octal) can represent Unicode ordinals up to U+01FF. a new escape sequence, \uXXXX , represents U+XXXX; it is a syntax error to have fewer than 4 digits after \u .
For an explanation of possible values for errors see the Codec section below. Examples: u 'abc' -> U + 0061 U + 0062 U + 0063 u ' \u1234 ' -> U + 1234 u 'abc \u1234\n ' -> U + 0061 U + 0062 U + 0063 U + 1234 U + 005 c The ‘raw-unicode-escape’ encoding is defined as follows: \uXXXX sequence represent the U+XXXX Unicode character if and only if the number of leading backslashes is odd all other characters represent themselves as Unicode ordinal (e.g. ‘b’ -> U+0062) Note that you should provide some hint to the encoding you used to write your programs as pragma line in one the first few comment lines of the source file (e.g. ‘# source file encoding: latin-1’).
If you only use 7-bit ASCII then everything is fine and no such notice is needed, but if you include Latin-1 characters not defined in ASCII, it may well be worthwhile including a hint since people in other countries will want to be able to read your source strings too. Unicode Type Object Unicode objects should have the type UnicodeType with type name ‘unicode’, made available through the standard types module. Unicode Output Unicode objects have a method .encode([encoding=<default encoding>]) which returns a Python string encoding the Unicode string using the given scheme (see Codecs). print u := print u . encode () # using the <default encoding> str ( u ) := u . encode () # using the <default encoding> repr ( u ) := "u %s " % repr ( u . encode ( 'unicode-escape' )) Also see Internal Argument Parsing and Buffer Interface for details on how other APIs written in C will treat Unicode objects.
Unicode Ordinals Since Unicode 3.0 has a 32-bit ordinal character set, the implementation should provide 32-bit aware ordinal conversion APIs: ord ( u [: 1 ]) ( this is the standard ord () extended to work with Unicode objects ) --> Unicode ordinal number ( 32 - bit ) unichr ( i ) --> Unicode object for character i ( provided it is 32 - bit ); ValueError otherwise Both APIs should go into __builtins__ just like their string counterparts ord() and chr() .
Note that Unicode provides space for private encodings. Usage of these can cause different output representations on different machines. This problem is not a Python or Unicode problem, but a machine setup and maintenance one. Comparison & Hash Value Unicode objects should compare equal to other objects after these other objects have been coerced to Unicode. For strings this means that they are interpreted as Unicode string using the <default encoding>.
Unicode objects should return the same hash value as their ASCII equiv
The rest of this page is on the live site.