Issue 8271: str.decode('utf8', 'replace') -- conformance with Unicode 5.2.0 - Python tracker
Python Home About News Documentation Downloads Community Foundation Developer's Guide Issue Tracker Issues Search Random Issue Summaries Issues with patch Easy issues Stats User Administration User List Committer List Help Tracker Documentation Tracker Development Report Tracker Problem Issue8271 ➜ This issue tracker has been migrated to GitHub , and is currently read-only . For more information, see the GitHub FAQs in the Python's Developer Guide.
Created on 2010-03-31 02:28 by sjmachin , last changed 2022-04-11 14:56 by admin . This issue is now closed . Files File name Uploaded Description Edit issue8271.diff ezio.melotti , 2010-04-01 08:33 Incomplete patch against trunk. review issue8271v2.diff ezio.melotti , 2010-04-02 22:27 New patch against trunk review issue8271v3.diff ezio.melotti , 2010-04-04 05:49 Final patch review issue8271v4.diff ezio.melotti , 2010-04-07 04:08 More final patch review issue8271v5.diff ezio.melotti , 2010-06-04 16:22 Even more final patch review issue8271v6.diff ezio.melotti , 2011-04-19 12:30 Patch to fix the number of FFFD review issue8271-3.3-fast-3.patch serhiy.storchaka , 2012-06-23 21:21 Ezio's patch updated to current sources review Messages (66) msg101972 - (view) Author: John Machin (sjmachin) Date: 2010-03-31 02:28 Unicode 5.2.0 chapter 3 (Conformance) has a new section (headed "Constraints on Conversion Processes) after requirement D93.
Recent Pythons e.g. 3.1.2 don't comply. Using the Unicode example: >>> print(ascii(b"\xc2\x41\x42".decode('utf8', 'replace'))) '\ufffdB' # should produce u'\ufffdAB' Resynchronisation currently starts at a position derived by considering the length implied by the start byte: >>> print(ascii(b"\xf1ABCD".decode('utf8', 'replace'))) '\ufffdD' # should produce u'\ufffdABCD'; resync should start from the *failing* byte. Notes: This applies to the 'ignore' option as well as the 'replace' option.
The Unicode discussion mentions "security exploits". msg102013 - (view) Author: Daniel Graña (dangra) Date: 2010-03-31 14:59 Some background for this report at http://stackoverflow.com/questions/2547262/why-is-python-decode-replacing-more-than-the-invalid-bytes-from-an-encoded-string/2548480 msg102024 - (view) Author: Marc-Andre Lemburg (lemburg) * Date: 2010-03-31 18:07 I guess the term "failing" byte somewhat underdefined.
Page 95 of the standard PDF ( http://www.unicode.org/versions/Unicode5.2.0/ch03.pdf ) suggests to "Replace each maximal subpart of an ill-formed subsequence by a single U+FFFD". Fortunately, they explain what they are after: if a subsequent byte in the sequence does not have the high bit set, it's not to be considered part of the UTF-8 sequence of the code point. Implementing that should be fairly straight-forward by adjusting the endinpos variable accordingly.
Any takers ? msg102061 - (view) Author: John Machin (sjmachin) Date: 2010-04-01 03:19 @lemburg: "failing byte" seems rather obvious: first byte that you meet that is not valid in the current state. I don't understand your explanation, especially "does not have the high bit set". I think you mean "is a valid starter byte". See example 3 below. Example 1: F1 80 41 42 43. F1 implies a 4-byte character. 80 is OK. 41 is not in 80-BF.
It is the "failing byte"; high bit not set. Required action is to emit FFFD then resync on the 41, causing 0041 0042 0043 to be emitted. Total output: FFFD 0041 0042 0043. Current code emits FFFD 0043. Example 2: F1 80 FF 42 43. F1 implies a 4-byte character. 80 is OK. FF is not in 80-BF. It is the "failing byte". Required action is to emit FFFD then resync on the FF. FF is not a valid starter byte, so emit FFFD, and resync on the 42, causing 0042 0043 to be emitted.
Total output: FFFD FFFD 0042 0043. Current code emits FFFD 0043. Example 3: F1 80 C2 81 43. F1 implies a 4-byte character. 80 is OK. C2 is not in 80-BF. It is the "failing byte". Required action is to emit FFFD then resync on the C2. C2 and 81 have the high bit set, but C2 is a valid starter byte, and remaining bytes are OK, causing 0081 0043 to be emitted. Total output: FFFD 0081 0043. Current code emits FFFD 0043. msg102062 - (view) Author: Ezio Melotti (ezio.melotti) * Date: 2010-04-01 03:28 Having the 'high bit set' means that the first bit is set to 1.
All the continuation bytes (i.e. the 2nd, 3rd or 4th byte in a sequence) have the first two bits set to 1 and 0 respectively, so if the first bit is not set to 1 then the byte shouldn't be considered part of the sequence. I'm trying to work on a patch. msg102063 - (view) Author: John Machin (sjmachin) Date: 2010-04-01 06:08 @ezio.melotti: Your second sentence is true, but it is not the whole truth. Bytes in the range C0-FF (whose high bit *is* set) ALSO shouldn't be considered part of the sequence because they (like 00-7F) are invalid as continuation bytes; they are either starter bytes (C2-F4) or invalid for any purpose (C0-C2 and F5-FF).
Further, some bytes in the range 80-BF are NOT always valid as the first continuation byte, it depends on what starter byte they follow. The simple way of summarising the above is to say that a byte that is not a valid continuation byte in the current state ("failing byte") is not a part of the current (now known to be invalid) sequence, and the decoder must try again ("resync") with the failing byte. Do you agree with my example 3? msg102064 - (view) Author: Ezio Melotti (ezio.melotti) * Date: 2010-04-01 06:14 Yes, right now I'm considering valid all the bytes that start with '10...'.
C2 starts with '11...' so it's a "failing byte". msg102065 - (view) Author: John Machin (sjmachin) Date: 2010-04-01 07:29 #ezio.melotti: """I'm considering valid all the bytes that start with '10...'""" Sorry, WRONG. Read what I wrote: """Further, some bytes in the range 80-BF are NOT always valid as the first continuation byte, it depends on what starter byte they follow.""" Consider these sequences: (1) E0 80 80 (2) E0 9F 80.
Both are invalid sequences (over-long). Specifically the first continuation byte may not be in 80-9F. Those bytes start with '10...' but they are invalid after an E0 starter byte. Please read "Table 3-7. Well-Formed UTF-8 Byte Sequences" and surrounding text in Unicode 5.2.0 chapter 3 (bearing in mind that CPython (for good reasons) doesn't implement the surrogates restriction, so that the special case for starter byte ED is not used in CPython).
Note the other 3 special cases for the first continuation byte. msg102066 - (view) Author: Ezio Melotti (ezio.melotti) * Date: 2010-04-01 07:33 That's why I'm writing tests that cover all the cases, including overlong sequences. If the test will fail I'll change the patch :) msg102068 - (view) Author: Marc-Andre Lemburg (lemburg) * Date: 2010-04-01 07:44 John Machin wrote: > > John Machin < sjmachin@users.sourceforge.net > added the comment: > > @lemburg: "failing byte" seems rather obvious: first byte that you meet that is not valid in the current state.
I don't understand your explanation, especially "does not have the high bit set". I think you mean "is a valid starter byte". See example 3 below. I just had a quick look at the code and saw that it's testing for the high bit on the subsequent bytes. Looking closer, you're right and the situation is a bit more complex, but the solution still looks simple: only the endinpos has to be adjusted more carefully depending on what the various checks find.
That said, I find the Unicode consortium solution a bit awkward. In UTF-8 the first byte in a multi-byte sequence defines the number of bytes that make up a sequence. If some of those bytes are invalid, the whole sequence is invalid and the fact that some of those bytes may be interpretable as regular code points does not necessarily result in better results - the reason is that loss of bytes in a stream is far more unlikely than flipping a few bits in the data. msg102076 - (view) Author: Ezio Melotti (ezio.melotti) * Date: 2010-04-01 08:33 Here is an incomplete patch.
It seems to solve the p
The rest of this page is on the live site.