Bug in tokenizer # comments - Python Help - Discussions on Python.org
Bug in tokenizer # comments Python Help NickC (Nick) November 18, 2026, 1:57pm 1 Have found a small bug in _tokenize.TokenizerIter with comments getting indented & dedented incorrectly. This may seek slightly irrelevant but has an effect on the ‘compiler’ pre-processor I am working on using tokenize/untokenize. import tokenize as t from io import BytesIO source_code = """ if True: # comment one <- gets incorrectly dedented, indent doesn't happen until after this line x = 1 # comment two -> gets incorrectly indented, dedent doesn't happen until after this line x = 2 # comment three """ print(f"{source_code=}") modi_toks = [] toks = t.tokenize(BytesIO(source_code.encode()).readline) for tok in toks: print(f"{tok=}") modi_toks.append((tok.type, tok.string)) modi_source = t.untokenize(modi_toks) print(modi_source.decode()) Rosuav (Chris Angelico) November 18, 2026, 2:05pm 2 There are a lot of (usually insignificant) differences in the source code.
Compare your initial and final, and you’ll see quite a few things (I notice a space in “if True :”, no space in “x =1”, and that’s just what first catches the eye). The resultant code will have the same effect as the original, but it may look different. (BTW, if anyone else tries to run this, add from io import BytesIO at the top.) Is the unindenting of comments going to cause you problems? If so, I would recommend using something other than tokenize ; you may want a simple line-by-line processor instead.
NickC (Nick) November 18, 2026, 2:25pm 3 @Rosuav sorry about missing out the BytesIO import, original post now fixed. I started out trying to use ast but hit problems of going line-by-line with an if as it was a single line and needed a following indented line to avoid an error. Then there was the problem of multi-line statements which was a separate can-of-worms to contend with. I was hoping that perhaps someone might be able to see the bug and fix this #comments issue in CPython’s tokenizer.
Rosuav (Chris Angelico) November 18, 2026, 3:16pm 5 When I said “line by line”, I really meant line by line. Not through the AST. Lines of source code. It’s not a bug though. The relevant tokens are being emitted. Those tokens are sufficient to compile the code, but comments don’t care about indentation. Why do you need the token stream? What kinds of tokens are you looking at? There’s a reason that line-by-line preprocessors (knowing nothing about the syntax) are by far the most common.
It won’t make a difference to the compiled code. Note the guarantee in the docs you’re quoting: “”" The result is guaranteed to tokenize back to match the input so that the conversion is lossless and round-trips are assured. The guarantee applies only to the token type and token string as the spacing between tokens (column positions) may change. “”" The guarantee does not apply to spacing between tokens, but it does apply to things that will be code-relevant. mfile_bay ( mfile_bay) November 18, 2026, 3:19pm 6 Sorry, I think I missed that and misinterpreted “Any additional sequence elements are ignored”.
I think I’ll delete my post. I think I’ll add back the method that untokenizes losslessly: pass back tokens unchanged ( online runner ) inventshah (Sachin S) November 18, 2026, 3:29pm 7 You may want tokenize-rt . The stdlib tokenize module does not properly roundtrip. This wrapper around the stdlib provides two additional tokens ESCAPED_NL and UNIMPORTANT_WS , and a Token data type. Use src_to_tokens and tokens_to_src to roundtrip.
This library is useful if you’re writing a refactoring tool based on the python tokenization. NickC (Nick) November 18, 2026, 3:31pm 8 @mfile_bay That’s a shame I was just reading that. @Rosuav but comments don’t care about indentation. . Yep but just because the docs seem to suggest this ‘incorrect’ behaviour is allowed wouldn’t it be better if the indentation of comments was actually corrected in the first place?
It is only the order the indent / dedent tokens are getting written after a #comments line which cause this, surely this can’t be too difficult to fix can it. I am using these comments as part of the compiler directives so in this case the faulty indentation may actually be important. aroberge (André Roberge) November 18, 2026, 3:46pm 9 @inventshah mentions using tokenize-rt. Another option is to use token-utils which also guarantees perfect round-trip.
Adapting your code to use token-utils: from token_utils import tokenize, untokenize source_code = """ if True: # comment one <- gets incorrectly dedented, indent doesn't happen until after this line x = 1 # comment two -> gets incorrectly indented, dedent doesn't happen until after this line x = 2 # comment three """ modi_toks = [] toks = tokenize(source_code) for tok in toks: modi_toks.append(tok) modi_source = untokenize(modi_toks) assert modi_source == source_code In addition, since you want to modify the source code, tokens from token-utils are mutable. jrivers (J.
Rivers) November 18, 2026, 3:47pm 10 A line with a comment by itself does not actually need to be correctly indented, so the tokenizer is right to not emit INDENT / DEDENT tokens there. For example: if True: # should the INDENT still come before the comment here? pass if True: # where should the INDENT be emitted in this case? pass It’s fine for the comment to go anywhere on the line, so these examples are both correct. mfile_bay ( mfile_bay) November 18, 2026, 3:56pm 11 I deleted my post because the post is based on a wrong understanding except for the working method and I think it would be distraction.
I can send it to you privately if you want NickC (Nick) November 18, 2026, 4:32pm 12 Just tried that but fails with AttributeError: 'tuple' object has no attribute 'src' import tokenize_rt as t from io import BytesIO source_code = """ if True: # comment one <- gets incorrectly dedented, indent doesn't happen until after this line x = 1 # comment two -> gets incorrectly indented, dedent doesn't happen until after this line x = 2 # comment three """ print(f"{source_code=}") modi_toks = [] # toks = t.tokenize(BytesIO(source_code.encode()).readline) toks = t.src_to_tokens(source_code) for tok in toks: modi_toks.append((tok.name, tok.src)) modi_source = t.tokens_to_src(modi_toks) print(modi_source NickC (Nick) November 18, 2026, 4:53pm 13 @aroberge tested that but when creating the new tokens using (tok.type, tok.string) it fails with File “D:\Dev\Python\ImportHook.venv\Lib\site-packages\token_utils\tokenizing.py”, line 270, in _untokenize assert isinstance(tokens[0], (Token, str)) ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError from token_utils import tokenize, untokenize source_code = """ if True: # comment one <- gets incorrectly dedented, indent doesn't happen until after this line x = 1 # comment two -> gets incorrectly indented, dedent doesn't happen until after this line x = 2 # comment three """ modi_toks = [] toks = tokenize(source_code) for tok in toks: # modi_toks.append(tok) modi_toks.append((tok.type, tok.string)) modi_source = untokenize(modi_toks) inventshah (Sachin S) November 18, 2026, 4:56pm 14 Based on the usage guide tokens_to_src takes an iterable of Token objects not of tuples.
Did you try modi_toks.append(tok) instead? You can modify the token objects too. tokenize-rt is specifically not a direct API compatible drop-in for tokenize . NickC (Nick) November 18, 2026, 5:00pm 15 @jrivers When it is just a plain comment it obviously doesn’t matter, although it would still be nice to see it correct. However as I am using comments as part of the pre-process compiler directive their correct indentation is important. aroberge (André Roberge) November 18, 2026, 5:09pm 16 Tokens from token-utils are NOT two-tuples and cannot be replaced by two-tuples.
The documentation , and in particular the “recipes” , should have all that you need to find out how to modify a token stream. In particular, if you sim
The rest of this page is on the live site.