Adding conditional compilation to Python - Ideas - Discussions on Python.org
Adding conditional compilation to Python Ideas TobiasHT (TobiasHT) February 15, 2022, 7:32am 1 Introduction There has been lots of changes introduced to the python language in a period of a few years. Some of these changes are internal for example switching from the LL(1) parser to the PEG parser, other changes are more public for example the numerous modifications to the builtin objects and many other added feature. My main focus however is the changes made to the python syntax.
There have been a few modifications to the syntax of python, but these modifications are quite huge when it gets to adoption. The most recent syntax change was the addition of the match and case soft keywords which without a doubt have numerous applications. Not to mention the modification of union type hints to be created with | , the addition of the walrus operator in python 3.8 and many other examples. Problem These changes are sources of evolution to the language, which is a good thing, but we increasingly see code which would want to adopt to this new syntax but cannot change as they would introduce syntax errors for other teams relying on their code and yet running lower versions of python.
Great examples are big projects like django and the like, which still use the minimum supported python syntax. And if they’d want to try out something new provided by a new version of python, they’d have to drop support for an older version. Currently Django supports 3.8, 3.9 and 3.10, but still cannot use 3.10 specific features as 3.8 is the minimum version supported, and in doing so, it would introduce syntax errors. solution So in accordance to the problem above.
I was suggesting the introduction of conditional compilation in python. Conditional compilation in python is not new as stated by Andre in his idea for Python 101 here . My suggestion was to be able to extend the existing machinery for conditional compilation to support this feature. How it would work So I was suggesting that a compiler feature be added to the __future__ module. Once someone imports compiler at the top level of the module, then the python compiler would be prepped to be ready for conditional compilation.
Then for the actual conditions. I was suggesting we extend the existing ability of using comments to switch the default encoding of python to support # -*- compileif:version -*- and # -*- endcompileif -*- . With this, someone can be able to activate conditional compilation for a specific section of python code in their module. example usage from __future__ import compiler def check (val): # -*- compileif:3.10 -*- match val: case 1: print("got one") case _: print(" got no one") # -*- compileif:3.8.7 -*- if val == 1: print("got one") else: print("got no one") # -*- endcompileif -*- so that’s how the code would kinda look.
The compilerif comment could also support comparative operations such as # -*- compileif: <= 3.9 -*- . The other idea I thought about was to be able to extend the compile builtin function to support conditional compilation. I’ve not thought out much how this would compile sections of code as yet, though I’m sure it’s possible. pros enables quick adoption of new python features allows for the framework, library or module to evolve along with the python language cons The new workflow might be hard or confusing to adopt I don’t think it can be easily added to more stable versions of python.
So I’m not sure if this is something worth undertaking, I invite your criticisms for this idea. steven.daprano (Steven D'Aprano) February 15, 2022, 9:44am 2 An interesting idea. During the transition between Python 2 and 3, most of the code I wrote had to support versions 2.4 through to 3.3 and beyond. There were quite a few syntactic changes between them. Simple things like except ValueError as err: don’t work in Python 2.4.
I mostly was able to work around the differences in functionality pretty easily. Differences in syntax were harder to deal with. I played around with conditional compilation using compile and exec: try: code = compile("""block of code""", "", "exec") except SyntaxError: code = compile("""alternative block of code""", "", "exec") exec(code) but generally speaking I found it unsatisfactory. It worked okay if the code block was very small, but otherwise it wasn’t ideal.
It added a lot of complexity and complication to my code, and made it difficult or impossible to test the implementations separately. The trick I found was to factor my code so that the syntax differences were containing in the smallest possible unit (say, a single function), then write two versions of that function, and use importing to choose between them: try: from timerlib import Timer except SyntaxError: from timerlib24 import Timer obj = Timer(*args) That worked pretty well up to the point that I know long needed to care about 2.4 and 2.5 and could abandon the practice Being in separate file, I could test both versions of the function: python2.4 timerlib24.py python3.3 timerlib.py (unit test that specific function) independently, which was great for finding bugs, as well as the full app: python2.4 myapp.py python3.3 myapp.py (sort of like integration testing).
So based on my experience, I would not use conditional compilation. Some more points. Since from __future__ import compiler doesn’t exist now and cannot be back-ported to 3.8, your examples are impossible. If we add this feature to 3.11, no library can use it until it has abandoned support for versions below 3.11. So in practical terms that probably means that if we add it now, people who need it won’t be able to use it until 3.14 or 3.15 or so.
Using comments for code is an anti-pattern (citation required). We can get away with it for the coding cookie because the interpreter cannot even begin to parse the source text until it knows what encoding to use, so the encoding line cannot be anything that is read at runtime. It has to be something that is skipped at runtime, hence a comment. But we should not emulate that with new features that occur at runtime. There are languages with conditional compilation (although not usually for syntactic features, more like platform features).
You should survey those languages to see whether conditional compilation is considered a good feature or a source of excessive complexity. My instinct is to think that even if we wanted this, it would be too hard to implement. The from __future__ import feature is more of a compiler instruction than a real import, and it must come first , before any executable code. It is too hard for the compiler to jump back and forth between “this code can be compiled” and “this code cannot be compiled, but don’t raise an error, just skip it”.
Not quite impossible though: we can do something similar in Python today, with the compile function, which can be told to ignore or use future imports. So maybe this is plausible. But I still wouldn’t use it I would refactor out the code into separate modules and import the required one. 5 Likes gforcada (Gil Forcada Codinachs) February 15, 2022, 5:36pm 3 Taken into account what @steven.daprano already mentioned, syntax wise, a context manager would be a natural fit IMHO, or rather an if/else syntax: with python-39: my-special-python 3.9 code with python-38: my other special code if python-version > 3.5: do something else: do something else Though the problem is backporting this to older versions barry (Barry Warsaw) February 16, 2022, 1:50am 4 A tried and true technique when porting/writing straddling code for Python 2 and 3.
EpicWink (Laurie O) February 16, 2022, 8:10am 5 Related: PEP 638: Syntactic Macros . While also only useful a few versions after implementation, it could allow for 2to3 -style libraries allowing for sytactic backward compatibility 1 Like sinoroc (sinoroc) February 16, 2022, 4:40pm 6 This idea here reminded me of those projects: ideas · PyPI syntactic · PyPI macropy3 · PyPI inlinec · PyPI Exploit
The rest of this page is on the live site.