About speed.python.org and aarch64 - Core Development - Discussions on Python.org
About speed.python.org and aarch64 Core Development diegor (Diego Russo) January 6, 2023, 3:47pm 1 Hello all, Diego from Arm Ltd here. I’m assessing the status of CPython and its ecosystem on aarch64 and during my investigation I bumped into speed.python.org and pyperformance package. I really like integration between pyperformance and codespeed and I was able to setup an internal instance within Arm with aarch64 results.
Being Arm architecture present on all mayor cloud providers and Python a critical programming language for cloud workflows, it would be great if we can integrate aarch64 performance results on speed.python.org . Pointless to say that we are more than happy to help out with both development and by providing infrastructure (in some form) to achieve this goal. Cheers, Diego 7 Likes guido (Guido van Rossum) January 6, 2023, 6:12pm 2 Hi Diego, Happy New Year!
I had in the back of my mind to write you about how ARM could help Python, and I got overwhelmed with other things on my TODO list. It’s great that you managed to set up an ARM instance of speed.python.org . Is there a chance you can make that public? It doesn’t have to be on a *.python.org URL for now. I’d like to see if we can learn something by comparing the ARM results to the x64 results. I’d also like to point you to Mike Droettboom’s efforts, which you can follow here: GitHub - faster-cpython/benchmarking-public: A public mirror of our benchmarking runner repository (and also to some extent in GitHub - faster-cpython/ideas ).
2 Likes diegor (Diego Russo) January 9, 2023, 3:11pm 3 Guido, happy new year too! Apart from the performance point of view, is there anything else you have in mind we could help with? I just want to gather feedback on this. I will try my best to make it happen. What’s the rationale behind having a separate instance instead of integrating directly to speed.python.org ? That’s useful indeed, thanks for the pointers! guido (Guido van Rossum) January 9, 2023, 10:48pm 4 Mostly just that speed.python.org has exactly one maintainer and he’s not got a lot of time for it, so anything that would require coordinating with him would just be extra red tape that I’d like to save you from.
1 Like diegor (Diego Russo) January 12, 2023, 3:53pm 5 Guido, understood, thanks for explaining. I see what I can do and update here as soon as I have news (it might take some time) 1 Like diegor (Diego Russo) March 31, 2023, 9:29am 6 Just a quick update. Things are progressing well but before making anything public, we’ve been running some internal CI infrastructure to cross validate some data. I will be off for the next three weeks but as soon as I come back, I’ll continue working on this.
3 Likes diegor (Diego Russo) September 18, 2023, 3:15pm 7 Hello everyone, here I am with some exciting news First of all I want to apologise the delay: I’ve been busy with work, EuroPython 2023 (organising it, sponsor it and presenting ) and I took annual/parental leaves on the way. So the whole thing took more than expected. I’m happy to announce that I was able to replicate the benchmark infrastructure that shows CPython performance across Arm CPUs: Neoverse-N1, Neoverse-N2, and Neoverse-V1.
The full address is https://speed.python.arm.com/ . We are tracking the following branches: 3.11 3.12 main Older branches should be more stable in terms of metrics so we preferred to focus on newer ones. If you think we need to include older, let us know and we will do it. We’ve had this infra in place for more than a month now, (since August 15th) because we needed to solve a few of issues with the infrastructure. Until September 14th commits benchmarked on machines might be different depending on the machine: this because the machines were available at different times (with a few hours difference) hence picked the latest commit of the branch at the time of execution of the job.
Since September 14th you should see more consistent data: we first pick the commit and then we schedule the job on different machines with this specific commit. Data might be uploaded at different times but when it arrives it is consistent with other machines. The web server and the database are running on an aarch64 machine sponsored by the Works on Arm project and the benchmarks are running on physical machines behind the Arm firewalls.
Jobs are managed by our own Jenkins instance. I think that’s it for now, have a look and please share your thoughts. 11 Likes guido (Guido van Rossum) September 27, 2023, 10:39pm 8 Hi Diego, Thanks for doing this! I was glancing at the timelines, and the graph for the coverage benchmark struck me: CPython on aarch64 : Timeline Between August 19 and August 27 the time this took shot up from 100ms to 800ms. Almost everything else was pretty stable.
I didn’t see a similar jump for this benchmark on speed.python.org : Python Speed Center : Timeline 1 Like diegor (Diego Russo) October 12, 2023, 5:31pm 9 @guido , that was a good catch! When selecting “Display all in a grid” it’s impossible to spot it. Anyway I’ve investigated this further and I saw the same behaviour in our internal infrastructure as well. The interesting fact was that the issue happened across all our machines (both aarch64 and x86 ).
Here for example happening on an Intel Skylake: skylake 2223×479 42.9 KB I was able to restrict the window of the change to August 21st. Around that day we didn’t change anything in our infrastructure and we have pyperformance pinned to 1.0.8. When looking at the logs of the build I noticed though a difference on how the coverage package was built. Before the 21st there was something like this: [2023-08-19T06:38:45.800Z] 2023-08-19 06:38:45,660: Building wheels for collected packages: coverage [2023-08-19T06:38:45.800Z] 2023-08-19 06:38:45,662: Building wheel for coverage (setup.py): started [2023-08-19T06:38:48.283Z] 2023-08-19 06:38:47,779: Building wheel for coverage (setup.py): finished with status 'done' [2023-08-19T06:38:48.292Z] 2023-08-19 06:38:47,780: Created wheel for coverage: filename=coverage-6.4.1-cp313-cp313-linux_aarch64.whl size=216910 sha256=8cd103bb7a9c0017c85fa19b03a3730ef05d45f71383366c1737e08cf91b01e7 After the 21st: [2023-08-23T00:35:50.189Z] 2023-08-23 00:35:50,087: Building wheels for collected packages: coverage [2023-08-23T00:35:50.204Z] 2023-08-23 00:35:50,089: Building wheel for coverage (setup.py): started [2023-08-23T00:35:51.292Z] 2023-08-23 00:35:51,000: Building wheel for coverage (setup.py): finished with status 'done' [2023-08-23T00:35:51.300Z] 2023-08-23 00:35:51,001: Created wheel for coverage: filename=coverage-6.4.1-py3-none-any.whl size=176625 sha256=81073f3f8773eeb93b90da2d4457bf1303426b5fa73a180fbe055f531c021747 Notice the difference in the wheel size and the platform tag: one is cp313-cp313-linux_aarch64 and the other is py3-none-any .
The wheel with the cp313-cp313-linux_aarch64 platform tag has a shared object file tracer.cpython-310-aarch64-linux-gnu.so whilst the other one no. This explains the difference in size and in the platform tag. Coverage provides two implementations of the tracer: the ctracer (faster) and the python based one ( official documentation ) Hence since August 21st we see that coverage switched to using the python implementation of the tracer of coverage.
The next question was to understand why this happens. So I went and built myself with a version of cpython before and after August 21st to replicate the issue. The one before the 21st with with no problems whilst the one after raised an error when trying to build the ctracer. Here is the error: creating build/temp.linux-aarch64-cpython-313/coverage/ctracer gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -I/home/ent-user/ci-scripts/venv/cpython3.13-b51db007c75b-compat-e3aaa88db39d/include -I/home/ent-user/ci-scripts/tmpdir/prefix/include/python3.13 -c coverage/ctracer/datastack.c -o b uild/temp.linux-aarch64-cpython-313/cov
The rest of this page is on the live site.