Brainstorming: Eliminating Dynamic Metadata - Packaging - Discussions on Python.org
Brainstorming: Eliminating Dynamic Metadata Packaging mitsuhiko (Armin Ronacher) November 16, 2024, 9:01pm 1 I’m spinning this out of PEP 751: now with graphs! to avoid derailing the topic there too much. Today the Python packaging ecosystem supports dynamic metadata. This in theory can be use with almost any key in pyproject.toml but most commonly causes problems with version where it’s also very commonly used. The reasons people like to use dynamic metadata are not too surprising and on the surface it sounds really convenient!
Unfortunately having metadata be dynamic causes challenges and they are a pretty significant tax to the ecosystem. So in some sense it would be nice to be able to do the same thing, but without actually being dynamic. So I was thinking maybe we have a discussion here why dynamic metadata is used and what alternatives would be, that are not actually dynamic, or at least are not quite as dynamic. Challenges: Caching : caching of dynamic metadata is hard, because it’s not known what can invalidate it.
Build dependencies needed: in order to retrieve dynamic metadata you first need to install the dependencies necessary to run the code that emits that metadata. This can be a rather expensive step which can dramatically slow down package installation. In particular it punishes systems like uv that want to automatically re-sync the virtualenv constantly. The time spent resolving metadata dynamically for a single package can be slower than all other operations [1] .
Unstable metadata: dynamic metadata is often taking external state such as git commits or general git status into account, which results in metadata changing under the hood without the installer or resolver knowing about it. Yet that metadata is in fact frozen in the metadata upon installation. This is particularly odd for editable installs where you might have installed the package at version 0.0.1+deadbeef , but you are already multiple commits away from that.
In the past that often meant that entrypoints no longer found the package, and you had to run pip install --editable again. Uses of Dynamic Metadata: version : pretty commonly people try to make version match the latest git tag, the current git revision hash or similar. In part the motivation here is to also just sync up __version__ in a package with the installed metadata. dependencies (and other keys with similar functionality): a motivating example for this I have seen is to make the attribute match a provided requirements.txt file.
Sometimes it’s also just set as dynamic because for legacy reasons the requirements are still set in setup.py . readme : this surprised me, but there is a package hatch-fancy-pypi-readme that has some adoption which apparently re-composes the readme from other inputs scripts , gui-scripts , entry-points : these are typically set when used with setuptools (eg: when setup.py is still in use) Complicating factors: non-installed packages: I don’t know how much of a problem this is still today, but one of the reasons people cannot use the common pattern __version__ = importlib.metadata.version("MyDist") is that they support importing a package without (editable or else) installation.
When dynamic metadata is not an issue: As more and more packages are published as wheels, dynamic metadata becomes less of an issue as the metadata is frozen within the wheel. It’s thus more of an issue for editable installs and sdist installations. I intentionally don’t want to prime this topic with proposed solutions or alternatives and see if we can do some basic brainstorming here of what a world would look like that does not have dynamic metadata, or greatly cuts down on how much dynamic metadata is permitted.
Motivating example: you can put your version to an attribute like package.__version__ with the setuptools backend. Importing that package on some large applications can take seconds, if that root package in itself decides to import part of the system. Sometimes this involves shelling out to git or other tools to read version information. ↩︎ 18 Likes PEP 751: now with graphs! fungi (fungi) November 16, 2024, 9:41pm 2 Wheels have static metadata, sdists (currently) allow metadata to be generated at build time.
If future sdists mandate static metadata, then there will simply emerge a proto-format that gets prebuilt into static sdists. The need for dynamic metadata doesn’t go away simply by declaring it inconvenient. 11 Likes sirosen (Stephen Rosen) November 16, 2024, 9:43pm 3 Focusing narrowly on version info, the motivating use case which was provided was to push each commit to test pypi. This seems satisfiable with some kind of “version smudge script” which could easily be packaged up for reuse.
That leaves the question of what the un-smudged version should be, since it presumably doesn’t matter for most development purposes. The problem seems solvable today, with existing tools, but I have the vague sense that it’s missing some of the desired workflow characteristics of SCM-derived versions. jamestwebber (James Webber) November 16, 2024, 10:19pm 4 ^ This use-case is supported by tools like bump2version , but they’re definitely a little clunkier than using a dynamic value.
Basically it just does a find/replace on your code base, based on a config file. Not something I’d use but I’ve encountered it. I’ll throw in another motivation for a dynamic version, which isn’t common but does come up: for a Rust extension package using maturin , the version is defined in Cargo.toml and the Python extension gets it from there [1] . In this case it’s not all that dynamic but it’s not literally in the file.
I nudged them to declare this in the pyproject.toml so it followed the spec ↩︎ pf_moore (Paul Moore) November 16, 2024, 10:25pm 5 Sdists can have static metadata, and as build backends improve their support, it will become much easier for data to be static unless there’s a specific need for it to be dynamic. The biggest blocker here is with setuptools, where the ability to write plugins means that even if data looks static, it’s hard for the backend to guarantee that it is.
But that’s a problem that should get fixed over time - the intention is clearly there to use static data whenever possible. Assuming that’s the case, we’re left with a situation where metadata is dynamic because the user genuinely needs it to be dynamic. We can’t just wish those cases away, or declare them invalid. We have to understand why the need is there, and find a way of addressing the need using only static data.
This may be possible, but it will need proper, focused research and outreach to projects that currently use dynamic metatdata. A discussion here is unlikely to come to any usable conclusions without that sort of research. I think it’s extremely important to remember here that the project version is not allowed to be dynamic in a sdist . Everyone makes a big deal about setuptools_scm and similar tools for getting the version from VCS, but they simply aren’t an issue for sdists.
The problem with versions only occurs when building from arbitrary source trees. This is a problem, certainly, but it’s arguably more of an issue for project workflow tools than for packaging tools. I’d be more than happy, for example, for someone to publish an installer that only supported installing from formally packaged projects (sdists and wheels). I think it’s critical that we keep in mind the scenarios we’re concerned about here.
Managing arbitrary source trees is a significant concern for workflow tools like uv, poetry, hatch or PDM. But there are plenty of other problems there as well - environment management, locking, editable installs, etc. However, there are plenty of other packaging tools which are not workflow managers, and which don’t have the same issues to deal with. It’s worth remembering that almost nothing about source trees (as opposed to sdists) is standard
The rest of this page is on the live site.