Allow package references as version specifiers - Packaging - Discussions on Python.org
Allow package references as version specifiers Packaging jens (Jens Troeger) September 18, 2022, 9:17pm 1 Hello, I’d like to illustrate the problem using the grpcio and grpcio_tools packages: both are currently at version v1.49.0 and both live in the same Github repo , in subfolders src/python/grpcio/ and tools/distrib/python/grpcio_tools/ , respectively. Together with other packages and other languages, their versions bump in lock-step when released.
When I use both packages in a project then I can’t pin them because other package dependencies would conflict with that pin, and pip would be unable to resolve the conflict. So I end up with dependency declarations like dependencies = [ "grpcio >=1.46.0,<2.0.0", "grpcio-tools >=1.46.0,<2.0.0", ] And that’s where things get a little iffy: it may happen due to dependencies declared in other packages that the two packages install in different versions.
If the packages use semantic versioning correctly then all may be well, as is the case with grpcio-tools and its dependency on grpcio ( code ) — still there is a good chance that the two packages install at different versions. * Packages whose type stubs ship as a third-party package are other examples of the problem. I wonder if it would make sense to express a “package reference” as a version specifier (expanding on PEP 440 ), for example: dependencies = [ "grpcio >=1.46.0,<2.0.0", "grpcio-tools @=grpcio", ] meaning that both packages have the same version range but eventually are expected to resolve to the same installed version within that range.
If the target package of a @= isn’t specified then that’d be an error; if the target package pins then that same pinned version would apply. Considering that @ is already used for direct file references , using @= may be confusing or ambiguous. I’m curious what people make of this Jens ————— * Other packages, however, are out of lockstep completely as is the case with googleapi-common-protos at v1.56.4 and its third-party, unmaintained stubs package at v2.0.0.
Likewise, the protobufs package at v4.21.6 (for Python) and stubs in typeshed at v3.20. Ideally, I think, they ought to release at the same versions but that’s a different issue altogether. pradyunsg (Pradyun Gedam) September 18, 2022, 9:48pm 2 I’m not too keen on this. This changes the context for parsing and handling a dependency requirement from something that can be evaluated and processed independently to something that has external dependencies itself and needs a preprocessing step.
Is there some reason this couldn’t be achieved outside of custom syntax in the Packaging tooling, such as autogenerating this via cog or something similar? 1 Like pf_moore (Paul Moore) September 18, 2022, 10:14pm 3 I agree, this seems like something where you would have been better off designing things differently so that they worked with the existing mechanisms. I don’t really see why the 2 libraries have to be in lock step like this.
But if they do, then judging from the names it seems like if you use grpcio-tools, you need grpcio. So why not have grpcio-tools X.Y.Z depend on grpcio X.Y.Z? That expresses the dependency explicitly. Or if the two libraries really are completely independent (which seems weird given the tight version coupling) you could create a dummy package grpcio-base, and have grpcio X.Y.Z depend on grpcio-base X.Y.Z, and grpcio-tools X.Y.Z also depend on grpcio-base X.Y.Z.
I’ve probably missed something in the above. But my point stands, this sounds like something you could probably solve within the existing functionality, as long as you’re willing to adjust your package structure to make it happen. I don’t really think we should add extra functionality just for a single, very specific use case like this. 4 Likes ofek (Ofek Lev) September 18, 2022, 10:47pm 4 A Hatch user just solved a similar use case with a custom metadata hook: Question: How to replace the relative path to version while building? · Issue #469 · pypa/hatch · GitHub pyproject.toml : [tool.hatch.metadata.hooks.custom] hatch_build.py : from hatchling.metadata.plugin.interface import MetadataHookInterface class CustomMetadataHook(MetadataHookInterface): def update(self, metadata): metadata['dependencies'] = [...] If you just want relative direct references, Hatch supports this use case too, see “Tip” in Dependencies - Hatch 1 Like pradyunsg (Pradyun Gedam) September 18, 2022, 11:08pm 5 And approach of handling this in a custom plugin for a package build system takes seems like a good one (and might be a better answer than the cog-style code generation I suggested above).
I’d prefer that this be handled there, instead of in the dependency specifiers that indexes and installers like PyPI/pip need to handle. Having build dependencies available as install dependencies layday (layday) September 19, 2022, 12:41pm 6 They should really be declaring dependencies as dynamic if they are doing that ofek (Ofek Lev) September 19, 2022, 2:12pm 7 3 Likes jens (Jens Troeger) September 26, 2022, 10:03pm 8 I’d like to clarify and perhaps expand with a more general example: We see many packages out there, popular ones included, that don’t contain typing annotations.
For example, issue #795 of the Babel package discusses the problem and hints at a second package in Typeshed. Fortunately, these two are versioned in lock-step: Babel is currently at release v2.10.2 and Its type stubs are declared as v2.10.* which matches. If I’d like to use Babel and its types then the above idea would enable me to specify dependencies = [ "babel >=2.8.0,<2.11.0", "types-babel @=babel", ] and whichever tool resolves package dependencies would ensure that both package versions are the same within the specified range.
Of course, packages whose types are not versioned in lockstep with their package would fail here. @ofek I’m curious to learn more about your suggestion—it would require me to switch from Flit to Hatch I assume, and then… ? layday (layday) September 27, 2022, 7:27am 9 types-babel does not track babel . For X.Y.Z , only X and Y are kept in sync with babel ; the micro/patch version denotes the version of the type definitons for babel ’s X.Y release.
The latest version of babel is 2.10.3 and that of types-babel is 2.10.0. This is the case for the majority of third-party stubs maintained in the typeshed which (are assumed to) follow semver. (Calver package stubs get a fourth a segment with the stub version, e.g. 22.9.27.0 , with 0 being the stub version.) Depending on how you envision types-babel @=babel to work it might either be unresolvable or it might “artificially” resolve babel down to an older version. brettcannon (Brett Cannon) September 28, 2022, 7:04pm 10 I would assume the type checkers will download the type annotations as needed based on the version of the library installed so you don’t have to list the explicit dependency. layday (layday) September 28, 2022, 7:45pm 11 I’m not aware of a type checker that does that.
How would they know which version to download? brettcannon (Brett Cannon) September 28, 2022, 7:48pm 12 You can get it from the .dist-info directory for the installed library. layday (layday) September 28, 2022, 7:50pm 13 How would they know which version of the published stubs is compatible with the installed library? pradyunsg (Pradyun Gedam) September 28, 2022, 8:47pm 14 mypy has --install-types . https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-install-types layday (layday) September 28, 2022, 9:29pm 15 Well, like it says on the tin, that includes only a smaller number of stub packages, and I don’t think mypy does any version checking.
But it’s good to know that the option exists CAM-Gerlach (C.A.M. Gerlach) September 28, 2022, 9:33pm 16 Unfortunately, AFAIK that mostly if not exclusively just uses the centrally maintained, automatically deployed typeshed-generated stub packages plus maybe a fe
The rest of this page is on the live site.