2024 Python package manager overview - Packaging - Discussions on Python.org
2024 Python package manager overview Packaging h-vetinari (H. Vetinari) July 9, 2024, 8:41am 1 A colleague of mine (does not seem to have a handle here) pointed out the following article, which I think is a pretty nice – even if opinionated – high-level overview over the current state of affairs in python packaging land (at least from the POV of user-facing tools): The one nitpick I have is that I’d wish the article underscored a bit more is how pixi is explicitly trying to bridge the gap between the world of PyPI -based packages and conda packages, which is (in my biased opinion) one of the coolest things to happen in python packaging in a long time.
PS. I’ve found the author on github , but doesn’t seem to be on DPO either – in any case, thanks! 5 Likes kknechtel (Karl Knechtel) July 9, 2024, 1:21pm 2 Curiously enough, one of my friends, who I also don’t know to have a handle here, showed me the same article Personally I wish more were written about how we got to this point and why the problems aren’t trivial (including ones that are neatly, almost accidentally sidestepped by others). jeanas (Jean Abou Samra) July 9, 2024, 1:22pm 3 I wrote this thing a while ago, but it’s in French: L’installation et la distribution de paquets Python (1/4) - LinuxFr.org / L'installation et la distribution de paquets Python (2/4) - LinuxFr.org .
1 Like sinoroc (sinoroc) July 9, 2024, 4:35pm 4 There has been some. Let’s link them! The more we link them, the more likely people will find them and read them. The ones I recall: 6 Likes kknechtel (Karl Knechtel) July 9, 2024, 5:19pm 5 I thought I’d read Pradyun’s post before, but now I’m not so sure. I’ll probably want to dig into it (again). I’m not a huge fan of the threaded-short-message style of discourse (Twitter, Mastodon etc.) generally, and would much rather have had those points as a blog article.
But Stargirl’s main points are well taken. I guess it’s worth directly linking the previous PDO thread that got mentioned in the replies there, as well as the other blog post that came up a few times (since in particular, pointing out “how everyone else is doing it” doesn’t explain why - or if - we can’t follow suit, but is a very useful starting point for analysis): barry (Barry Warsaw) July 9, 2024, 6:04pm 6 I just (re-read?) it and it’s a great article.
Not to take away one iota of the great points Pradyun is making, but I think it underplays the perspective of the myriad of personas that are involved here, which is one of the fundamental reasons why IMHO, “fixing packaging” is so difficult. The article does touch on this under the On formal UX analysis section. I say this knowing full well that Pradyun totally does understand the complexities of the user cases and varying needs of all those personas.
Agreed [1] ! Quoting from Stargirl’s thread: You gotta deal with the Python Core team and the steering council. They have consistently abdicated the details of packaging to the community. They aren’t, at this time, very interesting in taking over packaging and telling the community how to manage their dependencies. There are decades of history for why and how we got in the current situation. Past active disinterest is certainly one component of that.
While I can’t speak for my SC colleagues, or for the SC officially, I can say that I am personally of the opinion that we’re way past the point where this laissez-faire approach is serving us well. I’m not sure I can say what the shape of a more hands-on attitude toward packaging looks like, but I think we need to recognize that this is causing considerable pain for our users, and it’s our collective responsibility to commit to a compassionate response to making this better, even though it may take another decade of sustained effort to solve. on both counts ↩︎ 17 Likes sinoroc (sinoroc) July 9, 2024, 6:12pm 7 @kknechtel I thought you meant to share resources that are written by the makers of Python packaging ecosystem as a counterpoint to the usual rants that keep popping up again and again.
I would not recommend re-sharing Chris Warrick’s article, it has too many inaccuracies in my opinion. This kind of articles articles keep getting “viral”, everyone has seen at least a couple of them. It feels to me like they attract too much negativity and are often inaccurate which does not help motivate the volunteers actually doing the work. 3 Likes kknechtel (Karl Knechtel) July 9, 2024, 7:38pm 8 I did, originally.
But now I see value in gathering all of these sorts of think-pieces. From what I could make of Chris’ post, it isn’t inaccurate (aside from perhaps a biased summary of prior PDO discussion) - it just presents opinions that are less than sympathetic, and fails to anticipate how people might respond if they got what they think they want. (I have more thoughts inspired by what’s said there about venvs in particular; but they belong in a separate thread.) A friend gave me another blog link I like in this context: https://blog.williammanley.net/2022/02/23/pip-and-cargo-are-not-the-same.html 1 Like brettcannon (Brett Cannon) July 10, 2024, 9:41pm 9 My requisite response to that XKCD comic .
4 Likes ofek (Ofek Lev) July 12, 2024, 4:48am 10 There is a condensed, UX representation of that comic at 4:25 https://youtu.be/BgxklT94W0I?si=EGqYv7hvziFIfMWi&t=265 This is why I am adamant about not expanding the capabilities of pip and always push back against ideas to ship a full project manager with every Python distribution. Users should only be downloading a tool like Hatch and this tool will manage everything for you with a clean UX.
Although Python has to support many more gnarly use cases compared to Cargo or NPM, there is no reason why everything should not be exposed through a centralized interface that lives independent of a Python installation. 4 Likes ncoghlan (Alyssa Coghlan) July 12, 2024, 2:16pm 11 While I was writing about CPython at the time, Considering Python's Target Audience | Curious Efficiency also ties into @theacodes ’s Mastodon thread about the diversity of use cases that Python’s packaging ecosystem needs to cover.
Specifically, the listed Python use cases frequently also have different needs and expectations around what they want to get out of a package management tool (ranging from “I don’t want to deal with any dependencies that aren’t already preinstalled” all the way up to “I want an opinionated workflow management tool that tells me how to structure my entire development project rather than just installing the packages it is told to install where it is told to install them”).
5 Likes BrenBarn (Brendan Barnwell) July 29, 2024, 5:57am 12 Very interesting. Some discussion about these tools is, it seems, at cross purposes because there are multiple base functions of which each tool performs a subset: environment manager package manager project manager Conda is an environment and package manager. pip is a package manager; venv is an environment manager. Poetry seems to be all three, but with the proviso that the project part is required (i.e., it cannot manage environments/packages independent of a defined project).
Articles like this talk about each tool without making clear that they differ in which of these tasks they even attempt. I still think it makes most sense to start with an environment manager, because packages and projects can be layered on top of that, but not vice versa. Also I get a kick out of this remark: The core tradeoff with conda is that it attempts to do package metadata the “right” way […] This can be onerous to adapt in a company that already has many internal python packages.
This looks to me like the all-too-common tragedy of “this tool’s main/only disadvantage is that everyone isn’t using it already”. 1 Like Kwpolska (Chris Warrick) July 29, 2024, 10:03pm 13 Instead of “too many inaccuracies”, it would be more constructive
The rest of this page is on the live site.