Addressing dependency version confusion exploit chain - Packaging - Discussions on Python.org
Addressing dependency version confusion exploit chain Packaging themanifold (themanifold) February 10, 2021, 10:24am 1 Hello, In light of recent articles about package managers preferring higher version packages. The exploit path is essentially this: An organisation creates a custom python library with a certain version The org places the library in a requirements file with a relaxed or no version spec, along with a mixture of modules available in pypi Someone comes and squats on their custom python library in pypi, with a higher version pip will then install the squatted library from pypi, as pip prefers newer versions We actually talked about this very problem here : If the version in svn is later than any on PyPI, it will be picked. if not, it won’t.
If the latest version on PyPI is the same as the version number in svn, then either can be picked, but they should be identical (because that’s what having the same version number means) and so why would you care? If you want to pick the version in svn in preference to those on PyPI, give it a higher version number. Only in this ticket, we’re just talking about svn. I suppose we can say that orgs have a responsiblity to pin their version numbers, but I think this warrants a discussion about what our defaults are.
One of suggestions in the linked issue is to use --no-index but that requires splitting requirements into things that are internal and public. This may be fine, but at least we should note and document it somewhere as a special use case. uranusjr (Tzu-ping Chung) February 10, 2021, 10:42am 2 The general recommendation for organisational usages like this to build a private proxy instead of accessing PyPI directly, since files distributed on PyPI (those with different names from your private packages) are inherently no more trustworthy to you as well.
There are multiple tools for building a auditable private PyPI mirror for your organisation, such as bandersnatch and devpi . If you really want to arbitrarily pick and choose what you believe is safe, there are also “pass-through proxy” tools that provide half-assed security guarantees. devpi provides many related functionalities, and I built another myself if you don’t want to set up a full-fledged high-volume web server.
The idea is you point --index-url to your own server so you have full control of what package name means what. 1 Like themanifold (themanifold) February 10, 2021, 10:52am 3 Thanks for your response @uranusjr Where is the documentation for this? I’m not disbelieving you, I just don’t know where it is. But yes, overall, I agree with your sentiment, I just wonder if there isn’t more that we can do. pf_moore (Paul Moore) February 10, 2021, 11:04am 4 I don’t believe there is much documentation on this.
It’s inherently something that larger companies building Python applications would be interested in, but we’ve not had much contribution from such organisations in the form of documentation of best practices like this. The majority of PyPA members are volunteers, so ideally this sort of documentation would be contributed by third parties who have experience with this sort of model. 1 Like kushaldas (Kushal Das) February 11, 2021, 11:47am 5 I gave a talk in PyCon US 2019 explaining how the SecureDrop project does this.
I also wrote a blog post today explaining the newer changes we brought into the workflow for the same. All of our code is available in the related git repository. We have reproducible source tarballs reproducible wheels (from both public and private packages) OpenPGP signature verification of the hashes and finally reproducible Debian packages from them 4 Likes themanifold (themanifold) February 12, 2021, 9:22am 6 This is really great.
So how do we go about introducing this stuff to the wider community? steve.dower (Steve Dower) February 12, 2021, 10:44am 7 The disclosure of this attack is so far the best way of getting people’s attention on it. It’s just going to have to be announced repeatedly in the hope that people will look for solutions. I got Microsoft to publish a generic (and almost entirely non-advertisment) whitepaper on it with high level guidance.
We didn’t know that “dependency confusion” would stick as the name. Microsoft warns enterprises of new 'dependency confusion' attack technique | ZDNet https://aka.ms/pkg-sec-wp It’s deliberately not a list of products or companies that offer workarounds, but the hope is that it’ll get people looking for them. Conference talks and blog posts (which are dated and not expected to be “forever” references like a paper) are better venues for sharing specifics like that, and hopefully citing our paper will give them enough legitimacy to be accepted by those who write them.
6 Likes westurner (Wes Turner) February 12, 2021, 2:01pm 8 Pulp and DevPi have docs on this: Project Summaries - Python Packaging User Guide devpi-latest Documentation https://pulp-python.readthedocs.io/en/latest/ And there’s this: westurner (Wes Turner) February 12, 2021, 2:27pm 9 Are there CWE (Common Weakness Enumeration) or CAPEC URIs for this new attack pattern? Is it a: “CWE-610: Externally Controlled Reference to a Resource in Another Sphere” CWE - “CWE-829: Inclusion of Functionality from Untrusted Control Sphere” CWE - CWE-1104: Use of Unmaintained Third Party Components" CWE - Is this correct: if you specify a non-PyPI index-url, it’s possible that you’ll install compromised packages (that have a higher version number); and the potential impact is the same as for other software supply chain issues?
Does TUF specify a signed mapping between package names and keys? AFAIU, this issue is not resolved in e.g. the DEB and RPM ecosystems either (any GPG key imported to the GPG keyring can sign for any version of any package from any index-url); though you can specify: channel priority which packages to allow to be installed from a given channel: allow/block per-package hashes for all platforms (like pipenv) per-package hashes for just one platform (e.g.
“arm64” in pip requirements.txt hashes) It’s also an option to solve the deptree locally, freeze it to requirements.txt with package hashes for the target platform,and then install that with pip install --no-deps ; or generate a distributable binary with bundled static deps and no at-dev-time/at-install-time dynamic retrieval of updated dependencies with something like blaze or pants. kpfleming (Kevin P. Fleming) February 12, 2021, 5:01pm 10 Keep in mind that for the specific attack published last week, it is sufficient to just specify pinned versions.
That’s all. The attack was predicated entirely on being able to publish a version number higher than any legitimately-published version number, and relying on unpinned dependencies in installation processes. Hash-comparison, signature-checking, etc. are all welcome as additional ways to protect against tampered packages, illegitimate packages, etc. but aren’t strictly necessary for this attack. 3 Likes steve.dower (Steve Dower) February 12, 2021, 7:32pm 11 Afraid not, at least not in the general case (possibly for pip as it stands today).
All the package managers out there treat package name+version as the primary key, and so if two indexes claim to have the same version they are able to choose from whichever they like. NuGet takes the fastest response (presumed lowest latency), when I looked, pip doesn’t seem to choose one reliably. My guess would be that it normally ends up with --index-url rather than any --extra-index-url , but it may also get the first processed response (presumably, lowest latency).
If your private server has MyPackage==1.0.0 and one day PyPI also has it, you’ve got some chance of getting either. There are no defined or specified rules for how pip decides (nor any other package manager (except Conda)). For this attack, it is sufficient to claim the name of each internal package on every single feed you referen
The rest of this page is on the live site.