PyPI's redesigned project pages with clearer security signals are now live (blog.pypi.org)
190 points by rbanffy 8 hours ago
time4tea 6 hours ago
Not very keen that the whole story of trust is basically centralisation - requiring "big cloud" and essentially promoting lock-in.
Did I get that right? Seems a bit anti-pythonic.
rtpg 2 hours ago
I believe you got that wrong.
Here's some extra context [0]: https://github.com/pypi/warehouse/issues/20927. Loooots of links to click in.
the point here being small service providers are _also_ able to get into this system, but I think there's an expectation of success.
The bar for trusted publishers is very high though. I don't think it's high "on purpose" to leave out certain platforms for being too small, but in effect if (for example) your provider doesn't have a security contact point then that's going to rule them out here.
duskdozer 6 hours ago
I think at this point the group of devs+ who are not fully on board with control by and dependence on big tech/VC is a small niche unfortunately.
my-huge-pony 3 hours ago
I don't think this is true. From my experience, developers are the people who care about this topic the most. Just look at HN, almost everyone (who speaks out) is against.
akerl_ 3 hours ago
125ajh 6 hours ago
You got that right. In this thread we have a ClosedAI employee promoting ClosedID services. Look at the sponsors of ClosedID:
https://openid.net/foundation/sponsoring-members/
We know why PyPI ripped out PGP support and has refused any verification mechanisms that other distributions use for decades.
cozzyd 6 hours ago
I trust myself way more to with with a pgp key on my computer than figuring out how to use GitHub actions properly...
theteapot 5 hours ago
bbor 5 hours ago
1. "ClosedAI" is not gonna win people over to your side, IMHO. Apple doesn't plant trees, Microsoft makes big hardware, and Python is not satanic.
2. I'm not entirely sure what the link is supposed to prove -- would appreciate a word of clarification if you find the time. It almost feels like a dogwhistle of some kind, but A) I doubt anyone on HN is anywhere near that problematic, and B) there's no clear pattern in those companies other than "interested in ID tech", and none of the usual dogwhistle targets made the list. Also, y'know, OpenAI isn't on it...?
I guess my fundamental confusion isn't about the list and more about what "we know". Are you implying that they are against pure, unmanaged PGP for this system because... I mean, why? Because they're playing the long game to promote a verification/attestation/identification (?) technology that you think offers less stringent privacy guarantees, and thus to sell more ads in 2036?
If so, doesn't that seem like a kinda roundabout way to raise Display Ads revenue, otherwise known as "that one dying industry"?
bbor 5 hours ago
It's anti lots of things, but I don't think it's anti-Pythonic. Python is about pragmatism & empathy if it's about anything -- bitcoin and such don't factor into it. See: Astral/OpenAI, Anaconda, most of the PSF board historically being employees of big corporate users of python, NVidia's relation to the ecosystem in general, etc.
As a passionate anarchosyndicalist, I, for one, am thrilled about it. Decentralization is a tool in our toolbox for identifying and dismantling unjust hierarchies, but it is not a panacea, and can be quite damaging when taken to the extreme. See: cryptocurrency speedrunning 200y of bank regulations in 20y.
To put my trust for Python in more concrete terms: they support GitLab. As a passionate GitLab shill, I really couldn't be more impressed! It continues to shock me how infrequently the second most popular forge is acknowledged, much less supported. This shows they're not just, IDK, bowing to Microsoft's demands.
woodruffw 4 hours ago
> This shows they're not just, IDK, bowing to Microsoft's demands.
Not only don't we, but I must emphasize that Microsoft did not pay for anything involving the implementation of either Trusted Publishing or attestations on PyPI. I wrote a personal blog post explaining some of my own rationale for why we prioritized GitHub and GitLab for PyPI's initial support targets, and it shouldn't be surprising to anybody who knows me that the reasons are pragmatic, not dogmatic[1].
[1]: https://blog.yossarian.net/2024/11/18/Security-means-securin...
pdonis 3 hours ago
time4tea 4 hours ago
chrisweekly 3 hours ago
djoldman 7 hours ago
Some examples:
dlt (no provenance): https://pypi.org/project/dlt/#files
datasette (yes provenance): https://pypi.org/project/datasette/#files
MeteorMarc 7 hours ago
The source distributions have icons while the wheels have not. Strange and confusing.
woodruffw 6 hours ago
That's probably a mobile bug (I can't reproduce it on a desktop browser). Could you open an issue here[1]?
[1]: https://github.com/pypi/warehouse/issues/new?template=bug-re...
Latty 6 hours ago
I see the icon for both, but in different places (below the name for the wheel, alongside for the source).
samhclark 6 hours ago
This is great! I want sigstore-based or at least sigstore-style "keyless" short lived signatures to become more and more popular, so that support gets better everywhere. In particular, if having secure OIDC handling in a CI/CD pipeline becomes something that keeps people on GHA or GitLab then I'd hope third-party hosted CI/CD services begin to take that gap seriously and work on meeting those requirements.
I'm bought in on keyless signing, and I want there to be many more providers, so I like to see more adoption which builds more momentum.
rcxdude 23 minutes ago
>I'd hope third-party hosted CI/CD services begin to take that gap seriously and work on meeting those requirements.
How is a third party service, especially a self-hosted one, expected to meet those requirements? They seem to be lengthy, onerous, and also require a detailed vetting by the pypi team that is unlikely to scale beyond a few large hosting providers.
(What's more, this does relatively little to actually signal trust to someone downloading them. All it does is provide a signature that one of the big providers has built the package from a given source (but nothing about the trustworthiness of that source). This is a problem much better solved by reproducable builds and distributed verification of those builds)
ImJasonH 5 hours ago
Tangled and Codeberg both seem to support issuing JWTs during actions runs.
Codeberg: https://codeberg.org/mfenniak/oidc-jwt Tangled: https://tangled.org/docs/spindles/workflows/#audience
I think OIDC tokens from workflows is basically table stakes at this point, since the alternative is a static secret, which is basically guaranteed to leak eventually.
All that's left is for PyPI to accept these tokens in their set of trusted publishers.
woodruffw 4 hours ago
Please see the other comments in this thread (as well as on PyPI's issue tracker) for some important context here. It's not enough to throw JWTs over the wall, PyPI also needs to know that it can trust any given service to operationalize the ugly and risk-prone parts of managing PKI.
(I haven't looked at Tangled at all. If Tangled has a secret management story for its PKI that's stronger than a private key on disk and can demonstrate that a non-trivial number of PyPI users would benefit from federation, then it's a strong candidate.)
ImJasonH 2 hours ago
mhils 7 hours ago
Very cool to see PyPI put more emphasis towards exposing provenance info. It would be even cooler if trusted publishing would finally support reusable workflows (https://github.com/pypi/warehouse/issues/11096).
pdonis 6 hours ago
It would be even nicer if there were a way to have an attestation without publishing with a workflow controlled by one of the tech giants. There's no way for me, someone who has had a PyPI account for many years and has used it for all that time to upload packages, to attach my own public key to my PyPI account and then sign uploaded distributions with my private key so the signature can be verified and count as an attestation.
angry_octet 42 minutes ago
I think the problem is this type of attestation is hard to weigh as a trust signal, vs the cloud CI version.
Inherently, if someone steals your key, or gains access to a box that can use that key for signing, they are indistinguishable from you. Hand PGP signing also lacks time-bound signing, any ratchet mechanism, CT log, consistent key hierarchy, or standard to bind signer name and project identity.
pdonis 4 minutes ago
tzlahp 6 hours ago
If only there had been PGP signatures. Oh wait ...
duskdozer 6 hours ago
Maybe they'll implement PGoogleP, where your private key is safely and securely stored on Google's servers
cowlevel 7 hours ago
This is about detecting formerly benign projects that have had their pypi entries hacked, but not their private keys. It does not detect malicious projects.
Kwpolska 6 hours ago
And about detecting projects which dared to move their release builds away from GitHub.com/GitLab.com.
eesmith 7 hours ago
Am I correct in my understanding that the only trusted publishers are for-profit US companies? And these all use proprietary implementations which cannot easily be replicated elsewhere?
Like, if I want to host on Codeberg, or Sourcehut, what is the process needed for those platforms to become a trusted publisher?
As it happens, I build my releases on my own hardware using a pypa/manylinux release. Can I provide attestation, or am I required to use a third-party service to get that "x Without provenance" removed from my entry?
woodruffw 6 hours ago
Support for Codeberg is being discussed here[1]. I think there's broad consensus among PyPI's maintainers and admins that more publisher providers are desired, but we also need them to demonstrate strong key management and rotation controls due to the degree of trust and access that becoming a publisher provider grants them (effectively, they become the PyPI credential broker for the projects that use their service).
(If you pan through that thread, you'll note that one of open issues with Codeberg is that their underlying server wants to store OIDC private keys on disk at the moment. This is okay for small deployments of personal services, but not for something that PyPI needs to federate with and trust.)
eesmith 5 hours ago
So I, as someone who does my builds locally, can never have attested builds?
Will PyPI be presenting warnings or statements of caution which might cause users to be concerned about using my distributions? Or imply that my build practices are somehow inferior to depending on a third-party build system?
guipsp 5 hours ago
woodruffw 5 hours ago
tzlahp 6 hours ago
Of course: In principle we'd welcome a change, but there are technical difficulties in perpetuity.
woodruffw 6 hours ago
eddythompson80 6 hours ago
Sounds like it. Currently it seems you'll need to setup your own OIDC provider[0], then work with PyPI to get them to trust your instance as a "Trusted Publisher". It seems like a system to scale for multi-tenant publishers (like millions of GitHub or GitLab or Google build or ActiveState users). The design itself should be able to scale for individual OIDC providers through webfinger and OIDC discovery like Tailscale does it[1], but it doesn't look like it's there yet. Not sure if it's planned either the github issue mentiones future PyPI verified email attestation, though I'm not sure if that implies webfinger/OIDC like ts does it.
[0] https://docs.pypi.org/trusted-publishers/
[1] https://tailscale.com/docs/integrations/identity/custom-oidc
woodruffw 6 hours ago
> The design itself should be able to scale for individual OIDC providers through webfinger and OIDC discovery like Tailscale does it[1], but it doesn't look like it's there yet.
This is the right idea, but it's actually not that simple unfortunately: Trusted Publishing relies on per-IdP claims that are more precise than the `sub` claim (because doing pattern matching on the `sub` claim is brittle and prone to security problems). So we require publisher providers (OIDC IdPs) to provide distinct claims that we can match on. For GitHub and GitLab that's things like the repository name, workflow filename, etc., while for others it might be distinct username and immutable ID claims, etc.
This is all downstream of OIDC-driven federation being a much weaker and more bespoke process than all parties would like it to be. I think the discourse around this could be much improved by people taking the time to understand these complexities and why they pose a challenge to groups with limited engineering resources (like PyPI).
WhyNotHugo an hour ago
Provenance attestation is about security compliance, not real security. At a big organisation, you can check off compliance check-boxes because you're using Microsoft's big hosted service.
For regular developers, provenance makes no difference; whether the package was made in a box controlled by Microsoft with a proprietary stack or on a box controlled by the developer make no difference.
darkamaul 7 hours ago
Not directly the subject but they switched from Material for MkDocs to Zensical.
It’s quite sad to see that one of the main (or the biggest?) documentation system for the Python completely alienated its user base and led to this fork.
duskdozer 6 hours ago
Judging by the readme and homepage, there were probably some large differences brewing. Can't say I'm a fan of how it looks like they'll take the UI
s_u_d_o 6 hours ago
Great news! Would love to see provenance on Homebrew (brew) as well.
woodruffw 6 hours ago
Homebrew actually has full provenance. We added it in 2024 (I think, that’s from memory). We never flipped the default verification policy because of some integration concerns, but my understanding is that various downstreams and MDMs do consume and verify Homebrew’s attestations.
Edit: I should qualify: Homebrew’s own processes verify its attestations. But that’s not enabled by default on end user machines because it slows installations down and currently requires the `gh` CLI. Neither of those is a tenable long term constraint for a default feature.
angry_octet 18 minutes ago
For curious readers, to turn it on:
brew install gh echo HOMEBREW_VERIFY_ATTESTATIONS=1 >> $HOME/.zshrc brew verify gh
Downloading attestations will be quicker if you are logged in to github: gh auth login gh auth status
For a box you don't want full login privs on, you can use a GH PAT token with minimal privs.
If too slow/GitHub down:
HOMEBREW_NO_VERIFY_ATTESTATIONS=1 brew upgrade
zzzeek 6 hours ago
let me tell you something incredibly important about the attestations / trusted publisher thing:
if you have any of your files published by a trusted publisher with attestations, then you must have all of your files published that way (in that package).
otherwise pypi determines your project may have been compromised and puts some big red warnings up like [1]. it does not try to look at "this particular source file was never part of trusted publishing" or anything like that. and, CRITICALLY, it does not send you an email or anything like that about the non-acceptable file that you just uploaded, which it REALLY should. I mean the model is, they think these uploads are malicious by an attacker. Emailing the project maintainer(s) would be pretty important! (When pypi wanted to get rid of GPG signatures, they started sending me an email for every single signature that got uploaded. so i know you guys know how to send emails).
it's also AFAICT a permanent switch. once you put a single trusted/attested file up there, your account must now do all files in this way forever (or risk big red warnings. which maybe you're OK with that).
It would all be easier to take if trusted publishers weren't limited to solely large for-profit corporations (I'm too lazy to research if they're trying to fix this).
woodruffw 6 hours ago
> it's also AFAICT a permanent switch. once you put a single trusted/attested file up there, your account must now do all files in this way forever (or risk big red warnings. which maybe you're OK with that).
It's not[1].
> It would all be easier to take if trusted publishers weren't limited to solely large for-profit corporations (I'm too lazy to research if they're trying to fix this).
As mentioned in several other places, we're always interested in adding more publisher providers, provided they meet our key management and notability requirements.
(This is also the easiest way to reason abductively about PyPI's goals. If PyPI's goal was to support "large for-profit corporations," we'd have dozens of publisher providers of dubious quality and community value, rather than the small subset we currently have.)
[1]: https://discuss.python.org/t/pypis-new-security-indicators-a...
zzzeek 6 hours ago
> This is also the easiest way to reason abductively about PyPI's goals. If PyPI's goal was to support "large for-profit corporations," we'd have dozens of publisher providers of dubious quality and community value, rather than the small subset we currently have.)
I never claimed pypi is trying to "support large for-profit corporations". I fully understand the "large corporations as trusted publisher" thing is because this particular subset, e.g. github, gitlab, google, are widely known institutions with "good" reputations (in this area). The issue is that pypi is not, as of yet, supporting individual projects that want trusted publishing without relying on a large for-profit corporation. I'm not too put off by it but it is definitely a thing that a private for-profit host overnight becomes horrible (see e.g. bitbucket) and I have to get off that company ASAP. Having some non-profit entities here would be extremely helpful.
woodruffw 6 hours ago
tzlahp 6 hours ago
You want the ones that employ core devs through Microsoft or are approved by OpenAI, which codeberg.org is not.
tzlahp 6 hours ago
This should be the top comment. PyPI as usual screws up and spreads misinformation about projects.
LdBaringSwallow 6 hours ago
> https://blog.pypi.org/posts/2024-11-14-pypi-now-supports-dig...
Thought we already learned years ago an IP address is not a person?
That is to say; what the screen on my end communicates is not guaranteed to be representative of ground truth.
The IT world was recently making fun of Sam Altman's eye scanner because remote attestation can be faked.
All the layers of abstraction wrapped around the code is security through obscurity.
The code itself must still be verified as safe.
The cloud era has resulted in a lot of people operating at layers of abstraction so far removed from physics this shit has more in common with baseless dogmas, gospels, and spiritualities at this point.
We anthropomorphized the cloud inserting ourselves into it. It's just a bunch of networked computers.
All this stuff is a spoon that doesn't exist.
bonoboTP 6 hours ago
You're speaking in such shorthand that I have no idea how to untangle what you're actually trying to say/propose. I suppose you're saying that attested code can still be malware because malicious actors can also go through the attestation process?
rcxdude 5 hours ago
Yeah, this check basically just proves that you hosted your source code on a big platform and then built the pypi package with their CI. This is not particularly strong as a trust signal. Especially since it doesn't particularly protect against compromise of access to the code repository. I worry that this is going to distract from better approaches (like reproducable builds and/or distributed review).
mistursinistur 7 hours ago
Nobody is looking at these pages
tzlahp 6 hours ago
True, therefore downvoted.