Hey @nvie, is there any chance you could verify if the jazzband bot account on PyPI has an Owner privilege or just a Maintainer?

Hi @webknjaz — sorry for the delay as I was flooded with GitHub notifications and this one didn't stand out enough. I just checked for you and indeed the Jazzband bot was a Maintainer, not an Owner. I just changed that for you. Let me know if there is anything else I can help you with! 🙏

@nvie thanks! This should let Jannis configure TP. I don't have access to the bot account.

Alternatively, I could ask you to configure TP if you're up for it. And someone with the repo settings access would also need to configure another bit.

Unfortunately I don't know what TP is (I'm no longer active in the Python community and haven't caught up enough with recent developments in the ecosystem).

@jezdez Given that the Jazzband bot is now an Owner, do you have enough to invite other Owners to the project as you see fit? I will let you handle that. If there is anything I can assist with, just let me know though!

@nvie oh, trusted publishing is a thing where PyPI can be configured to trust a specific GitHub Actions workflow and we can then upload new releases w/o needing to stick any secrets into the GitHub repo settings. Plus it now enables automatic digital attestations + other provenance bits through this OIDC-based mechanism.

Jazzband doesn't give the members direct access to PyPI, it's being proxied through a special server where people can preview the uploads. I'm seeking to get rid of that middle link, as it's now possible to implement everything within GitHub.

That said, @jezdez hasn't been available for a while so I figured I'd ask you for the PyPI setup confirmation, at least. Technically, it's Jannis who is supposed to configure things but we now established that it wouldn't be possible w/o you anyway. With the new bot privileges, though, Jannis will be able to add the configuration.

@jezdez this is what's needed to be configured on this page https://pypi.org/manage/project/pip-tools/settings/publishing/.

Owner: jazzband
Repository name: pip-tools
Workflow name: release.yml
Environment name: pypi

Just submit that form and the trust link will be set up.

It'd also be good to set up required reviewers for the pypi GitHub Environment in the GitHub repository settings. To do this, go to https://github.com/jazzband/pip-tools/settings/environments, create an environment called pypi, and enable required reviewers inside. Don't forbid self-review, add people/teams that would be allowed to green-light the release (up to 6 entries).

I've met @webknjaz at PyCon US, and we figured it out. Thanks, @nvie for your help, this is much appreciated. We have all the permissions we need.

Are these builds reproducible? We ought to include instructions on how to reproduce them.

Not entirely. setuptools doesn't create fully reproducible dists. There's a issue about that epoch timestamp not being applied to everything.

And this PR doesn't even attempt including the epoch. Though, I'd like to make many updates to the CI/CD, just don't want to pile up unrelated things.

I'm not sure I see the value in implementing this.

It provides assurance to downstream consumers that the builds happened in the infrastructure of a specific third party (i.e.: Microsoft).

Without reproducible builds, downstream has no justification for trusting this third party anyway.

Although fully reproducible builds would be even better, they aren't necessary for the trusted publishing configuration to offer value.

This configuration protects against compromise of an account or credential which owns or maintains a package. Downstreams can verify the attestations from PyPI, and can see that the build happened in a CI system -- in this case, GitHub Actions -- rather than my laptop or an attacker's device.

This also provides package signatures via sigstore. Others (including @webknjaz) are much more expert than I am in this topic, but I believe the signing is an integral step in the Trusted Publishing mechanism.

This configuration protects against compromise of an account or credential which owns or maintains a package.

Yes, it means there's no need to store PyPI tokens on your machine or inside GH settings.

It also makes releasing much easier and quicker, so maintainers can get new features and fixes out much more frequently. (I know there are other factors, but I've been waiting for the release of a fix that was merged in May 2024.)

@hugovk one complexity with migration to TP in Jazzband was that the standard project adoption process was that the previous maintainer would add the jazzband bot account as a Maintainer on PyPI. And that gave the bot an ability to use API-token based uploads. But to configure TP, the Owner-level access is required. It took time to gain said access just for pip-tools. Other projects in the org will face a similar challenge, which was something we've been occasionally discussing with Jannis when meeting at conferences.

Another bit was that the process includes uploading to Jazzband's own index from GHA and that server has a button to upload further into PyPI (which is where the API token is actually set up). The idea is that the release managers would be able to perform final inspection of the dists before green-lighting the final upload.

For many years, actions/upload-artifact was built in a way that GHA wouldn't show the artifacts until the workflow run was complete. With v4, this changed, and those artifacts now show up upon job completion, allowing us to use the GH UI to verify the dists before the final upload (and having a protected GitHub Environment assigned to the TP-enabled job).

But the intermediate Jazzband index has a few other uses. In particular, the release stats. And so when Jannis and I talked this though, we settled on still keeping that upload in the workflow.

That said, pip-tools should be ready to make use of TP now.

I believe the signing is an integral step in the Trusted Publishing mechanism.

@sirosen well, it's technically orthogonal. TP itself is about verifying the uploader identity (a workflow in a repo) and issuing a short-lived API token. That relies on OIDC and producing attestations also rely on OIDC (to authenticate in Sigstore). So they appear more coupled than they actually are.

Without reproducible builds, downstream has no justification for trusting this third party anyway.

As Stephen mentioned, this isn't just for downstreams. This is the upload trust + ability for us to simplify the release process. Reproducible builds is something that would be good to tackle additionally, but it's not in the scope of this PR.

I'm now going to merge it with the understanding that more improvements are possible later on.