WIP: Release pipeline without offline steps (PR to discuss the ideas) #6246

andife · 2024-07-21T07:57:10Z

Description

Remove offline steps for the official release build in order to increase supply chain security.

In a nutshell:

all 4 release jobs are combined by converting them to reusable workflows and invoked by a caller job create_release
publishing to pypi is removed from the single os jobs
trusted publishing is used (for the wheels, TODO: the source distribution has to be adapted)
Only if all builds and tests of the different operating systems are successful, the next step is the Prep Release. This is where the wheels are signed and the individual wheels are merged (this was made necessary by the github action upgrade). After this step, in case of a release, the signatures are added to the github release and the wheels are pushed to pypi.
As before, I would be less strict with the Weekly and will push it if the run was successful from any operating system.

Points that have not yet been fully implemented

I added sigstore directly and not the github attestation functionality, but that could be debated.
This step is independent of the pipeline change, I would have added it right away.

From my point of view, our pipeline is still missing the step of manually starting a dev build and pushing the results to testpypi

I would still prefer it if the release builds of the different OS would contain the same functionality

I would also take the building of the source build out of the mac pipeline
And the upload to *pypi

These points do not need to be part of the PR in the hope of keeping the PR “manageable”?

I look forward to discussing the aspects and adapting them to our needs.

Motivation and Context

(https://github.com/onnx/onnx/issues/5927)](https://github.com/onnx/onnx/issues/5927)

Signed-off-by: Andreas Fehlner <fehlner@arcor.de>

codecov · 2024-07-21T08:01:32Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 57.22%. Comparing base (83194ed) to head (1635c6c).
Report is 123 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6246      +/-   ##
==========================================
+ Coverage   56.95%   57.22%   +0.26%     
==========================================
  Files         506      507       +1     
  Lines       30467    31398     +931     
  Branches     4592     4691      +99     
==========================================
+ Hits        17353    17968     +615     
- Misses      12285    12577     +292     
- Partials      829      853      +24

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Andreas Fehlner <fehlner@arcor.de>

justinchuby · 2024-07-23T17:26:03Z

.github/workflows/create_release.yml

+    name: Release (Publish to pypi and add files to github release)
+    runs-on: ubuntu-latest
+    needs: [prepare-release]
+    if: always() && (needs.prepare-release.result == 'success') && startsWith(github.base_ref, 'rel-') # TODO only if a release is requested


Is it here that we should check that the trigger is scheduled?

What exactly do you mean by that?

I think one should take another look at all my steps to see if the checks were chosen correctly.

I currently also assume that the Github release is generated via the GUI and I only add the files in this step accordingly. In this release step, you would now also have to define a Github deployment environment, which you link to Pypi, so that the reviewers first have to approve the step before the Whl is actually pushed. I think you would have 30 days for the review, you can download the wheels beforehand for local testing. Maybe the handling of the RC candidates is currently missing? Do we also want them on pypi? or would they be better off on testpypi?

.github/workflows/release_mac.yml

docs/ArtifactAuthenticity.md

justinchuby · 2024-07-23T17:30:11Z

docs/ArtifactAuthenticity.md

What would be the status of this document? Does this need to go through infra sig?

I would say what our process offers or suggests. I am happy to present and discuss my ideas. According to the PR title where I originally wanted to remove the offline steps, I thought the signing step made sense now.
I would definitely want to add slsa later, but I thought I wouldn't overload the PR that way.

I think the document could do the job so far.

In any case, the adjustments in https://github.com/onnx/onnx/blob/main/docs/OnnxReleases.md are still pending, but I think a certain amount of discussion is still necessary, but I will make a suggestion.

These changes then definitely have to go through SIG/Infra, or we need the comments from the people who have made releases.

docs/ArtifactAuthenticity.md

justinchuby · 2024-07-23T17:35:36Z

Some minor refinement may be needed but looks good to me otherwise. Thank you for the huge improvement!

Co-authored-by: Justin Chu <justinchuby@users.noreply.github.com> Signed-off-by: Andreas Fehlner <fehlner@arcor.de>

andife · 2024-07-23T20:42:52Z

Because I was asked for that. In my opinion we should create deployment environments (one video describing that functionality could be found here: https://www.youtube.com/watch?v=EOlm3ft0VPo )

Finally it would like that in production

In my opinion we should use teams here. Maybe "arch/infra" or "former release manager"? Has to be discussed.

andife · 2024-08-28T09:26:39Z

First draft of the release (rc and final) workflow

graph TB
    subgraph GitHub GUI
        A[Trigger: Draft a new Release in Github GUI and select the 1.X.X Branch] --> B[SaveDraft or Publish Release?] 
    end

    subgraph CI Pipeline
        B --> D1[create_release.yml is called]

        D1 --> D{All 4 different OSPipelines were successful?}
        D -->|Yes| E[Job 3: Call Prepare Release Workflow]
        D -->|No| F[Job: Send Failure Notification]
        E --> G[Job 4: Prepare Release Workflow]
    end
    
    subgraph Prepare Release Workflow
        G --> H[Job 5: Download all artifacts]
        H --> I[Job 6: Sign all artifacts with sigstore]
    end

    subgraph Integration Testing
        I --> J[Job 7: External Integration Tests]
        J --> K{Integration Tests Passed?}
        K -->|Yes| L[Job 8: Manual Approval for Release]
        K -->|No| M[Job: Rollback & Investigate]
        M --> F
    end

    subgraph Release
        L --> N{Is it an official release?}
        N --> |Yes| O1[Job 9: Publish to PyPI]
        N --> |No| O2[Job 9: Publish to rc-candidate to TestPyPI]        
        O1 --> O[Job 10: Add signature files to GithubRelease]
        O2 --> O[Job 10: Add signature files to GithubRelease]
        O --> P[Job 11: Tag Release in GitHub]
        P --> Q1[Job 12: Create Release Notes]
        Q1 --> Q[Job 13: Add files to release]
    end


    Q --> R[End Pipeline]
    F --> R
    M --> R

@justinchuby

justinchuby · 2024-08-28T15:25:08Z

The diagram flow looks very good and clean, thanks!

andife · 2024-08-29T04:50:25Z

Actually, I would also like to list our RC candidates in the github releases, but that could get confusing? (see here https://github.com/vercel/next.js/releases). There doesn't seem to be a good way to filter via the GUI (https://github.com/orgs/community/discussions/6108) ? Since we have mostly managed with 2 RC candidates so far, this would not be a restriction for me at least (could be nicer for me, but ok)

Think the RC process should be as close as possible to the regular process (it should also be tested)

Any comments?
@cjvolzka @justinchuby

Signed-off-by: Andreas Fehlner <fehlner@arcor.de>

andife · 2024-09-21T05:38:12Z

For testing the pipeline I would prefer alpha and beta. We don't have to start with rc-candidates

…nnx#6277) ### Description  This pull request should be the start of a pipeline (current status of the considerations under onnx#6246) that automatically generates a whls after tagging, signs it and publishes it directly to Pypi to improve supply chain security. As this is all too much at once, the changes will be introduced and tested gradually. Changes related to this PR: Reusable Workflows are used to merge the individual OS release builds together (we need one process as we want to have all artifacts in one pipeline, so there is not need to download them manuelly). The workflows are currently not really "reusable", as we more or less use pypi creditionals in the workflow, among other things. I reused our testpypi weekly repo as I didn't want to change our behavoir for pypi and testpypi. Maybe it would make sense to adapt the process right away so that you can trigger our "Weekly" manually via the Github interface for testing and don't have to wait a week ;-) #### Key points for the review: * How do we want to configure concurrency * What concurrency-groups do we need? * Where do we need "cancel-in-progress"? #### Not part of the pull request: * Extract publish whl to pypi/testpyp * Introduce trusted publishing * ... ### Motivation and Context * see onnx#6246 --------- Signed-off-by: Andreas Fehlner <fehlner@arcor.de> Co-authored-by: Justin Chu <justinchuby@users.noreply.github.com> Signed-off-by: Linsho Kaku <linsho@preferred.jp>

andife · 2024-11-05T09:18:35Z

Interesting read:

https://discuss.python.org/t/pre-pep-exposing-trusted-publisher-provenance-on-pypi/42337/38

mihaimaruseac · 2024-11-05T13:48:09Z

We're planning to go towards trusted publishing after model signing stable release in https://github.com/sigstore/model-transparency/

gramalingam · 2025-02-18T01:59:01Z

Hi Andreas, should we plan to get these into the upcoming 1.18 release? We'd like to pick a deadline for the release soon

Margen67 · 2025-03-16T02:21:58Z

.github/workflows/release_mac.yml

@@ -55,13 +59,14 @@ jobs:
        # Install Protobuf from source
        export NUM_CORES=`sysctl -n hw.logicalcpu`
        source workflow_scripts/protobuf/build_protobuf_unix.sh $NUM_CORES $(pwd)/protobuf/protobuf_install
-        if [ '${{ github.event_name }}' == 'schedule' ]; then
+        if [ '${{ github.event_name }}' == 'schedule' ]; then      


Whitespace

Suggested change

if [ '${{ github.event_name }}' == 'schedule' ]; then

if [ '${{ github.event_name }}' == 'schedule' ]; then

Margen67 · 2025-03-16T02:22:11Z

.github/workflows/release_mac.yml

          sed -i '' 's/name = "onnx"/name = "onnx-weekly"/' 'pyproject.toml'
          export ONNX_PREVIEW_BUILD=1
        fi
-        python -m build --wheel
+        python -m build --wheel    


Whitespace

Suggested change

python -m build --wheel

python -m build --wheel

Margen67 · 2025-03-16T02:25:29Z

.github/workflows/release_win.yml

+    
+    - name: Build ONNX wheel      


Whitespace

Suggested change

- name: Build ONNX wheel

- name: Build ONNX wheel

Signed-off-by: Andreas Fehlner <fehlner@arcor.de>

Margen67 · 2025-03-19T00:28:09Z

.github/workflows/release_mac.yml

@@ -20,20 +22,22 @@ permissions:

 jobs:
  build:
-    if: github.event_name != 'pull_request' || startsWith( github.base_ref, 'rel-') || contains( github.event.pull_request.labels.*.name, 'run release CIs')
+    if: github.event_name != 'pull_request' || startsWith( github.base_ref, 'rel-') || contains( github.event.pull_request.labels.*.name, 'run release CIs') 


Whitespace

Suggested change

if: github.event_name != 'pull_request' || startsWith( github.base_ref, 'rel-') || contains( github.event.pull_request.labels.*.name, 'run release CIs')

if: github.event_name != 'pull_request' || startsWith( github.base_ref, 'rel-') || contains( github.event.pull_request.labels.*.name, 'run release CIs')

andife · 2025-08-08T19:18:02Z

Our release workflow does not contain offline steps for onnx 1.19 any more. Documentation parts should be improved

andife added 5 commits July 21, 2024 06:30

update pipeline

85e9035

Signed-off-by: Andreas Fehlner <fehlner@arcor.de>

remove separete date step

c837276

Signed-off-by: Andreas Fehlner <fehlner@arcor.de>

update

70676d8

Signed-off-by: Andreas Fehlner <fehlner@arcor.de>

update

86932e4

Signed-off-by: Andreas Fehlner <fehlner@arcor.de>

update

5e0484d

Signed-off-by: Andreas Fehlner <fehlner@arcor.de>

andife requested a review from a team as a code owner July 21, 2024 07:57

andife temporarily deployed to testpypi July 21, 2024 07:57 — with GitHub Actions Inactive

update

a1449d5

Signed-off-by: Andreas Fehlner <fehlner@arcor.de>

andife temporarily deployed to testpypi July 21, 2024 10:45 — with GitHub Actions Inactive

andife added 6 commits July 21, 2024 12:58

update

65fc2ef

Signed-off-by: Andreas Fehlner <fehlner@arcor.de>

update

b379125

Signed-off-by: Andreas Fehlner <fehlner@arcor.de>

update

08557e4

Signed-off-by: Andreas Fehlner <fehlner@arcor.de>

update sigstore github action

d3389ea

Signed-off-by: Andreas Fehlner <fehlner@arcor.de>

update

13cf836

Signed-off-by: Andreas Fehlner <fehlner@arcor.de>

update

01875e2

Signed-off-by: Andreas Fehlner <fehlner@arcor.de>

andife changed the title ~~WIP: release pipeline without manual steps~~ Release pipeline without manual steps Jul 22, 2024

andife changed the title ~~Release pipeline without manual steps~~ Release pipeline without offline steps Jul 22, 2024