Add test environment approval step for CI #13297
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
ko3n1g
previously approved these changes
Apr 28, 2025
ko3n1g
reviewed
Apr 28, 2025
ko3n1g
reviewed
Apr 28, 2025
ko3n1g
reviewed
Apr 28, 2025
ko3n1g
reviewed
Apr 28, 2025
ko3n1g
reviewed
Apr 28, 2025
| if result: | ||
| total_workflows += 1 | ||
| else: | ||
| print(f"Failed to approve deployment {deployment['id']}") |
There was a problem hiding this comment.
What are reasons for a failed deployment? What else could do we in that situation? Can we just retry? Or would it better to send a team alert? I guess in this situation we have a free slot but we’re not making use it
There was a problem hiding this comment.
well, a retry is probably just something that happens in a 5 minutes. Probably we can send an alert to the channel. I'll see if I can add it.
Co-authored-by: oliver könig <okoenig@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: oliver könig <okoenig@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: oliver könig <okoenig@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
ko3n1g
approved these changes
Apr 28, 2025
yuanzhedong
pushed a commit
that referenced
this pull request
May 1, 2025
* Add test environment approval step for CI Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Add cron job to approve test workflows in the queue Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Run Approve Test Queue on push for testing Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Debug test queue script Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Rewrite and debug test approval in Python Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Fix script to approve workflows Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Revert "Run Approve Test Queue on push for testing" This reverts commit 889d48a. Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Update .github/workflows/cicd-approve-test-queue.yml Co-authored-by: oliver könig <okoenig@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Update .github/workflows/cicd-approve-test-queue.yml Co-authored-by: oliver könig <okoenig@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Update .github/workflows/cicd-approve-test-queue.yml Co-authored-by: oliver könig <okoenig@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Notify automation team via slack if test queue approval bot failed Signed-off-by: Charlie Truong <chtruong@nvidia.com> --------- Signed-off-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> Signed-off-by: Yuanzhe Dong <yudong@nvidia.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Add test environment approval step for CI
Our current CI tests will run unit tests first and then e2e tests across different groups. However, if there are many PRs opened at the same time, then GHA will do a round robin on all the PRs rather than focusing on a single PR to completion. This can cause the queue to grow over time. This funnels all CI tests through a deployment approval first. Then a cron job is ran every 5 minutes to check if more jobs can be approved given a MAX_CONCURRENCY variable.
I ran this manually by pushing to this branch successfully.
https://github.com/NVIDIA/NeMo/actions/runs/14696800163/job/41239549859?pr=13297
MAX_CONCURRENCY is set to
1for now in themainenvironment. We could even pause CI tests now by setting it to0and then also allow other jobs to jump the queue by manually approving ourselves if necessary.Collection: [Note which collection this PR will affect]
Changelog
Usage
# Add a code snippet demonstrating how to use thisGitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information