Add CI test for local checkpointing #13012

ananthsub · 2025-04-14T16:49:36Z

Important

The Update branch button must only be pressed in very rare occassions.
An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.

What does this PR do ?

Adds single node 2gpu test for local checkpointing support

Collection: llm

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

tests/collections/llm/test_local_ckpt.py

nemo/collections/llm/recipes/log/default.py

nemo/lightning/nemo_logger.py

tests/collections/llm/test_local_ckpt.py

Signed-off-by: ananthsub <ananthsub@users.noreply.github.com>

.github/workflows/cicd-main-nemo2.yml

Signed-off-by: Ananth Subramaniam <ananth.subramaniam@gmail.com>

ko3n1g

good from automation perspective, let me know when to merge

ananthsub · 2025-05-06T18:43:57Z

@ko3n1g this is good to merge

* Add end to end test for local checkpoint Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * update tests Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lint Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * Update tests/functional_tests/L2_NeMo_2_llama3_local_ckpt.sh Signed-off-by: Ananth Subramaniam <ananth.subramaniam@gmail.com> Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * address comments from review Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * update test script Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * update test Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lint Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * clean up assertions in log Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * combine converage files Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * use nemo run for test launch Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * rebase after new CI workflow change Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * Apply isort and black reformatting Signed-off-by: ananthsub <ananthsub@users.noreply.github.com> * Update .github/workflows/cicd-main-nemo2.yml Signed-off-by: Ananth Subramaniam <ananth.subramaniam@gmail.com> --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: Ananth Subramaniam <ananth.subramaniam@gmail.com> Signed-off-by: ananthsub <ananthsub@users.noreply.github.com> Co-authored-by: ananthsub <ananthsub@users.noreply.github.com> Co-authored-by: oliver könig <okoenig@nvidia.com>

* Add end to end test for local checkpoint Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * update tests Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lint Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * Update tests/functional_tests/L2_NeMo_2_llama3_local_ckpt.sh Signed-off-by: Ananth Subramaniam <ananth.subramaniam@gmail.com> Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * address comments from review Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * update test script Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * update test Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lint Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * clean up assertions in log Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * combine converage files Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * use nemo run for test launch Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * rebase after new CI workflow change Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * Apply isort and black reformatting Signed-off-by: ananthsub <ananthsub@users.noreply.github.com> * Update .github/workflows/cicd-main-nemo2.yml Signed-off-by: Ananth Subramaniam <ananth.subramaniam@gmail.com> --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: Ananth Subramaniam <ananth.subramaniam@gmail.com> Signed-off-by: ananthsub <ananthsub@users.noreply.github.com> Co-authored-by: ananthsub <ananthsub@users.noreply.github.com> Co-authored-by: oliver könig <okoenig@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com>

* Add end to end test for local checkpoint Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * update tests Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lint Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * Update tests/functional_tests/L2_NeMo_2_llama3_local_ckpt.sh Signed-off-by: Ananth Subramaniam <ananth.subramaniam@gmail.com> Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * address comments from review Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * update test script Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * update test Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lint Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * clean up assertions in log Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * combine converage files Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * use nemo run for test launch Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * rebase after new CI workflow change Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * Apply isort and black reformatting Signed-off-by: ananthsub <ananthsub@users.noreply.github.com> * Update .github/workflows/cicd-main-nemo2.yml Signed-off-by: Ananth Subramaniam <ananth.subramaniam@gmail.com> --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: Ananth Subramaniam <ananth.subramaniam@gmail.com> Signed-off-by: ananthsub <ananthsub@users.noreply.github.com> Co-authored-by: ananthsub <ananthsub@users.noreply.github.com> Co-authored-by: oliver könig <okoenig@nvidia.com>

….0` (#13472) * Add CI test for local checkpointing (#13012) * Add end to end test for local checkpoint Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * update tests Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lint Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lints Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * Update tests/functional_tests/L2_NeMo_2_llama3_local_ckpt.sh Signed-off-by: Ananth Subramaniam <ananth.subramaniam@gmail.com> Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * address comments from review Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * update test script Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * update test Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * lint Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * clean up assertions in log Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * combine converage files Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * use nemo run for test launch Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * rebase after new CI workflow change Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * Apply isort and black reformatting Signed-off-by: ananthsub <ananthsub@users.noreply.github.com> * Update .github/workflows/cicd-main-nemo2.yml Signed-off-by: Ananth Subramaniam <ananth.subramaniam@gmail.com> --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: Ananth Subramaniam <ananth.subramaniam@gmail.com> Signed-off-by: ananthsub <ananthsub@users.noreply.github.com> Co-authored-by: ananthsub <ananthsub@users.noreply.github.com> Co-authored-by: oliver könig <okoenig@nvidia.com> * Add local checkpoint test to CI file Signed-off-by: Charlie Truong <chtruong@nvidia.com> --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: Ananth Subramaniam <ananth.subramaniam@gmail.com> Signed-off-by: ananthsub <ananthsub@users.noreply.github.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: ananthsub <ananthsub@users.noreply.github.com> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com>

ananthsub requested review from pablo-garay, ko3n1g, thomasdhc and chtruong814 as code owners April 14, 2025 16:49

github-actions bot added the CI label Apr 14, 2025

ananthsub changed the title ~~Local ckpt test~~ Add CI test for local checkpointing Apr 14, 2025

github-advanced-security bot found potential problems Apr 14, 2025

View reviewed changes

tests/collections/llm/test_local_ckpt.py Fixed Show fixed Hide fixed

tests/collections/llm/test_local_ckpt.py Fixed Show fixed Hide fixed

tests/collections/llm/test_local_ckpt.py Fixed Show fixed Hide fixed

tests/collections/llm/test_local_ckpt.py Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Apr 14, 2025

View reviewed changes

tests/collections/llm/test_local_ckpt.py Fixed Show fixed Hide fixed

ananthsub added Run CICD r2.3.0 Pick this label for auto-cherrypicking into v2.3.0 labels Apr 23, 2025

ananthsub force-pushed the local-ckpt-test branch from d266610 to 50e8320 Compare April 23, 2025 08:02

ko3n1g added Run CICD and removed Run CICD labels Apr 23, 2025

ananthsub force-pushed the local-ckpt-test branch from b3cc07c to 59dc4ca Compare April 23, 2025 08:09

ko3n1g added Run CICD and removed Run CICD labels Apr 23, 2025

github-advanced-security bot found potential problems Apr 23, 2025

View reviewed changes

nemo/collections/llm/recipes/log/default.py Fixed Show fixed Hide fixed

nemo/lightning/nemo_logger.py Fixed Show fixed Hide fixed

tests/collections/llm/test_local_ckpt.py Fixed Show fixed Hide fixed

ananthsub force-pushed the local-ckpt-test branch from 6512ab3 to 2d71cb8 Compare April 23, 2025 08:19

ko3n1g added Run CICD and removed Run CICD labels Apr 23, 2025

ananthsub force-pushed the local-ckpt-test branch from 11192f9 to 1f0de38 Compare April 23, 2025 08:20

ko3n1g added Run CICD and removed Run CICD labels Apr 23, 2025

github-advanced-security bot found potential problems Apr 23, 2025

View reviewed changes

tests/collections/llm/test_local_ckpt.py Fixed Show fixed Hide fixed

ko3n1g added Run CICD and removed Run CICD labels Apr 29, 2025

Apply isort and black reformatting

898a1fb

Signed-off-by: ananthsub <ananthsub@users.noreply.github.com>

ko3n1g added Run CICD and removed Run CICD labels Apr 29, 2025

maanug-nv previously approved these changes Apr 29, 2025

View reviewed changes

ananthsub added Run CICD and removed Run CICD labels Apr 30, 2025

ko3n1g had a problem deploying to test April 30, 2025 19:29 — with GitHub Actions Error

Merge branch 'main' into local-ckpt-test

06878db

ko3n1g added Run CICD and removed Run CICD labels May 2, 2025

ko3n1g temporarily deployed to test May 4, 2025 08:59 — with GitHub Actions Inactive

ananthsub commented May 5, 2025

View reviewed changes

.github/workflows/cicd-main-nemo2.yml Outdated Show resolved Hide resolved

Update .github/workflows/cicd-main-nemo2.yml

3129df9

Signed-off-by: Ananth Subramaniam <ananth.subramaniam@gmail.com>

ananthsub dismissed maanug-nv’s stale review via 3129df9 May 5, 2025 17:35

ko3n1g added Run CICD and removed Run CICD labels May 5, 2025

Merge branch 'main' into local-ckpt-test

5e4c4a5

ko3n1g added Run CICD and removed Run CICD labels May 6, 2025

ko3n1g temporarily deployed to test May 6, 2025 16:08 — with GitHub Actions Inactive

ko3n1g approved these changes May 6, 2025

View reviewed changes

ko3n1g merged commit f2db26d into NVIDIA:main May 6, 2025
204 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add CI test for local checkpointing #13012

Add CI test for local checkpointing #13012

Uh oh!

ananthsub commented Apr 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ko3n1g left a comment

Uh oh!

ananthsub commented May 6, 2025

Uh oh!

Uh oh!

Uh oh!

Add CI test for local checkpointing #13012

Add CI test for local checkpointing #13012

Uh oh!

Conversation

ananthsub commented Apr 14, 2025

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ko3n1g left a comment

Choose a reason for hiding this comment

Uh oh!

ananthsub commented May 6, 2025

Uh oh!

Uh oh!

Uh oh!