feat: Context Parallelism + Sequence Packing in Megatron + Dtensor #651
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Sahil Jain <sahilj@nvidia.com>
Signed-off-by: Sahil Jain <sahilj@nvidia.com>
…{Dict, List, Tuple} to primitive dict, list tuple
Signed-off-by: Sahil Jain <sahilj@nvidia.com>
Signed-off-by: Sahil Jain <sahilj@nvidia.com>
Signed-off-by: Sahil Jain <sahilj@nvidia.com>
Signed-off-by: Sahil Jain <sahilj@nvidia.com>
Signed-off-by: Sahil Jain <sahilj@nvidia.com>
Signed-off-by: Sahil Jain <sahilj@nvidia.com>
Signed-off-by: Sahil Jain <sahilj@nvidia.com>
Signed-off-by: Sahil Jain <sahilj@nvidia.com>
Signed-off-by: Sahil Jain <sahilj@nvidia.com>
Signed-off-by: Sahil Jain <sahilj@nvidia.com>
Signed-off-by: Sahil Jain <sahilj@nvidia.com>
Signed-off-by: Sahil Jain <sahilj@nvidia.com>
Signed-off-by: Sahil Jain <sahilj@nvidia.com>
Signed-off-by: Sahil Jain <sahilj@nvidia.com>
Signed-off-by: Sahil Jain <sahilj@nvidia.com>
Signed-off-by: Terry Kong <terryk@nvidia.com> wip Signed-off-by: Terry Kong <terryk@nvidia.com> fix it Signed-off-by: Terry Kong <terryk@nvidia.com> patthing fix Signed-off-by: Terry Kong <terryk@nvidia.com> wip Signed-off-by: Terry Kong <terryk@nvidia.com> doesn't look like i needed that Signed-off-by: Terry Kong <terryk@nvidia.com> fix Signed-off-by: Terry Kong <terryk@nvidia.com> revert stuff Signed-off-by: Terry Kong <terryk@nvidia.com> make it better Signed-off-by: Terry Kong <terryk@nvidia.com> go Signed-off-by: Terry Kong <terryk@nvidia.com> cleanup Signed-off-by: Terry Kong <terryk@nvidia.com> mix it up Signed-off-by: Terry Kong <terryk@nvidia.com> touch up Signed-off-by: Terry Kong <terryk@nvidia.com> clean Signed-off-by: Terry Kong <terryk@nvidia.com> better Signed-off-by: Terry Kong <terryk@nvidia.com> clean up Signed-off-by: Terry Kong <terryk@nvidia.com> add it in Signed-off-by: Terry Kong <terryk@nvidia.com> mcore extra Signed-off-by: Terry Kong <terryk@nvidia.com> instructions Signed-off-by: Terry Kong <terryk@nvidia.com> works Signed-off-by: Terry Kong <terryk@nvidia.com> revert to 3.10, 3.12 didn't seem necessary Signed-off-by: Terry Kong <terryk@nvidia.com> ci has to recursively clone Signed-off-by: Terry Kong <terryk@nvidia.com> bump build workflow Signed-off-by: Terry Kong <terryk@nvidia.com> add megatron.core import Signed-off-by: Terry Kong <terryk@nvidia.com> potential fix for unit test on CI Signed-off-by: Terry Kong <terryk@nvidia.com> fix the test Signed-off-by: Terry Kong <terryk@nvidia.com> this should fix test (it was a collision of namespace) Signed-off-by: Terry Kong <terryk@nvidia.com> remove fp8 from test Signed-off-by: Terry Kong <terryk@nvidia.com> add shallow Signed-off-by: Terry Kong <terryk@nvidia.com> fix base build Signed-off-by: Terry Kong <terryk@nvidia.com> fix instructions Signed-off-by: Terry Kong <terryk@nvidia.com> fix the messed up indenting Signed-off-by: Terry Kong <terryk@nvidia.com> fix Signed-off-by: Terry Kong <terryk@nvidia.com> try nesting Signed-off-by: Terry Kong <terryk@nvidia.com> okay, got it to work Signed-off-by: Terry Kong <terryk@nvidia.com> fix up the readme Signed-off-by: Terry Kong <terryk@nvidia.com> ok Signed-off-by: Terry Kong <terryk@nvidia.com> touchup Signed-off-by: Terry Kong <terryk@nvidia.com> add the lock file back Signed-off-by: Terry Kong <terryk@nvidia.com> got Signed-off-by: Terry Kong <terryk@nvidia.com>
Signed-off-by: Terry Kong <terryk@nvidia.com>
Signed-off-by: Terry Kong <terryk@nvidia.com>
Signed-off-by: Terry Kong <terryk@nvidia.com>
Signed-off-by: Terry Kong <terryk@nvidia.com>
… tied worker groups Signed-off-by: Sahil Jain <sahilj@nvidia.com>
…session scope Signed-off-by: Terry Kong <terryk@nvidia.com>
5816d64 to
35da884
Compare
parthchadha
previously approved these changes
Jul 21, 2025
Signed-off-by: Xuehan <xxman@google.com>
Signed-off-by: ashors1 <ashors@nvidia.com>
Signed-off-by: ashors1 <ashors@nvidia.com>
Signed-off-by: ashors1 <ashors@nvidia.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
Signed-off-by: Yuki Huang <yukih@nvidia.com> Co-authored-by: Terry Kong <terryk@nvidia.com>
Signed-off-by: Sahil Jain <sahilj@nvidia.com>
16e61ac to
7847c1b
Compare
Signed-off-by: Sahil Jain <sahilj@nvidia.com>
fa7e308 to
75bb1b2
Compare
|
closing since newer PR makes this one obsolete #704 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Enables Context Parallelism + Sequence Packing for Megatron-Core based training