Fix Configuration for Micro Batch Size in Megatron's Ref Policy #1700
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ETOgaosion
reviewed
May 26, 2025
verl/workers/megatron_workers.py
Outdated
| else: | ||
| if self.config.ref.get("log_prob_micro_batch_size_per_gpu", None): | ||
| self.config.ref.ppo_micro_batch_size_per_gpu = self.config.ref.log_prob_micro_batch_size_per_gpu | ||
| elif self.config.ref.get("ppo_micro_batch_size_per_gpu", None): |
There was a problem hiding this comment.
Thanks for contribution!
I think that here is a typo, so we may not need to consider ppo_micro_batch_size_per_gpu, you can simply judge the key above~
There was a problem hiding this comment.
fix, del the ppo_micro_batch_size_per_gpu
ETOgaosion
approved these changes
May 28, 2025
ETOgaosion
pushed a commit
to Jianbing-D/verl
that referenced
this pull request
Jun 8, 2025
…cengine#1700) ### What does this PR do? Fix Configuration for Micro Batch Size in Megatron's Ref Policy ### High-Level Design This pull request addresses an issue with the micro batch size configuration in the ref policy of Megatron. The default ppo_megatron_trainer.yaml only includes two configurations: log_prob_micro_batch_size and log_prob_micro_batch_size_per_gpu. https://github.com/volcengine/verl/blob/54c9b7364c2d188b2ba4107404cfa3c2b446df19/verl/trainer/config/ppo_megatron_trainer.yaml#L119-L120 However, in `megatron_workers.py`, the required configuration is ref.log_prob_micro_batch_size_per_gpu https://github.com/volcengine/verl/blob/54c9b7364c2d188b2ba4107404cfa3c2b446df19/verl/workers/megatron_workers.py#L517-L518 or in `megatron_actor.py ` the required configuration is ref.ppo_micro_batch_size_per_gpu, https://github.com/volcengine/verl/blob/54c9b7364c2d188b2ba4107404cfa3c2b446df19/verl/workers/actor/megatron_actor.py#L271-L274 which are not directly related to ppo_micro_batch_size. To resolve this, I have made modifications to the configuration calculations and added raise ValueError statements to ensure that the necessary parameters are correctly defined. This update ensures that the required parameters are properly handled, preventing runtime errors and improving the overall robustness of the training process. ### Changes Made: - Modified the configuration calculations in megatron_workers.py. - Added raise ValueError statements to check for the presence of log_prob_micro_batch_size_per_gpu and ppo_micro_batch_size_per_gpu.
wwwjn
pushed a commit
to wwwjn/verl
that referenced
this pull request
Jun 10, 2025
…cengine#1700) ### What does this PR do? Fix Configuration for Micro Batch Size in Megatron's Ref Policy ### High-Level Design This pull request addresses an issue with the micro batch size configuration in the ref policy of Megatron. The default ppo_megatron_trainer.yaml only includes two configurations: log_prob_micro_batch_size and log_prob_micro_batch_size_per_gpu. https://github.com/volcengine/verl/blob/54c9b7364c2d188b2ba4107404cfa3c2b446df19/verl/trainer/config/ppo_megatron_trainer.yaml#L119-L120 However, in `megatron_workers.py`, the required configuration is ref.log_prob_micro_batch_size_per_gpu https://github.com/volcengine/verl/blob/54c9b7364c2d188b2ba4107404cfa3c2b446df19/verl/workers/megatron_workers.py#L517-L518 or in `megatron_actor.py ` the required configuration is ref.ppo_micro_batch_size_per_gpu, https://github.com/volcengine/verl/blob/54c9b7364c2d188b2ba4107404cfa3c2b446df19/verl/workers/actor/megatron_actor.py#L271-L274 which are not directly related to ppo_micro_batch_size. To resolve this, I have made modifications to the configuration calculations and added raise ValueError statements to ensure that the necessary parameters are correctly defined. This update ensures that the required parameters are properly handled, preventing runtime errors and improving the overall robustness of the training process. ### Changes Made: - Modified the configuration calculations in megatron_workers.py. - Added raise ValueError statements to check for the presence of log_prob_micro_batch_size_per_gpu and ppo_micro_batch_size_per_gpu.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fix Configuration for Micro Batch Size in Megatron's Ref Policy
High-Level Design
This pull request addresses an issue with the micro batch size configuration in the ref policy of Megatron. The default ppo_megatron_trainer.yaml only includes two configurations: log_prob_micro_batch_size and log_prob_micro_batch_size_per_gpu.
verl/verl/trainer/config/ppo_megatron_trainer.yaml
Lines 119 to 120 in 54c9b73
However, in
megatron_workers.py, the required configuration is ref.log_prob_micro_batch_size_per_gpuverl/verl/workers/megatron_workers.py
Lines 517 to 518 in 54c9b73
or in
megatron_actor.pythe required configuration is ref.ppo_micro_batch_size_per_gpu,verl/verl/workers/actor/megatron_actor.py
Lines 271 to 274 in 54c9b73
which are not directly related to ppo_micro_batch_size.
To resolve this, I have made modifications to the configuration calculations and added raise ValueError statements to ensure that the necessary parameters are correctly defined.
This update ensures that the required parameters are properly handled, preventing runtime errors and improving the overall robustness of the training process.
Changes Made:
Modified the configuration calculations in megatron_workers.py.
Added raise ValueError statements to check for the presence of log_prob_micro_batch_size_per_gpu and ppo_micro_batch_size_per_gpu.