🚨🚨🚨 [Trainer] Enable average_tokens_across_devices by default in TrainingArguments
#39395
Conversation
Fixes huggingface#39392 This change improves loss calculation correctness for multi-GPU training by enabling proper token averaging across devices by default.
|
@qgallouedec, did you face any issues after changing this default in trl ? Happy to do it transformers otherwise |
|
@SunMarc Hey! Happy to help test this on the trl side as well if needed. Let me know if you’d like me to run any specific checks! |
|
No. No issue that I'm aware of |
average_tokens_across_devices by default in TrainingArguments
|
Hey @SunMarc , I noticed the tests are failing on ci/circleci: tests_torch and run_tests, while the rest of the checks seem fine. |
No it's fine, i'll take care of merging this PR |
Thanks! |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
…rainingArguments` (huggingface#39395) Enable average_tokens_across_devices by default in TrainingArguments Fixes huggingface#39392 This change improves loss calculation correctness for multi-GPU training by enabling proper token averaging across devices by default. Co-authored-by: Krishnan Vignesh <krishnanvignesh@Krishnans-MacBook-Air.local> Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
Fixes #39392
This change improves loss calculation correctness for multi-GPU training by enabling proper token averaging across devices by default.
What does this PR do?
Changes the default value of
average_tokens_across_devicesfromFalsetoTrueinTrainingArguments. This ensures more accurate loss calculation in multi-GPU training scenarios by properly averaging tokens across devices.As noted in the original issue, this feature provides reproducibility and correctness benefits with no downsides, so there's no reason to keep it disabled by default.
Before submitting
average_tokens_across_devicesby default in Trainer #39392Who can review?
@zach-huggingface @SunMarc @qgallouedec (trainer maintainers)