Skip to content

⚖️ Add option not to scale rewards (Dr. GRPO) #3135

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Mar 22, 2025
Merged

⚖️ Add option not to scale rewards (Dr. GRPO) #3135

merged 2 commits into from
Mar 22, 2025

Conversation

qgallouedec
Copy link
Member

What does this PR do?

Fixes # (issue)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a GitHub issue? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@HuggingFaceDocBuilderDev
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@zaddy6
Copy link

zaddy6 commented Mar 22, 2025

Using scale_rewards=False, throws this error

[rank5]: Traceback (most recent call last):
[rank5]:   File "/workspace/never_peft.py", line 900, in <module>
[rank5]:     trainer.train()
[rank5]:   File "/opt/conda/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 2245, in train
[rank5]:     return inner_training_loop(
[rank5]:            ^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/opt/conda/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 2556, in _inner_training_loop
[rank5]:     tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank5]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/opt/conda/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 3712, in training_step
[rank5]:     inputs = self._prepare_inputs(inputs)
[rank5]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/opt/conda/envs/unsloth_env/lib/python3.11/site-packages/trl/extras/profiling.py", line 87, in wrapper
[rank5]:     return func(self, *args, **kwargs)
[rank5]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/opt/conda/envs/unsloth_env/lib/python3.11/site-packages/trl/trainer/grpo_trainer.py", line 647, in _prepare_inputs
[rank5]:     inputs = self._generate_and_score_completions(inputs)
[rank5]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/opt/conda/envs/unsloth_env/lib/python3.11/site-packages/trl/trainer/grpo_trainer.py", line 861, in _generate_and_score_completions
[rank5]:     self._metrics[mode]["reward_std"].append(std_grouped_rewards.mean().item())
[rank5]:                                              ^^^^^^^^^^^^^^^^^^^
[rank5]: UnboundLocalError: cannot access local variable 'std_grouped_rewards' where it is not associated with a value
  File "/opt/conda/envs/unsloth_env/lib/python3.11/site-packages/trl/extras/profiling.py", line 87, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/unsloth_env/lib/python3.11/site-packages/trl/trainer/grpo_trainer.py", line 647, in _prepare_inputs
    inputs = self._generate_and_score_completions(inputs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/unsloth_env/lib/python3.11/site-packages/trl/trainer/grpo_trainer.py", line 861, in _generate_and_score_completions
    self._metrics[mode]["reward_std"].append(std_grouped_rewards.mean().item())
                                             ^^^^^^^^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'std_grouped_rewards' where it is not associated with a value
[rank0]: Traceback (most recent call last):
[rank0]:   File "/workspace/never_peft.py", line 900, in <module>
[rank0]:     trainer.train()
[rank0]:   File "/opt/conda/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 2245, in train
[rank0]:     return inner_training_loop(
[rank0]:            ^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 2556, in _inner_training_loop
[rank0]:     tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 3712, in training_step
[rank0]:     inputs = self._prepare_inputs(inputs)
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/envs/unsloth_env/lib/python3.11/site-packages/trl/extras/profiling.py", line 87, in wrapper
[rank0]:     return func(self, *args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/envs/unsloth_env/lib/python3.11/site-packages/trl/trainer/grpo_trainer.py", line 647, in _prepare_inputs
[rank0]:     inputs = self._generate_and_score_completions(inputs)
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/envs/unsloth_env/lib/python3.11/site-packages/trl/trainer/grpo_trainer.py", line 861, in _generate_and_score_completions
[rank0]:     self._metrics[mode]["reward_std"].append(std_grouped_rewards.mean().item())
[rank0]:                                              ^^^^^^^^^^^^^^^^^^^
[rank0]: UnboundLocalError: cannot access local variable 'std_grouped_rewards' where it is not associated with a value

@qgallouedec
Copy link
Member Author

Thanks, it should be fixed now

@qgallouedec qgallouedec merged commit 9b38b0b into main Mar 22, 2025
9 of 14 checks passed
@qgallouedec qgallouedec deleted the dr-grpo branch March 22, 2025 20:47
@Peng-YM Peng-YM mentioned this pull request Mar 25, 2025
Closed
@lkevinzc lkevinzc mentioned this pull request Mar 25, 2025
Closed
kashif pushed a commit to kashif/trl that referenced this pull request Mar 28, 2025
@qgallouedec @kashif
⚖️ Add option not to scale rewards (Dr. GRPO) (huggingface#3135)
33f00a2
yxliu-TAMU pushed a commit to mincheolseong/ECEN743-GRPO-Project-Proposal that referenced this pull request Apr 20, 2025
@qgallouedec
⚖️ Add option not to scale rewards (Dr. GRPO) (huggingface#3135)
2b61a20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kashif kashif kashif approved these changes

@edbeeching edbeeching Awaiting requested review from edbeeching

@lewtun lewtun Awaiting requested review from lewtun

@shirinyamani shirinyamani Awaiting requested review from shirinyamani

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@qgallouedec @HuggingFaceDocBuilderDev @zaddy6 @kashif