Skip to content

fix: fix a answer parsing bug in MMLU-Pro. #598

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jul 8, 2025
Merged

fix: fix a answer parsing bug in MMLU-Pro. #598

merged 1 commit into from
Jul 8, 2025

Conversation

xxman-google
Copy link
Contributor

What does this PR do ?

Fix a bug in answer parsing in MMLU-Pro
The previous answer parsing function is designed for MMLU where only 4 options are available. In MMLU-Pro, there are ten options.

Issues

N/A

Usage

  • Run MMLU-Pro evaluation.
uv run examples/run_eval.py \
--config examples/configs/evals/mmlu_pro.yaml \
cluster.gpus_per_node=8

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

N/A

terrykong
terrykong requested changes Jul 3, 2025
View reviewed changes
Copy link
Contributor

@terrykong terrykong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the fix. do you mind writing a unit test for this?

@terrykong terrykong requested a review from yuki-97 July 3, 2025 04:51
@xxman-google
Copy link
Contributor Author

thanks for the fix. do you mind writing a unit test for this?

Done.

yuki-97
yuki-97 reviewed Jul 3, 2025
View reviewed changes
yuki-97
yuki-97 reviewed Jul 3, 2025
View reviewed changes
@xxman-google
Copy link
Contributor Author

@yuki-666 comments addressed, PTAL again.

@xxman-google
fix a answer parsing bug in MMLU-Pro.
db3bc3e 
Signed-off-by: Xuehan <xxman@google.com>
@yuki-97
Copy link
Contributor

yuki-97 commented Jul 4, 2025

Thanks @xxman-google ! Approved.

@parthchadha @terrykong Can you take a review and enqueue it if it looks good to you?

@parthchadha parthchadha enabled auto-merge July 7, 2025 21:28
@terrykong terrykong self-requested a review July 7, 2025 21:49
@parthchadha parthchadha added this pull request to the merge queue Jul 7, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jul 7, 2025
@xxman-google
Copy link
Contributor Author

@yuki-666 do you know why this PR cannot be merged?

@parthchadha parthchadha added this pull request to the merge queue Jul 7, 2025
@parthchadha
Copy link
Contributor

@yuki-666 do you know why this PR cannot be merged?

I have added this PR to merge queue.

@xxman-google
Copy link
Contributor Author

@yuki-666 do you know why this PR cannot be merged?

I have added this PR to merge queue.

Terry added it but the bot removed it due to some failed status.

@parthchadha
Copy link
Contributor

@yuki-666 do you know why this PR cannot be merged?

I have added this PR to merge queue.

Terry added it but the bot removed it due to some failed status.

Oh, CI failed due to https://github.com/NVIDIA-NeMo/RL/actions/runs/16128642365/job/45512107088. Looks like an intermittent issue not related to your PR.

@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jul 8, 2025
@terrykong terrykong added this pull request to the merge queue Jul 8, 2025
Merged via the queue into NVIDIA-NeMo:main with commit 8c329aa Jul 8, 2025
13 of 14 checks passed
@xxman-google xxman-google deleted the xx/fix_mmlu_pro branch July 11, 2025 16:41
jialei777 pushed a commit to jialei777/nemo-rl that referenced this pull request Jul 23, 2025
@xxman-google @jialei777
fix: fix a answer parsing bug in MMLU-Pro. (NVIDIA-NeMo#598)
76b6c89 
Signed-off-by: Xuehan <xxman@google.com>
Signed-off-by: Jialei Chen <jialeic@google.com>
KiddoZhu pushed a commit that referenced this pull request Jul 28, 2025
@xxman-google @KiddoZhu
fix: fix a answer parsing bug in MMLU-Pro. (#598)
e300064 
Signed-off-by: Xuehan <xxman@google.com>
FannYYW pushed a commit to xxman-google/NeMo-RL that referenced this pull request Aug 5, 2025
@xxman-google
fix: fix a answer parsing bug in MMLU-Pro. (NVIDIA-NeMo#598)
1abe385 
Signed-off-by: Xuehan <xxman@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@terrykong terrykong terrykong approved these changes

@parthchadha parthchadha parthchadha approved these changes

@yuki-97 yuki-97 yuki-97 approved these changes

Assignees
No one assigned
Labels
external
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@xxman-google @yuki-97 @parthchadha @terrykong @snowmanwwg