Skip to content

feat: add MMMLU eval benchmark. #596

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jul 9, 2025
Merged

feat: add MMMLU eval benchmark. #596

merged 1 commit into from
Jul 9, 2025

Conversation

xxman-google
Copy link
Contributor

What does this PR do ?

Adds MMMLU eval benchmark.

Issues

N/A

Usage

  • Run eval on MMMLU (Chinese)
uv run examples/run_eval.py \
--config examples/configs/evals/mmlu.yaml \
data.dataset_name="mmlu_ZH-CN" \
cluster.gpus_per_node=8

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

*N/A

@github-actions github-actions bot added the documentation Improvements or additions to documentation label Jul 3, 2025
@terrykong terrykong requested a review from yuki-97 July 3, 2025 04:04
yuki-97
yuki-97 reviewed Jul 3, 2025
View reviewed changes
Copy link
Contributor

@yuki-97 yuki-97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @xxman-google , overall LGTM! Left some comments.

@xxman-google
Copy link
Contributor Author

@yuki-666 comments addressed. PTAL again.

yuki-97
yuki-97 reviewed Jul 4, 2025
View reviewed changes
yuki-97
yuki-97 reviewed Jul 4, 2025
View reviewed changes
@xxman-google
add MMMLU eval benchmark.
2358aec 
Signed-off-by: Xuehan <xxman@google.com>
@yuki-97
Copy link
Contributor

yuki-97 commented Jul 4, 2025

Thanks @xxman-google ! Approved.

@parthchadha @terrykong Can you take a review and enqueue it if it looks good to you?

@terrykong terrykong added this pull request to the merge queue Jul 4, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jul 4, 2025
@terrykong terrykong added this pull request to the merge queue Jul 8, 2025
Merged via the queue into NVIDIA-NeMo:main with commit 668fe48 Jul 9, 2025
13 of 14 checks passed
@xxman-google xxman-google deleted the xx/mmmlu branch July 11, 2025 16:36
jialei777 pushed a commit to jialei777/nemo-rl that referenced this pull request Jul 23, 2025
@xxman-google @jialei777
feat: add MMMLU eval benchmark. (NVIDIA-NeMo#596)
8711249 
Signed-off-by: Xuehan <xxman@google.com>
Signed-off-by: Jialei Chen <jialeic@google.com>
KiddoZhu pushed a commit that referenced this pull request Jul 28, 2025
@xxman-google @KiddoZhu
feat: add MMMLU eval benchmark. (#596)
39e69c0 
Signed-off-by: Xuehan <xxman@google.com>
FannYYW pushed a commit to xxman-google/NeMo-RL that referenced this pull request Aug 5, 2025
@xxman-google
feat: add MMMLU eval benchmark. (NVIDIA-NeMo#596)
2b3365d 
Signed-off-by: Xuehan <xxman@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@terrykong terrykong terrykong approved these changes

@yuki-97 yuki-97 yuki-97 approved these changes

Assignees
No one assigned
Labels
documentation Improvements or additions to documentation external
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@xxman-google @yuki-97 @terrykong @snowmanwwg