feat: supports evaluation of multiple-choice benchmarks #559

xxman-google · 2025-06-26T16:05:46Z

What does this PR do ?

Supports evaluation of multiple-choice benchmarks, such as GPQA/MMLU/etc.

Issues

N/A

Usage

Run evaluation on GPQA as follows:

uv run examples/run_eval.py --config examples/configs/evals/gpqa_eval.yaml

On GPQA, Qwen2.5-7B model scores 32.0 on average over 3 runs.

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

...

yuki-97

Awesome! Thanks for contributing, overall LGTM. Left some comments.

examples/configs/evals/gpqa_eval.yaml

examples/configs/evals/math_eval.yaml

examples/run_eval.py

xxman-google

Thanks for the review. All comments are addressed. PTAL agian.

yuki-97 · 2025-06-30T03:59:49Z

Thanks @xxman-google , all changes LGTM!

Could you also update the doc?

Add a list of currently supported datasets in docs/guides/eval.md.
Add brief examples of how to use supported datasets and local datasets at Run the Evaluation Script.

RL/docs/guides/eval.md

Line 38 in cfb803d

## Run the Evaluation Script

Update the outdated examples.

RL/docs/guides/eval.md

Lines 54 to 65 in cfb803d

    
           # Override specific config values via command line 
        
           # Example: Evaluation of DeepScaleR-1.5B-Preview on MATH-500 using 8 GPUs 
        
           #          Pass@1 accuracy averaged over 16 samples for each problem 
        
           uv run python examples/run_eval.py \ 
        
               generation.model_name=agentica-org/DeepScaleR-1.5B-Preview \ 
        
               generation.temperature=0.6 \ 
        
               generation.top_p=0.95 \ 
        
               generation.vllm_cfg.max_model_len=32768 \ 
        
               data.dataset_name=HuggingFaceH4/MATH-500 \ 
        
               data.dataset_key=test \ 
        
               eval.num_tests_per_prompt=16 \ 
        
               cluster.gpus_per_node=8

yuki-97 · 2025-06-30T04:03:45Z

Also, your PR appears to have a DCO error. It looks like this commit 42155d9 not signed.

xxman-google · 2025-06-30T05:29:37Z

Thanks @yuki-666 . Docs are updated and DCO erro is also fixed.

yuki-97 · 2025-06-30T05:57:49Z

@xxman-google Seems there is a conflict? Could you try merge main again to see if the conflict could be solved?

This branch has conflicts that must be resolved
Use the command line to resolve conflicts before continuing.

examples/configs/eval.yaml

Signed-off-by: Xuehan Xiong <xxman@google.com> Signed-off-by: Xuehan <xxman@google.com>

Signed-off-by: Xuehan Xiong xxman@google.com Signed-off-by: Xuehan Xiong <xxman@google.com> Signed-off-by: Xuehan <xxman@google.com>

Signed-off-by: Xuehan Xiong <xxman@google.com> Signed-off-by: Xuehan <xxman@google.com>

Signed-off-by: Xuehan Xiong xxman@google.com Signed-off-by: Xuehan Xiong <xxman@google.com> Signed-off-by: Xuehan <xxman@google.com>

Signed-off-by: Yuki Huang <yukih@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

…IA-NeMo#565) Signed-off-by: Parth Chadha <pchadha@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

Signed-off-by: Yuki Huang <yukih@nvidia.com> Signed-off-by: Parth Chadha <pchadha@nvidia.com> Co-authored-by: yuki <48991475+yuki-666@users.noreply.github.com> Signed-off-by: Xuehan <xxman@google.com>

…eMo#494) Signed-off-by: KiddoZhu <zhaochengz@nvidia.com> Signed-off-by: Sahil Jain <48468750+SahilJain314@users.noreply.github.com> Co-authored-by: Sahil Jain <48468750+SahilJain314@users.noreply.github.com> Signed-off-by: Xuehan <xxman@google.com>

Signed-off-by: Xuehan Xiong xxman@google.com Signed-off-by: Xuehan <xxman@google.com>

Signed-off-by: Xuehan Xiong <xxman@google.com> Signed-off-by: Xuehan <xxman@google.com>

…IDIA-NeMo#516) Signed-off-by: Luis Vega <vegaluisjose@users.noreply.github.com> Signed-off-by: Terry Kong <terrycurtiskong@gmail.com> Signed-off-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com> Co-authored-by: Terry Kong <terrycurtiskong@gmail.com> Co-authored-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com> Signed-off-by: Xuehan <xxman@google.com>

Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com> Signed-off-by: Parth Chadha <pchadha@nvidia.com> Co-authored-by: Parth Chadha <pchadha@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

…NeMo#570) Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

Signed-off-by: Yuki Huang <yukih@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

) Signed-off-by: Terry Kong <terryk@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

…req (NVIDIA-NeMo#574) Signed-off-by: Terry Kong <terryk@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

nemo_rl/evals/answer_parsing.py

Signed-off-by: Xuehan Xiong <xxman@google.com>

Signed-off-by: Shun Kiyono <shun.kiyono@sbintuitions.co.jp>

…-NeMo#583) Signed-off-by: Wei Du <wedu@nvidia.com>

Signed-off-by: Yuki Huang <yukih@nvidia.com>

Signed-off-by: ashors1 <ashors@nvidia.com>

Signed-off-by: Sahil Jain <sahilj@nvidia.com>

Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: Sahil Jain <48468750+SahilJain314@users.noreply.github.com> Co-authored-by: Sahil Jain <48468750+SahilJain314@users.noreply.github.com>

Signed-off-by: Atsunori Fujita <afujita@nvidia.com>

Signed-off-by: Parth Chadha <pchadha@nvidia.com>

Signed-off-by: Yuki Huang <yukih@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

Signed-off-by: Anna Shors <ashors@nvidia.com> Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

Signed-off-by: Xuehan <xxman@google.com>

) Signed-off-by: Xuehan Xiong <xxman@google.com> Signed-off-by: Xuehan <xxman@google.com> Signed-off-by: Xuehan Xiong xxman@google.com Signed-off-by: Yuki Huang <yukih@nvidia.com> Signed-off-by: Parth Chadha <pchadha@nvidia.com> Signed-off-by: KiddoZhu <zhaochengz@nvidia.com> Signed-off-by: Sahil Jain <48468750+SahilJain314@users.noreply.github.com> Signed-off-by: Luis Vega <vegaluisjose@users.noreply.github.com> Signed-off-by: Terry Kong <terrycurtiskong@gmail.com> Signed-off-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com> Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: Terry Kong <terryk@nvidia.com> Signed-off-by: Anna Shors <ashors@nvidia.com> Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com> Signed-off-by: Sahil Jain <sahilj@nvidia.com> Signed-off-by: Yuki Huang <scsse_hqh2016@126.com> Signed-off-by: Shun Kiyono <shun.kiyono@sbintuitions.co.jp> Signed-off-by: Wei Du <wedu@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: Atsunori Fujita <afujita@nvidia.com> Co-authored-by: yuki <48991475+yuki-666@users.noreply.github.com> Co-authored-by: Parth Chadha <pchadha@nvidia.com> Co-authored-by: Zhaocheng Zhu <zhaochengz@nvidia.com> Co-authored-by: Sahil Jain <48468750+SahilJain314@users.noreply.github.com> Co-authored-by: Luis Vega <vegaluisjose@users.noreply.github.com> Co-authored-by: Terry Kong <terrycurtiskong@gmail.com> Co-authored-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com> Co-authored-by: Yi-Fu Wu <yifu.wu@gmail.com> Co-authored-by: Anna Shors <ashors@nvidia.com> Co-authored-by: Dheeraj Peri <peri.dheeraj@gmail.com> Co-authored-by: Yuki Huang <scsse_hqh2016@126.com> Co-authored-by: Shun Kiyono <shunk52@gmail.com> Co-authored-by: Wei Du <wedu@nvidia.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: atfujita <40932835+AtsunoriFujita@users.noreply.github.com>

) Signed-off-by: Xuehan Xiong <xxman@google.com> Signed-off-by: Xuehan <xxman@google.com> Signed-off-by: Xuehan Xiong xxman@google.com Signed-off-by: Yuki Huang <yukih@nvidia.com> Signed-off-by: Parth Chadha <pchadha@nvidia.com> Signed-off-by: KiddoZhu <zhaochengz@nvidia.com> Signed-off-by: Sahil Jain <48468750+SahilJain314@users.noreply.github.com> Signed-off-by: Luis Vega <vegaluisjose@users.noreply.github.com> Signed-off-by: Terry Kong <terrycurtiskong@gmail.com> Signed-off-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com> Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: Terry Kong <terryk@nvidia.com> Signed-off-by: Anna Shors <ashors@nvidia.com> Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com> Signed-off-by: Sahil Jain <sahilj@nvidia.com> Signed-off-by: Yuki Huang <scsse_hqh2016@126.com> Signed-off-by: Shun Kiyono <shun.kiyono@sbintuitions.co.jp> Signed-off-by: Wei Du <wedu@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: Atsunori Fujita <afujita@nvidia.com> Co-authored-by: yuki <48991475+yuki-666@users.noreply.github.com> Co-authored-by: Parth Chadha <pchadha@nvidia.com> Co-authored-by: Zhaocheng Zhu <zhaochengz@nvidia.com> Co-authored-by: Sahil Jain <48468750+SahilJain314@users.noreply.github.com> Co-authored-by: Luis Vega <vegaluisjose@users.noreply.github.com> Co-authored-by: Terry Kong <terrycurtiskong@gmail.com> Co-authored-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com> Co-authored-by: Yi-Fu Wu <yifu.wu@gmail.com> Co-authored-by: Anna Shors <ashors@nvidia.com> Co-authored-by: Dheeraj Peri <peri.dheeraj@gmail.com> Co-authored-by: Yuki Huang <scsse_hqh2016@126.com> Co-authored-by: Shun Kiyono <shunk52@gmail.com> Co-authored-by: Wei Du <wedu@nvidia.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: atfujita <40932835+AtsunoriFujita@users.noreply.github.com> Signed-off-by: Jialei Chen <jialeic@google.com>

Signed-off-by: Xuehan Xiong <xxman@google.com> Signed-off-by: Xuehan <xxman@google.com> Signed-off-by: Xuehan Xiong xxman@google.com Signed-off-by: Yuki Huang <yukih@nvidia.com> Signed-off-by: Parth Chadha <pchadha@nvidia.com> Signed-off-by: KiddoZhu <zhaochengz@nvidia.com> Signed-off-by: Sahil Jain <48468750+SahilJain314@users.noreply.github.com> Signed-off-by: Luis Vega <vegaluisjose@users.noreply.github.com> Signed-off-by: Terry Kong <terrycurtiskong@gmail.com> Signed-off-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com> Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: Terry Kong <terryk@nvidia.com> Signed-off-by: Anna Shors <ashors@nvidia.com> Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com> Signed-off-by: Sahil Jain <sahilj@nvidia.com> Signed-off-by: Yuki Huang <scsse_hqh2016@126.com> Signed-off-by: Shun Kiyono <shun.kiyono@sbintuitions.co.jp> Signed-off-by: Wei Du <wedu@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: Atsunori Fujita <afujita@nvidia.com> Co-authored-by: yuki <48991475+yuki-666@users.noreply.github.com> Co-authored-by: Parth Chadha <pchadha@nvidia.com> Co-authored-by: Zhaocheng Zhu <zhaochengz@nvidia.com> Co-authored-by: Sahil Jain <48468750+SahilJain314@users.noreply.github.com> Co-authored-by: Luis Vega <vegaluisjose@users.noreply.github.com> Co-authored-by: Terry Kong <terrycurtiskong@gmail.com> Co-authored-by: Luis Vega <2478335+vegaluisjose@users.noreply.github.com> Co-authored-by: Yi-Fu Wu <yifu.wu@gmail.com> Co-authored-by: Anna Shors <ashors@nvidia.com> Co-authored-by: Dheeraj Peri <peri.dheeraj@gmail.com> Co-authored-by: Yuki Huang <scsse_hqh2016@126.com> Co-authored-by: Shun Kiyono <shunk52@gmail.com> Co-authored-by: Wei Du <wedu@nvidia.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: atfujita <40932835+AtsunoriFujita@users.noreply.github.com>

terrykong requested a review from yuki-97 June 26, 2025 16:25

yuki-97 reviewed Jun 27, 2025

View reviewed changes

github-actions bot added the documentation Improvements or additions to documentation label Jun 27, 2025

xxman-google commented Jun 27, 2025

View reviewed changes

xxman-google force-pushed the xx/new_eval branch from 57ccd9b to dbf4800 Compare June 30, 2025 05:26

xxman-google and others added 21 commits June 30, 2025 06:06

Adds multiple choice eval datasets.

405acc2

Signed-off-by: Xuehan Xiong <xxman@google.com> Signed-off-by: Xuehan <xxman@google.com>

Add a verify worker for multiple-choice problems.

67aae53

Signed-off-by: Xuehan Xiong <xxman@google.com> Signed-off-by: Xuehan <xxman@google.com>

add prompts for MMLU and GPQA.

4134fcb

Signed-off-by: Xuehan Xiong <xxman@google.com> Signed-off-by: Xuehan <xxman@google.com>

modifies eval script to support multiple-choice questions.

0ca559f

Signed-off-by: Xuehan Xiong <xxman@google.com> Signed-off-by: Xuehan <xxman@google.com>

add eval config files.

2163cbf

Signed-off-by: Xuehan Xiong <xxman@google.com> Signed-off-by: Xuehan <xxman@google.com>

add unit tests.

d9dd544

Signed-off-by: Xuehan Xiong xxman@google.com Signed-off-by: Xuehan Xiong <xxman@google.com> Signed-off-by: Xuehan <xxman@google.com>

add AIME 2024 dataset.

11a1de5

Signed-off-by: Xuehan Xiong <xxman@google.com> Signed-off-by: Xuehan <xxman@google.com>

add GPQA main version.

4da0c43

Signed-off-by: Xuehan Xiong xxman@google.com Signed-off-by: Xuehan Xiong <xxman@google.com> Signed-off-by: Xuehan <xxman@google.com>

fix: remove reference_model_buffers in fsdp2 (NVIDIA-NeMo#558)

5870e46

Signed-off-by: Yuki Huang <yukih@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

fix: Add assertion if async is disabled when using pp with vllm (NVID…

79690a1

…IA-NeMo#565) Signed-off-by: Parth Chadha <pchadha@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

fix: remove visualization code (NVIDIA-NeMo#566)

940049f

Signed-off-by: Yuki Huang <yukih@nvidia.com> Signed-off-by: Parth Chadha <pchadha@nvidia.com> Co-authored-by: yuki <48991475+yuki-666@users.noreply.github.com> Signed-off-by: Xuehan <xxman@google.com>

add GPQA main version.

9c59083

Signed-off-by: Xuehan Xiong xxman@google.com Signed-off-by: Xuehan <xxman@google.com>

updates doc.

628ef2d

Signed-off-by: Xuehan Xiong <xxman@google.com> Signed-off-by: Xuehan <xxman@google.com>

feat: Log code in wandb (NVIDIA-NeMo#175)

f6b948d

Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com> Signed-off-by: Parth Chadha <pchadha@nvidia.com> Co-authored-by: Parth Chadha <pchadha@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

fix: add dynamic_batching key to SFT OpenMathInstruct config (NVIDIA-…

4265fed

…NeMo#570) Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

feat: support async in non-colocated (NVIDIA-NeMo#523)

7c8367d

Signed-off-by: Yuki Huang <yukih@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

fix: correct mcore dtype + assertion on activation_func (NVIDIA-NeMo#572

d0dca5b

) Signed-off-by: Terry Kong <terryk@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

fix: move core ray port from 6379 -> 54258 to reduce port collision f…

e257d88

…req (NVIDIA-NeMo#574) Signed-off-by: Terry Kong <terryk@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

fix: fix overlap param gather (NVIDIA-NeMo#561)

c27ff44

Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

parthchadha enabled auto-merge July 2, 2025 18:00

parthchadha reviewed Jul 2, 2025

View reviewed changes

nemo_rl/evals/answer_parsing.py Show resolved Hide resolved

add missing copyright statements.

17fe405

Signed-off-by: Xuehan Xiong <xxman@google.com>

auto-merge was automatically disabled July 2, 2025 19:57
Head branch was pushed to by a user without write access

butsugiri and others added 10 commits July 2, 2025 21:20

docs: Add missing arguments to DeepScaler evaluation (NVIDIA-NeMo#502)

cf828d6

Signed-off-by: Shun Kiyono <shun.kiyono@sbintuitions.co.jp>

fix: prevent divisible error by dropping last batch in loader (NVIDIA…

01c3840

…-NeMo#583) Signed-off-by: Wei Du <wedu@nvidia.com>

feat: improve worker group args/kwargs (NVIDIA-NeMo#539)

658437d

Signed-off-by: Yuki Huang <yukih@nvidia.com>

fix: update gemma3 prefix (NVIDIA-NeMo#585)

2eb0301

Signed-off-by: ashors1 <ashors@nvidia.com>

fix: Added copyright to functest (NVIDIA-NeMo#584)

bc234a3

Signed-off-by: Sahil Jain <sahilj@nvidia.com>

chore: Update github url after org transfer (NVIDIA-NeMo#512)

2d876de

Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: Sahil Jain <48468750+SahilJain314@users.noreply.github.com> Co-authored-by: Sahil Jain <48468750+SahilJain314@users.noreply.github.com>

feat: add OpenAI format dataset for SFT (NVIDIA-NeMo#485)

ddac07c

Signed-off-by: Atsunori Fujita <afujita@nvidia.com>

fix: load HF model only on rank 0 (NVIDIA-NeMo#544)

283074a

Signed-off-by: Parth Chadha <pchadha@nvidia.com>

feat: support async in non-colocated (NVIDIA-NeMo#523)

e78af38

Signed-off-by: Yuki Huang <yukih@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

feat: Add megatron to hf converter (NVIDIA-NeMo#555)

4cd4568

Signed-off-by: Anna Shors <ashors@nvidia.com> Signed-off-by: ashors1 <ashors@nvidia.com> Signed-off-by: Xuehan <xxman@google.com>

github-actions bot added the CI Relating to CI label Jul 2, 2025

Merge branch 'main' into xx/new_eval

c44efc0

Signed-off-by: Xuehan <xxman@google.com>

github-actions bot removed the CI Relating to CI label Jul 2, 2025

parthchadha enabled auto-merge July 2, 2025 21:39

parthchadha approved these changes Jul 2, 2025

View reviewed changes

parthchadha added this pull request to the merge queue Jul 2, 2025

github-merge-queue bot removed this pull request from the merge queue due to a conflict with the base branch Jul 2, 2025

parthchadha added this pull request to the merge queue Jul 2, 2025

Merged via the queue into NVIDIA-NeMo:main with commit 3f6d52f Jul 3, 2025
13 of 14 checks passed

xxman-google deleted the xx/new_eval branch July 3, 2025 02:59

snowmanwwg added the external label Jul 20, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: supports evaluation of multiple-choice benchmarks #559

feat: supports evaluation of multiple-choice benchmarks #559

Uh oh!

xxman-google commented Jun 26, 2025 •

edited

Loading

Uh oh!

yuki-97 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xxman-google left a comment

Uh oh!

yuki-97 commented Jun 30, 2025

Uh oh!

yuki-97 commented Jun 30, 2025

Uh oh!

xxman-google commented Jun 30, 2025

Uh oh!

yuki-97 commented Jun 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

feat: supports evaluation of multiple-choice benchmarks #559

feat: supports evaluation of multiple-choice benchmarks #559

Uh oh!

Conversation

xxman-google commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Issues

Usage

Before your PR is "Ready for review"

Additional Information

Uh oh!

yuki-97 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xxman-google left a comment

Choose a reason for hiding this comment

Uh oh!

yuki-97 commented Jun 30, 2025

Uh oh!

yuki-97 commented Jun 30, 2025

Uh oh!

xxman-google commented Jun 30, 2025

Uh oh!

yuki-97 commented Jun 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xxman-google commented Jun 26, 2025 •

edited

Loading