Skip to content

[feat] Add math eval to CI #2652

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Dec 30, 2024
Merged

[feat] Add math eval to CI #2652

merged 1 commit into from
Dec 30, 2024

Conversation

XiaotongJiang
Copy link
Collaborator

@XiaotongJiang XiaotongJiang commented Dec 30, 2024 •
edited
Loading

Issue #2504

  • Compared SGL and vLLM's math eval score to address concerns on SGL being constant low on this eval (res linked below)
  • Add math eval to CI

Setup:

  • Prompt: simple eval from openai
  • Grading Model: gemini-1.5-flash
  • Temperature 0.0
inference engine Model # Examples score model official score
vllm llama 3.2 1b 500 0.248 0.306
sgl llama 3.2 1b 500 0.248 0.306
vllm llama 3.2 3b 500 0.476 0.48
sgl llama 3.2 3b 500 0.484 0.48
vllm llama 3.1 8b 500 0.50 0.519
sgl llama 3.1 8b 500 0.496 0.519
sgl llama 3.1 8b 65 0.492 0.519

@XiaotongJiang XiaotongJiang changed the title Add math eval to CI [Issue 2504] Add math eval to CI Dec 30, 2024
@zhyncs zhyncs changed the title [Issue 2504] Add math eval to CI [feat] Add math eval to CI Dec 30, 2024
@zhyncs zhyncs merged commit a11f8d5 into sgl-project:main Dec 30, 2024
@zhyncs
Copy link
Member

zhyncs commented Dec 30, 2024

Thanks!!

@zhaochenyang20
Copy link
Collaborator

@XiaotongJiang @zhyncs @merrymercy @shuaills I am talking with XIaotong and will take a look on this. Also, thanks so much. this having been suspended for so long 😂

merrymercy added a commit that referenced this pull request Dec 30, 2024
@merrymercy
Revert "[feat] Add math eval to CI (#2652)"
1deafae 
This reverts commit a11f8d5.
@merrymercy merrymercy mentioned this pull request Dec 30, 2024
Merged
@XiaotongJiang
Copy link
Collaborator Author

XiaotongJiang commented Dec 30, 2024 •
edited
Loading

sorry for the disruption

AssertionError: 0.505 not greater than or equal to 0.509

need to loose this assertion a bit

@XiaotongJiang XiaotongJiang mentioned this pull request Dec 30, 2024
Merged
XiaotongJiang added a commit to XiaotongJiang/sglang that referenced this pull request Jan 3, 2025
@XiaotongJiang
[feat] Add math eval to CI (sgl-project#2652)
15fdc6e
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
@XiaotongJiang
[feat] Add math eval to CI (sgl-project#2652)
1d1d202
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhyncs zhyncs zhyncs approved these changes

@merrymercy merrymercy Awaiting requested review from merrymercy

@Ying1123 Ying1123 Awaiting requested review from Ying1123

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@XiaotongJiang @zhyncs @zhaochenyang20