[Benchmark][Bug] Fix multiple bugs in bench and add args to spec_decode offline #20083

ekagra-ranjan · 2025-06-25T16:52:44Z

add random seed in random dataset. This is needed if we want to simulate prompt caching and ttft analysis
fix random dataset bug:

vllm/vllm/benchmarks/datasets.py

Line 380 in 0d06b53

prompt, add_special_tokens=False)[:input_lens[i]]

Currently it picks 1st input len num of tokens which means if prefix len is 32k and input len is 32k then it will pick 1st 32k tokens which is all the prefix len instead of all 64k. This leads to ~98% prefix cache hit. Similar bug was fixed in [Benchmark][Bugfix] Fix Dataset Length Calculation #19868 but the original intention of RandomDataset is to have prefix and input len independent of each other so that context length can be sampled from [input_len * (1 - range_ratio), input_len * (1 + range_ratio)] as per this
add --backend in serve.py for cases when sampling params are passed
add model config, eagle dir in offline inference

github-actions · 2025-06-25T16:52:52Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist

Summary of Changes

Hello @ekagra-ranjan, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request primarily focuses on enhancing the robustness and configurability of the benchmarking and offline inference systems. It addresses critical bugs in the random dataset generation for benchmarks, ensuring more accurate simulation of prompt caching and TTFT analysis. Additionally, it introduces new command-line arguments to provide greater control over model paths and speculative decoding configurations in offline inference, and adds a backend selection option for the benchmark server.

Highlights

Benchmark Dataset Bug Fix: Corrected a bug in the RandomDataset generation where the total_input_len was incorrectly calculated, leading to an overestimation of prefix cache hits. The fix ensures that the encoded sequence truncation correctly accounts for both prefix and input lengths.
Benchmark Reproducibility: Implemented random seed initialization for RandomDataset using both random and numpy to ensure reproducible benchmark results, especially important for simulating prompt caching and TTFT analysis.
Offline Inference Configurability: Added command-line arguments (--model-dir, --eagle-dir, --max-model-len) to spec_decode.py, allowing users to specify the main model, Eagle draft model, and maximum model length directly, enhancing flexibility for offline inference benchmarks.
Benchmark Server Backend Option: Introduced a --backend argument to vllm/benchmarks/serve.py, enabling explicit selection of the backend (e.g., vllm, tgi, hf) for benchmark runs, which is useful when passing specific sampling parameters.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

The pull request addresses a bug in the random dataset generation, adds arguments for backend selection in serve.py, and includes model configuration options for offline inference. The changes improve the accuracy of benchmark simulations and provide more flexibility in specifying model directories and configurations.

benchmarks/benchmark_dataset.py

vllm/benchmarks/datasets.py

ekagra-ranjan · 2025-06-25T17:36:27Z

cc: @robertgshaw2-redhat since a change on similar note was made in #19868

ywang96

Thanks for the bug fixes!

ekagra-ranjan · 2025-06-25T18:59:26Z

Thanks for the quick review!

…de offline (vllm-project#20083)

…de offline (vllm-project#20083) Signed-off-by: Will Eaton <weaton@redhat.com>

…de offline (vllm-project#20083)

…de offline (vllm-project#20083) Signed-off-by: avigny <47987522+avigny@users.noreply.github.com>

…de offline (vllm-project#20083)

ekagra-ranjan added 5 commits June 25, 2025 16:24

add seed

3d71b94

add backend

e1c9684

fix prefix and input len len

c56c2af

fix prefix and input len len

864aeeb

qol spec decode offline

3a61da4

gemini-code-assist bot reviewed Jun 25, 2025

View reviewed changes

mergify bot added documentation Improvements or additions to documentation performance Performance-related issues labels Jun 25, 2025

gemini-code-assist bot reviewed Jun 25, 2025

View reviewed changes

benchmarks/benchmark_dataset.py Outdated Show resolved Hide resolved

vllm/benchmarks/datasets.py Show resolved Hide resolved

ekagra-ranjan added 2 commits June 25, 2025 17:00

lint

92ead11

fix gemini suggestion

227e90a

ywang96 approved these changes Jun 25, 2025

View reviewed changes

ywang96 added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 25, 2025

ekagra-ranjan requested a review from ywang96 June 26, 2025 04:16

ywang96 merged commit 9502c38 into vllm-project:main Jun 26, 2025
78 checks passed

m-misiura pushed a commit to m-misiura/vllm that referenced this pull request Jun 26, 2025

[Benchmark][Bug] Fix multiple bugs in bench and add args to spec_deco…

965bb47

…de offline (vllm-project#20083)

gmarinho2 pushed a commit to gmarinho2/vllm that referenced this pull request Jun 26, 2025

[Benchmark][Bug] Fix multiple bugs in bench and add args to spec_deco…

91caa99

…de offline (vllm-project#20083)

xjpang pushed a commit to xjpang/vllm that referenced this pull request Jun 30, 2025

[Benchmark][Bug] Fix multiple bugs in bench and add args to spec_deco…

bab84da

…de offline (vllm-project#20083)

wseaton pushed a commit to wseaton/vllm that referenced this pull request Jun 30, 2025

[Benchmark][Bug] Fix multiple bugs in bench and add args to spec_deco…

ea89974

…de offline (vllm-project#20083) Signed-off-by: Will Eaton <weaton@redhat.com>

wseaton pushed a commit to wseaton/vllm that referenced this pull request Jun 30, 2025

[Benchmark][Bug] Fix multiple bugs in bench and add args to spec_deco…

4a30252

…de offline (vllm-project#20083)

wwl2755-google pushed a commit to wwl2755-google/vllm that referenced this pull request Jul 1, 2025

[Benchmark][Bug] Fix multiple bugs in bench and add args to spec_deco…

e434e43

…de offline (vllm-project#20083)

avigny pushed a commit to avigny/vllm that referenced this pull request Jul 31, 2025

[Benchmark][Bug] Fix multiple bugs in bench and add args to spec_deco…

4f13b41

…de offline (vllm-project#20083) Signed-off-by: avigny <47987522+avigny@users.noreply.github.com>

tanruixiang mentioned this pull request Aug 17, 2025

[Misc] Fix the benchmark's README and improve the error messages for the benchmark's argument checks #22654

Merged

4 tasks

googlercolin pushed a commit to googlercolin/vllm that referenced this pull request Aug 29, 2025

[Benchmark][Bug] Fix multiple bugs in bench and add args to spec_deco…

b3ddb68

…de offline (vllm-project#20083)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Benchmark][Bug] Fix multiple bugs in bench and add args to spec_decode offline #20083

[Benchmark][Bug] Fix multiple bugs in bench and add args to spec_decode offline #20083

Uh oh!

ekagra-ranjan commented Jun 25, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Jun 25, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

Uh oh!

ekagra-ranjan commented Jun 25, 2025 •

edited

Loading

Uh oh!

ywang96 left a comment

Uh oh!

ekagra-ranjan commented Jun 25, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[Benchmark][Bug] Fix multiple bugs in bench and add args to spec_decode offline #20083

[Benchmark][Bug] Fix multiple bugs in bench and add args to spec_decode offline #20083

Uh oh!

Conversation

ekagra-ranjan commented Jun 25, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 25, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

ekagra-ranjan commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ywang96 left a comment

Choose a reason for hiding this comment

Uh oh!

ekagra-ranjan commented Jun 25, 2025

Uh oh!

Uh oh!

Uh oh!

ekagra-ranjan commented Jun 25, 2025 •

edited by github-actions bot

Loading

ekagra-ranjan commented Jun 25, 2025 •

edited

Loading