[Misc][Benchmarking] Add variable request-rate ("ramp-up") to the benchmarking client. #19423

dtransposed · 2025-06-10T13:35:46Z

Purpose

The goal of this PR is to add to the benchmark the possibility to model more realistic request traffic.

Already existing request traffic scenarios (unthrottled traffic, constant request rate, burstiness) are useful, but now, with this "ramp-up" feature, we can gradually increase the request rate throughout the benchmark.

This is especially useful in real-world scenarios, when we want to find out how far we can push the request rate, while staying under some predetermined latency budget.

Test

# in parallel...
vllm serve NousResearch/Hermes-3-Llama-3.2-3B --disable-log-requests --max_model_len 1028
# ...and
python benchmarks/benchmark_serving.py   --backend vllm   --model NousResearch/Hermes-3-Llama-3.2-3B   --endpoint /v1/completions   --dataset-name hf --dataset-path=likaixin/InstructCoder   --num-prompts 500 --ramp-up-strategy exponential --ramp-up-start-rps 2 --ramp-up-end-rps 50 --save-result --save-detailed

Judging by the Grafana metrics, I can see an exponentially increasing load on my service. In the resulting .json file, I additionally save information when each RPS value has been reached (so we can later match RPS with e.g. E2E latency values in Grafana and determine the maximum RPS given the latency budget).

"rps_change_events": [{"rps": 2, "timestamp": "2025-06-10T13:05:18.170180"}, {"rps": 3, "timestamp": "2025-06-10T13:05:44.447834"}, {"rps": 4, "timestamp": "2025-06-10T13:05:55.711857"}, {"rps": 5, "timestamp": "2025-06-10T13:06:03.862542"}, {"rps": 6, "timestamp": "2025-06-10T13:06:10.818332"}, {"rps": 7, "timestamp": "2025-06-10T13:06:14.033719"}, {"rps": 8, "timestamp": "2025-06-10T13:06:16.064900"}, ...,  {"rps": 39, "timestamp": "2025-06-10T13:06:31.786289"}, {"rps": 40, "timestamp": "2025-06-10T13:06:31.855277"}, {"rps": 41, "timestamp": "2025-06-10T13:06:31.930826"}, {"rps": 42, "timestamp": "2025-06-10T13:06:32.024664"}, {"rps": 43, "timestamp": "2025-06-10T13:06:32.198493"}, {"rps": 44, "timestamp": "2025-06-10T13:06:32.440968"}, {"rps": 45, "timestamp": "2025-06-10T13:06:32.456862"}, {"rps": 46, "timestamp": "2025-06-10T13:06:32.507610"}, {"rps": 47, "timestamp": "2025-06-10T13:06:32.649112"}, {"rps": 48, "timestamp": "2025-06-10T13:06:32.724322"}, {"rps": 49, "timestamp": "2025-06-10T13:06:32.759751"}, {"rps": 50, "timestamp": "2025-06-10T13:06:32.791630"}], "mean_ttft_ms": 105.42370502464473, "median_ttft_ms": 80.89415729045868, "std_ttft_ms": 62.44401755826986, "p99_ttft_ms": 305.91290817596, "mean_tpot_ms": 48.75638048924164, "median_tpot_ms": 41.22003553985934, "std_tpot_ms": 18.93813305829003, "p99_tpot_ms": 103.71412990459548, "mean_itl_ms": 46.6959536836888, "median_itl_ms": 37.55524102598429, "std_itl_ms": 25.223713419899962, "p99_itl_ms": 157.10909254848957}

github-actions · 2025-06-10T13:35:56Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist

Summary of Changes

Hello @dtransposed, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the benchmarking tools by adding support for ramping up the request rate during a benchmark run. This feature allows users to simulate increasing load scenarios, providing better insights into the system's performance under stress. The implementation includes new command-line arguments, dynamic request rate calculation, argument validation, and reporting of ramp-up details in the results.

Highlights

Ramp-Up Request Rate: Introduces the ability to ramp up the request rate over the duration of a benchmark run in benchmark_serving.py and vllm/benchmarks/serve.py. This is useful for stress testing and finding the maximum throughput.
New Arguments: Adds command-line arguments --ramp-up-strategy (linear or exponential), --ramp-up-start-rps, and --ramp-up-end-rps to control the ramp-up behavior.
Dynamic Request Rate: Modifies the get_request async generator to calculate the current request rate dynamically based on the chosen ramp-up strategy and the number of requests already sent.
Argument Validation: Adds validation logic to ensure ramp-up arguments are used correctly (e.g., --request-rate is not used with ramp-up, start/end RPS are provided and valid).
Output Reporting: Includes the ramp-up configuration and timestamped RPS change events in the benchmark's JSON output results.
Documentation: Updates the benchmarks/README.md to document the new ramp-up feature and its usage.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configureGemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a ramp-up request rate feature to the benchmarking scripts. It allows users to specify a linear or exponential increase in requests per second (RPS) over the duration of a benchmark run. The changes include updates to the get_request and benchmark functions, argument parsing, validation for the new parameters, and documentation in the README. My suggestions focus on improving the clarity of the help text for the new CLI arguments and code readability.

benchmarks/benchmark_serving.py

vllm/benchmarks/serve.py

ywang96

Thanks for the contribution! I think a ramp-up mode makes sense but I have left some comments. Please take a look!

benchmarks/benchmark_serving.py

vllm/benchmarks/serve.py

benchmarks/benchmark_serving.py

Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal>

dtransposed · 2025-06-11T11:20:16Z

@ywang96 thanks for swift review. Addressed all your comments.

Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal>

ywang96

Some final comments otherwise LGTM

vllm/benchmarks/serve.py

mergify · 2025-06-12T17:33:19Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @dtransposed.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal>

dtransposed · 2025-06-13T10:15:18Z

@ywang96 CI/CD seems to be stuck.

…ure/damian/rampup

Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal>

dtransposed · 2025-06-23T14:59:13Z

@ywang96 kind reminder.

ywang96

@dtransposed Sorry for the late response - I've enabled CI!

dtransposed · 2025-06-24T06:12:33Z

@ywang96 looks like the failing tests are unrelated, WDYT?

ywang96 · 2025-06-24T06:18:47Z

@ywang96 looks like the failing tests are unrelated, WDYT?

Yea - though we typically don't force merge. I've updated this branch with upstream main so the CI failures should be resolved!

dtransposed · 2025-06-24T08:37:17Z

@ywang96 this is sadly not the case :(

ywang96 · 2025-06-24T16:54:26Z

@ywang96 this is sadly not the case :(

Looks like HF was down - I retried the tests so hopefully they all go through...

…chmarking client. (vllm-project#19423) Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal> Co-authored-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal> Co-authored-by: Roger Wang <hey@rogerw.me>

…chmarking client. (vllm-project#19423) Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal> Co-authored-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal> Co-authored-by: Roger Wang <hey@rogerw.me> Signed-off-by: Will Eaton <weaton@redhat.com>

…chmarking client. (vllm-project#19423) Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal> Co-authored-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal> Co-authored-by: Roger Wang <hey@rogerw.me>

…chmarking client. (vllm-project#19423) Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal> Co-authored-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal> Co-authored-by: Roger Wang <hey@rogerw.me> Signed-off-by: avigny <47987522+avigny@users.noreply.github.com>

…chmarking client. (vllm-project#19423) Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal> Co-authored-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal> Co-authored-by: Roger Wang <hey@rogerw.me>

gemini-code-assist bot reviewed Jun 10, 2025

View reviewed changes

dtransposed changed the title ~~Feature/damian/rampup~~ Add variable request-rate ("ramp-up") to the benchmarking client. Jun 10, 2025

dtransposed changed the title ~~Add variable request-rate ("ramp-up") to the benchmarking client.~~ [Misc] Add variable request-rate ("ramp-up") to the benchmarking client. Jun 10, 2025

dtransposed marked this pull request as ready for review June 10, 2025 17:21

dtransposed changed the title ~~[Misc] Add variable request-rate ("ramp-up") to the benchmarking client.~~ [Misc][Benchmarking] Add variable request-rate ("ramp-up") to the benchmarking client. Jun 10, 2025

ywang96 reviewed Jun 11, 2025

View reviewed changes

dtransposed added 3 commits June 11, 2025 11:17

initial commit

a13aee6

Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal>

formatting

203e1b8

Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal>

addressing PR comments

bc79e5c

Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal>

dtransposed force-pushed the feature/damian/rampup branch from 125eacc to bc79e5c Compare June 11, 2025 11:17

dtransposed requested a review from ywang96 June 11, 2025 11:19

change filename

93cd0c2

Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal>

ywang96 approved these changes Jun 11, 2025

View reviewed changes

vllm/benchmarks/serve.py Show resolved Hide resolved

vllm/benchmarks/serve.py Outdated Show resolved Hide resolved

mergify bot added the needs-rebase label Jun 12, 2025

dtransposed added 2 commits June 13, 2025 07:25

final touchups

252b7ae

Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal>

Merge remote-tracking branch 'upstream/main' into feature/damian/rampup

0758e33

mergify bot removed the needs-rebase label Jun 13, 2025

additional testing

ce4fb53

Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal>

Merge branch 'main' into feature/damian/rampup

1f0cdc6

mergify bot added the performance Performance-related issues label Jun 22, 2025

dtransposed added 2 commits June 23, 2025 04:57

Merge branch 'main' of https://github.com/vllm-project/vllm into feat…

c9b4215

…ure/damian/rampup

Update benchmark serving and test requirements

54b085d

Signed-off-by: dtransposed <damian@damian-ml-machine.europe-west3-b.c.jetbrains-grazie.internal>

ywang96 added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 24, 2025

ywang96 approved these changes Jun 24, 2025

View reviewed changes

Merge remote-tracking branch 'upstream/main' into feature/damian/rampup

872b070

ywang96 enabled auto-merge (squash) June 24, 2025 06:18

ywang96 merged commit c635c5f into vllm-project:main Jun 24, 2025
70 of 71 checks passed

dtransposed deleted the feature/damian/rampup branch June 25, 2025 05:47

Uh oh!

[Misc][Benchmarking] Add variable request-rate ("ramp-up") to the benchmarking client. #19423

[Misc][Benchmarking] Add variable request-rate ("ramp-up") to the benchmarking client. #19423

Uh oh!

Conversation

dtransposed commented Jun 10, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test

Uh oh!

github-actions bot commented Jun 10, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ywang96 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dtransposed commented Jun 11, 2025

Uh oh!

ywang96 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Jun 12, 2025

Uh oh!

dtransposed commented Jun 13, 2025

Uh oh!

dtransposed commented Jun 23, 2025

Uh oh!

ywang96 left a comment

Choose a reason for hiding this comment

Uh oh!

dtransposed commented Jun 24, 2025

Uh oh!

ywang96 commented Jun 24, 2025

Uh oh!

dtransposed commented Jun 24, 2025

Uh oh!

ywang96 commented Jun 24, 2025

Uh oh!

Uh oh!

Uh oh!

dtransposed commented Jun 10, 2025 •

edited by github-actions bot

Loading