[V1][Spec Decoding] Add num_drafts and num_accepted_tokens_per_position metrics #16665

markmc · 2025-04-15T14:08:06Z

As discussed in the design doc.

Record num_drafts so that it can be used to calculate mean acceptance length.

Record num_accepted_tokens per draft position and log these as per-position acceptance probabilities. In Prometheus, use a Counter with a position label - somewhat similar to how a Histogram works, but acknowledging token positions are discrete values and we're not bucketing continuous values.

github-actions · 2025-04-15T14:08:17Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

markmc · 2025-04-15T14:11:29Z

FYI @luyuzhe111 @WoosukKwon @LiuXiaoxuanPKU @ekagra-ranjan

vllm/v1/metrics/loggers.py

wwl2755

I think the PR is comprehensive and well-structured!

Just wondering should we have some comprehensive testing covers edge cases, because a lot of upcoming debuging work would depend on the metrics? Maybe we could make them in another PR since we could include all unit tests related to the V1 spec decoding metrics as they are missing right now.

markmc · 2025-04-16T13:51:42Z

I think the PR is comprehensive and well-structured!

Thanks!

Just wondering should we have some comprehensive testing covers edge cases, because a lot of upcoming debuging work would depend on the metrics? Maybe we could make them in another PR since we could include all unit tests related to the V1 spec decoding metrics as they are missing right now.

Yes, unit tests would be helpful 👍

markmc · 2025-04-16T13:55:30Z

For the record, I just tried:

VLLM_LOGGING_LEVEL=DEBUG VLLM_USE_V1=1 VLLM_ENABLE_V1_MULTIPROCESSING=1 vllm serve meta-llama/Meta-Llama-3-8B-Instruct --speculative-config '{"method": "eagle", "model" : "abhigoyal/EAGLE-LLaMA3-Instruct-8B-vllm", "num_speculative_tokens": 4}'

(some of those env vars are probably not necessary)

with

python ./benchmarks/benchmark_serving.py --model meta-llama/Meta-Llama-3-8B-Instruct --tokenizer meta-llama/Meta-Llama-3-8B-Instruct --endpoint /v1/completions --dataset-name sharegpt --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json --request-rate 3.0 --num-prompts 200

and got

INFO 04-16 09:49:04 [metrics.py:82] SpecDecoding metrics: Draft acceptance rate: 28.8%, Mean acceptance length: 1.15, Accepted: 686 tokens, Drafted: 2384 tokens, Per-position acceptance probabilities: 0.576, 0.327, 0.159, 0.089
INFO 04-16 09:49:14 [metrics.py:82] SpecDecoding metrics: Draft acceptance rate: 40.6%, Mean acceptance length: 1.62, Accepted: 2951 tokens, Drafted: 7264 tokens, Per-position acceptance probabilities: 0.695, 0.442, 0.286, 0.202
INFO 04-16 09:49:24 [metrics.py:82] SpecDecoding metrics: Draft acceptance rate: 30.5%, Mean acceptance length: 1.22, Accepted: 3345 tokens, Drafted: 10984 tokens, Per-position acceptance probabilities: 0.588, 0.336, 0.185, 0.109
INFO 04-16 09:49:34 [metrics.py:82] SpecDecoding metrics: Draft acceptance rate: 30.9%, Mean acceptance length: 1.24, Accepted: 4029 tokens, Drafted: 13024 tokens, Per-position acceptance probabilities: 0.600, 0.333, 0.189, 0.115
INFO 04-16 09:49:44 [metrics.py:82] SpecDecoding metrics: Draft acceptance rate: 30.5%, Mean acceptance length: 1.22, Accepted: 3682 tokens, Drafted: 12084 tokens, Per-position acceptance probabilities: 0.575, 0.335, 0.189, 0.119
INFO 04-16 09:49:54 [metrics.py:82] SpecDecoding metrics: Draft acceptance rate: 32.1%, Mean acceptance length: 1.28, Accepted: 2969 tokens, Drafted: 9260 tokens, Per-position acceptance probabilities: 0.581, 0.346, 0.216, 0.139
INFO 04-16 09:50:04 [metrics.py:82] SpecDecoding metrics: Draft acceptance rate: 29.4%, Mean acceptance length: 1.18, Accepted: 3545 tokens, Drafted: 12048 tokens, Per-position acceptance probabilities: 0.579, 0.319, 0.176, 0.103

It would be nice to do this online inferencing run with an equivalent setup to @ekagra-ranjan and @luyuzhe111 offline inferencing setup - e.g. mt_bench and yuhuili/EAGLE-LLaMA3-Instruct-8B - because the mean acceptance length looks quite different?

markmc · 2025-04-17T15:12:50Z

Just wondering should we have some comprehensive testing covers edge cases, because a lot of upcoming debuging work would depend on the metrics? Maybe we could make them in another PR since we could include all unit tests related to the V1 spec decoding metrics as they are missing right now.

Yes, unit tests would be helpful 👍

Ok, I extended the existing scheduler unit test for SpecDecodingStats - at least that's some coverage 👍

mergify · 2025-04-17T20:33:56Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @markmc.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify · 2025-04-22T03:50:18Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @markmc.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

WoosukKwon · 2025-04-24T05:14:53Z

@markmc Thanks for the PR!

One thing I'm confused about is the per-position acceptance probability. I've observed in the log that
mean acceptance length == sum(p1, p2, p3) when num_spec_tokens is 3.
Is this expected? I was expecting that the mean acceptance length would be 1 * p1 + 2 * p2 + 3 * p3.

INFO 04-23 22:03:06 [metrics.py:82] SpecDecoding metrics: Draft acceptance rate: 29.3%, Mean acceptance length: 0.88, Accepted: 754 tokens, Drafted: 2574 tokens, Per-position acceptance probabilities: 0.535, 0.233, 0.111

markmc · 2025-04-24T06:52:20Z

One thing I'm confused about is the per-position acceptance probability. I've observed in the log that mean acceptance length == sum(p1, p2, p3) when num_spec_tokens is 3. Is this expected? I was expecting that the mean acceptance length would be 1 * p1 + 2 * p2 + 3 * p3.

Wonderful question! I hadn't considered the relationship at all

1 * p1 + 2 * p2 + 3 * p3 would be correct if e.g. p2 was "the probability that a draft has exactly 2 tokens accepted", but it is actually "the probability that a draft has 2 or 3 tokens accepted".

Simple worked example

num_spec_tokens = 3

drafts:
- #1: 3 accepted
- #2: 1 accepted
- #3: 2 accepted
- #4: 2 accepted
- #5: 1 accepted

observe:
- num_drafts = 5
- num_draft_tokens = 15
- num_accepted_tokens = 9
- accepted_tokens_per_pos = [5, 3, 1]

compute:
- acceptance_rate = 9/15 = 0.6
- mean_acceptance_length = 1.8
- acceptance_probs_per_pos = [1.0, 0.6, 0.2]

HTH!

tests/v1/core/test_scheduler.py

vllm/v1/core/sched/scheduler.py

tests/v1/core/test_scheduler.py

vllm/v1/spec_decode/metrics.py

WoosukKwon

LGTM. Thanks for addressing my comments!

vllm/v1/spec_decode/metrics.py

Signed-off-by: Mark McLoughlin <markmc@redhat.com>

These must have been added in an earlier version of the PR where the SpecDecodingStats instance persisted. Signed-off-by: Mark McLoughlin <markmc@redhat.com>

Record num_drafts so that it can be used to calculate mean acceptance length. Rename SpecDecodingStats.observe() to observe_draft() to make it clear that each call records stats on a single draft. Signed-off-by: Mark McLoughlin <markmc@redhat.com>

Log these as per-position acceptance rates. In Prometheus, use a Counter with a position label - somewhat similar to how a Histogram works, but acknowledging token positions are discrete values and we're not bucketing continous values. Signed-off-by: Mark McLoughlin <markmc@redhat.com>

…d_tokens_per_pos Signed-off-by: Mark McLoughlin <markmc@redhat.com>

* [BugFix] Remove default multiproc executor `collective_rpc` timeout (vllm-project#17000) Signed-off-by: Nick Hill <nhill@redhat.com> * [Core][V1][TPU] Enable structured decoding on TPU V1 (vllm-project#16499) Signed-off-by: Chenyaaang <chenyangli@google.com> * [Bugfix] validate urls object for multimodal content parts (vllm-project#16990) Signed-off-by: Guillaume Calmettes <gcalmettes@scaleway.com> * add Dockerfile build vllm against torch nightly (vllm-project#16936) Signed-off-by: Yang Wang <elainewy@meta.com> * [Kernel][ROCM] Upstream prefix prefill speed up for vLLM V1 (vllm-project#13305) Signed-off-by: Sage Moore <sage@neuralmagic.com> Signed-off-by: root <root@banff-cyxtera-s73-5.ctr.dcgpu> Signed-off-by: Aleksandr Malyshev <maleksan@amd.com> Signed-off-by: root <root@banff-cyxtera-s65-4.amd.com> Signed-off-by: maleksan85 <maleksan@amd.com> Signed-off-by: <> Co-authored-by: Sage Moore <sage@neuralmagic.com> Co-authored-by: root <root@banff-cyxtera-s73-5.ctr.dcgpu> Co-authored-by: Aleksandr Malyshev <maleksan@amd.com> Co-authored-by: qli88 <qiang.li2@amd.com> Co-authored-by: root <root@banff-cyxtera-s65-4.amd.com> * [V1][DP] More robust DP/EP dummy request coordination (vllm-project#16277) Signed-off-by: Nick Hill <nhill@redhat.com> * [BugFix] Revert ROCm Custom Paged Attention Env Flag Check (vllm-project#17022) Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com> * Revert "[Misc] Add S3 environment variables for better support of MinIO." (vllm-project#17021) * [misc] tune some env vars for GB200 (vllm-project#16992) Signed-off-by: youkaichao <youkaichao@gmail.com> * [INTEL-HPU][v0] Port delayed sampling to upstream (vllm-project#16949) Signed-off-by: Michal Adamczyk <michal.adamczyk@intel.com> Signed-off-by: Chendi Xue <chendi.xue@intel.com> Co-authored-by: Michal Adamczyk <madamczyk@habana.ai> * [doc] add download path tips (vllm-project#17013) Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com> * [Bugfix] Triton FA function takes no keyword arguments (vllm-project#16902) Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com> * [V1] Avoid socket errors during shutdown when requests are in in-flight (vllm-project#16807) Signed-off-by: Nick Hill <nhill@redhat.com> * [BugFix] llama4 fa3 fix - RuntimeError: scheduler_metadata must have shape (metadata_size) (vllm-project#16998) Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> * [Misc] Improve readability of get_open_port function. (vllm-project#17024) Signed-off-by: gitover22 <qidizou88@gmail.com> * [Bugfix] Fix AssertionError: skip_special_tokens=False is not supported for Mistral tokenizers (vllm-project#16964) Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com> * [CI] Run v1/test_serial_utils.py in CI (vllm-project#16996) Signed-off-by: Russell Bryant <rbryant@redhat.com> * Mistral-format support for compressed-tensors (vllm-project#16803) Signed-off-by: mgoin <mgoin64@gmail.com> * Categorize `tests/kernels/` based on kernel type (vllm-project#16799) Signed-off-by: mgoin <mgoin64@gmail.com> * [Doc] Add top anchor and a note to quantization/bitblas.md (vllm-project#17042) Signed-off-by: windsonsea <haifeng.yao@daocloud.io> * Ensure that `pid` passed to `kill_process_tree` is `int` for `mypy` (vllm-project#17051) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [CI] Update structured-output label automation (vllm-project#17055) Signed-off-by: Russell Bryant <rbryant@redhat.com> * Improve Transformers backend model loading QoL (vllm-project#17039) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * `CacheConfig.block_size` should always be `int` when used (vllm-project#17052) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * Use `@property` and private field for `data_parallel_rank_local` (vllm-project#17053) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [Frontend] Support guidance:no-additional-properties for compatibility with xgrammar (vllm-project#15949) Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com> * [BugFix][V1] Fix int32 token index overflow when preparing input ids (vllm-project#16806) * [V1][Spec Decode] Always use argmax for sampling draft tokens (vllm-project#16899) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu> * [CI/Build] workaround for CI build failure (vllm-project#17070) Signed-off-by: csy1204 <josang1204@gmail.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> * [Quantization]add prefix for commandA quantized model (vllm-project#17017) * [Minor] Use larger batch sizes for A100/B100/B200/MI300x (vllm-project#17073) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu> * [Bugfix] Enable V1 usage stats (vllm-project#16986) Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: Nick Hill <nhill@redhat.com> Co-authored-by: Nick Hill <nhill@redhat.com> * More informative error when using Transformers backend (vllm-project#16988) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * Addendum Fix to support FIPS enabled machines with MD5 hashing (vllm-project#17043) Signed-off-by: sydarb <areebsyed237@gmail.com> * [Bugfix][Core] add seq_id_to_seq_group clearing to avoid memory leak when s… (vllm-project#16472) Signed-off-by: 开哲 <kaizhe.zy@alibaba-inc.com> Co-authored-by: 开哲 <kaizhe.zy@alibaba-inc.com> * [V1] Update structured output (vllm-project#16812) Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com> * [doc] update to hyperlink (vllm-project#17096) Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com> * Add docs for runai_streamer_sharded (vllm-project#17093) Signed-off-by: Omer Dayan (SW-GPU) <omer@run.ai> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> * [Chore] Remove Sampler from Model Code (vllm-project#17084) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu> * Disable enforce_eager for V1 TPU sampler and structured output tests (vllm-project#17016) Signed-off-by: mgoin <mgoin64@gmail.com> * Simplify `TokenizerGroup` (vllm-project#16790) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * Fix OOT registration test (vllm-project#17099) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [V1][PP] Optimization: continue scheduling prefill chunks (vllm-project#17080) Signed-off-by: Rui Qiao <ruisearch42@gmail.com> * [Misc] Remove OLMo2 config copy (vllm-project#17066) Signed-off-by: Isotr0py <2037008807@qq.com> * Improve static type checking in `LoRAModelRunnerMixin` (vllm-project#17104) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [V1][Structured Output] Clear xgrammar compiler object when engine core shut down to avoid nanobind leaked warning (vllm-project#16954) Signed-off-by: shen-shanshan <467638484@qq.com> * [Frontend] Using matryoshka_dimensions control the allowed output dimensions. (vllm-project#16970) * Add missing rocm_skinny_gemms kernel test to CI (vllm-project#17060) Signed-off-by: mgoin <mgoin64@gmail.com> * [Misc] refactor example series - structured outputs (vllm-project#17040) Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com> * [V1][Spec Decoding] Add num_drafts and num_accepted_tokens_per_position metrics (vllm-project#16665) Signed-off-by: Mark McLoughlin <markmc@redhat.com> * [CI] Add automation for the `tool-calling` github label (vllm-project#17118) Signed-off-by: Russell Bryant <rbryant@redhat.com> * Updating builkite job for IBM Power (vllm-project#17111) Signed-off-by: Aaruni Aggarwal <aaruniagg@gmail.com> * existing torch installation pip command fix for docs (vllm-project#17059) * Molmo Requirements (vllm-project#17026) Signed-off-by: Eyshika Agarwal <eyshikaengineer@gmail.com> Signed-off-by: eyshika <eyshikaengineer@gmail.com> * Add `:markdownhelp:` to `EngineArgs` docs so markdown docstrings render properly (vllm-project#17124) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * Improve configs - `LoRAConfig` + `PromptAdapterConfig` (vllm-project#16980) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [Docs] Generate correct github links for decorated functions (vllm-project#17125) Signed-off-by: Russell Bryant <rbryant@redhat.com> * Add collective_rpc to llm engine (vllm-project#16999) Signed-off-by: Yinghai Lu <yinghai@thinkingmachines.ai> * Add chat template for Llama 4 models (vllm-project#16428) Signed-off-by: Max de Bayser <mbayser@br.ibm.com> * [Misc] Add example to run DeepSeek with Ray Serve LLM (vllm-project#17134) Signed-off-by: Rui Qiao <ruisearch42@gmail.com> * Better error message for missing mistral params.json (vllm-project#17132) Signed-off-by: mgoin <mgoin64@gmail.com> * Use custom address for listening socket (vllm-project#15988) Signed-off-by: Jens Glaser <glaserj@ornl.gov> * [FEAT] [ROCm]: AITER Fused MOE V1 Support (vllm-project#16752) Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com> Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com> * [Attention] FA3 decode perf improvement - single mma warp group support for head dim 128 (vllm-project#16864) Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> * fix float16 support for kimi-vl (vllm-project#17156) Co-authored-by: zhouzaida <zhouzaida@msh.team> * [Doc] V1 : Update LoRA status (vllm-project#17133) Signed-off-by: varun sundar rabindranath <vsundarr@redhat.com> Co-authored-by: varun sundar rabindranath <vsundarr@redhat.com> * [Docs] Fix True->true in supported_models.md (vllm-project#17141) * Move missed `SchedulerConfig` args into scheduler config group in `EngineArgs` (vllm-project#17131) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [Misc] Clean up redundant code in uniproc_executor.py (vllm-project#16762) Signed-off-by: Lifu Huang <lifu.hlf@gmail.com> * [Bugfix][Misc] Use TritonPlaceholderModule to defensively import triton (vllm-project#15099) Signed-off-by: Mengqing Cao <cmq0113@163.com> * [Misc] Benchmark Serving Script Support Appending Results (vllm-project#17028) Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> * [Perf]Optimize rotary_emb implementation to use Triton operator for improved inference performance (vllm-project#16457) Signed-off-by: cynthieye <yexin93@qq.com> Co-authored-by: MagnetoWang <magnetowang@outlook.com> * [Bugfix] remove fallback in guided_json (int range, patterns) (vllm-project#16725) Signed-off-by: csy1204 <josang1204@gmail.com> Co-authored-by: 조상연[플레이스 AI] <sang-yeon.cho@navercorp.com> * [Quantization][FP8] Add support for FP8 models with input_scale for output projection and QK quantization (vllm-project#15734) Signed-off-by: Randall Smith <Randall.Smith@amd.com> Signed-off-by: Luka Govedič <lgovedic@redhat.com> Co-authored-by: Luka Govedič <lgovedic@redhat.com> * [Doc] Add headings to improve gptqmodel.md (vllm-project#17164) Signed-off-by: windsonsea <haifeng.yao@daocloud.io> * Only turn on FastIncrementalDetokenizer when tokenizers >= 0.21.1 (vllm-project#17158) * [Doc] Add two links to disagg_prefill.md (vllm-project#17168) Signed-off-by: windsonsea <haifeng.yao@daocloud.io> * [Doc] Move todo out of beam search docstring (vllm-project#17183) Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com> * [Bugfix] Fix mistral model tests (vllm-project#17181) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> * [Bugfix] Fix Mistral ChatCompletionRequest Body Exception (vllm-project#16769) Signed-off-by: Jasmond Loh <Jasmond.Loh@hotmail.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> * Fix API typo and remove FP8 on V1 restriction --------- Signed-off-by: Nick Hill <nhill@redhat.com> Signed-off-by: Chenyaaang <chenyangli@google.com> Signed-off-by: Guillaume Calmettes <gcalmettes@scaleway.com> Signed-off-by: Yang Wang <elainewy@meta.com> Signed-off-by: Sage Moore <sage@neuralmagic.com> Signed-off-by: root <root@banff-cyxtera-s73-5.ctr.dcgpu> Signed-off-by: Aleksandr Malyshev <maleksan@amd.com> Signed-off-by: root <root@banff-cyxtera-s65-4.amd.com> Signed-off-by: maleksan85 <maleksan@amd.com> Signed-off-by: <> Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com> Signed-off-by: youkaichao <youkaichao@gmail.com> Signed-off-by: Michal Adamczyk <michal.adamczyk@intel.com> Signed-off-by: Chendi Xue <chendi.xue@intel.com> Signed-off-by: reidliu41 <reid201711@gmail.com> Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> Signed-off-by: gitover22 <qidizou88@gmail.com> Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com> Signed-off-by: Russell Bryant <rbryant@redhat.com> Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: windsonsea <haifeng.yao@daocloud.io> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com> Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu> Signed-off-by: csy1204 <josang1204@gmail.com> Signed-off-by: sydarb <areebsyed237@gmail.com> Signed-off-by: 开哲 <kaizhe.zy@alibaba-inc.com> Signed-off-by: Omer Dayan (SW-GPU) <omer@run.ai> Signed-off-by: Rui Qiao <ruisearch42@gmail.com> Signed-off-by: Isotr0py <2037008807@qq.com> Signed-off-by: shen-shanshan <467638484@qq.com> Signed-off-by: Mark McLoughlin <markmc@redhat.com> Signed-off-by: Aaruni Aggarwal <aaruniagg@gmail.com> Signed-off-by: Eyshika Agarwal <eyshikaengineer@gmail.com> Signed-off-by: eyshika <eyshikaengineer@gmail.com> Signed-off-by: Yinghai Lu <yinghai@thinkingmachines.ai> Signed-off-by: Max de Bayser <mbayser@br.ibm.com> Signed-off-by: Jens Glaser <glaserj@ornl.gov> Signed-off-by: varun sundar rabindranath <vsundarr@redhat.com> Signed-off-by: Lifu Huang <lifu.hlf@gmail.com> Signed-off-by: Mengqing Cao <cmq0113@163.com> Signed-off-by: cynthieye <yexin93@qq.com> Signed-off-by: Randall Smith <Randall.Smith@amd.com> Signed-off-by: Luka Govedič <lgovedic@redhat.com> Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com> Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: Jasmond Loh <Jasmond.Loh@hotmail.com> Co-authored-by: Nick Hill <nhill@redhat.com> Co-authored-by: Chenyaaang <42742451+Chenyaaang@users.noreply.github.com> Co-authored-by: Guillaume Calmettes <gcalmettes@scaleway.com> Co-authored-by: Yang Wang <elainewy@meta.com> Co-authored-by: Aleksandr Malyshev <164964928+maleksan85@users.noreply.github.com> Co-authored-by: Sage Moore <sage@neuralmagic.com> Co-authored-by: root <root@banff-cyxtera-s73-5.ctr.dcgpu> Co-authored-by: Aleksandr Malyshev <maleksan@amd.com> Co-authored-by: qli88 <qiang.li2@amd.com> Co-authored-by: root <root@banff-cyxtera-s65-4.amd.com> Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com> Co-authored-by: Chauncey <chaunceyjiang@gmail.com> Co-authored-by: youkaichao <youkaichao@gmail.com> Co-authored-by: Chendi.Xue <chendi.xue@intel.com> Co-authored-by: Michal Adamczyk <madamczyk@habana.ai> Co-authored-by: Reid <61492567+reidliu41@users.noreply.github.com> Co-authored-by: reidliu41 <reid201711@gmail.com> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Co-authored-by: huafeng <qidizou88@gmail.com> Co-authored-by: Russell Bryant <rbryant@redhat.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> Co-authored-by: Michael Yao <haifeng.yao@daocloud.io> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Travis Johnson <tsjohnso@us.ibm.com> Co-authored-by: Yong Hoon Shin <48474650+sarckk@users.noreply.github.com> Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu> Co-authored-by: Sangyeon Cho <josang1204@gmail.com> Co-authored-by: Chen Xia <cxia0209@gmail.com> Co-authored-by: Areeb Syed <areebsyed237@gmail.com> Co-authored-by: 张宇 <zhangyuygss@outlook.com> Co-authored-by: 开哲 <kaizhe.zy@alibaba-inc.com> Co-authored-by: omer-dayan <omdayan@nvidia.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Rui Qiao <161574667+ruisearch42@users.noreply.github.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Shanshan Shen <467638484@qq.com> Co-authored-by: wang.yuqi <noooop@126.com> Co-authored-by: Mark McLoughlin <markmc@redhat.com> Co-authored-by: Aaruni Aggarwal <47731267+AaruniAggarwal@users.noreply.github.com> Co-authored-by: Atilla <48064466+atilla00@users.noreply.github.com> Co-authored-by: Eyshika Agarwal <eyshikaengineer@gmail.com> Co-authored-by: Yinghai Lu <yinghai@thinkingmachines.ai> Co-authored-by: Maximilien de Bayser <mbayser@br.ibm.com> Co-authored-by: jglaser <glaserj@ornl.gov> Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com> Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> Co-authored-by: zhouzaida <zhouzaida@msh.team> Co-authored-by: Varun Sundar Rabindranath <varunsundar08@gmail.com> Co-authored-by: varun sundar rabindranath <vsundarr@redhat.com> Co-authored-by: Lifu Huang <lifu.hlf@gmail.com> Co-authored-by: Mengqing Cao <cmq0113@163.com> Co-authored-by: yexin(叶鑫) <yexin93@qq.com> Co-authored-by: MagnetoWang <magnetowang@outlook.com> Co-authored-by: 조상연[플레이스 AI] <sang-yeon.cho@navercorp.com> Co-authored-by: rasmith <Randall.Smith@amd.com> Co-authored-by: Luka Govedič <lgovedic@redhat.com> Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com> Co-authored-by: Alex Brooks <alex.brooks@ibm.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk> Co-authored-by: Jasmond L <120363110+JasmondL@users.noreply.github.com>

…on metrics (vllm-project#16665) Signed-off-by: Mark McLoughlin <markmc@redhat.com>

davidxia · 2025-04-29T13:48:16Z

Hi, just curious if this changed the behavior of http_* metrics? I used these CPU dev setup instructions and built from source. I ran vllm serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 on the commit that was merged 340d7b1 into main branch and don't see anything with curl --silent localhost:8000/metrics | grep http. But on the parent commit 1bcbcbf, the metrics appear.

If I revert that commit on tip of main, these metrics also exist. But on git tag v0.8.5, the metrics don't exist.

If I compile from source for GPU, the metrics don't show up before 340d7b1 all the way back to at least v0.8.3.

Can someone confirm if this http_* metric change occurred and if it is intentional?

I don't see how this change removed the http_* metrics. The only metrics removed from vllm/v1/metrics/loggers.py are ones related to speculative decoding.

markmc · 2025-04-29T15:34:36Z

Hi, just curious if this changed the behavior of http_* metrics? I used these CPU dev setup instructions and built from source. I ran vllm serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 on the commit that was merged 340d7b1 into main branch and don't see anything with curl --silent localhost:8000/metrics | grep http.

Thanks for the report! But I can't reproduce this, using the same commit and same command, I see the HTTP metrics

But on the parent commit 1bcbcbf, the metrics appear.

If I revert that commit on tip of main, these metrics also exist. But on git tag v0.8.5, the metrics don't exist.

Can someone confirm if this http_* metric change occurred and if it is intentional?

I don't see how this change removed the http_* metrics. The only metrics removed from vllm/v1/metrics/loggers.py are ones related to speculative decoding.

The HTTP metrics were fixed in in V1 by #15894

davidxia · 2025-04-29T15:42:35Z

@markmc thanks for checking. How are you installing and running vllm? Are you installing from source for GPU with conda as documented here?

Feel free to message me on Slack. I'm dxia there. Relevant thread.

…on metrics (vllm-project#16665) Signed-off-by: Mark McLoughlin <markmc@redhat.com>

…on metrics (vllm-project#16665) Signed-off-by: Mark McLoughlin <markmc@redhat.com> Signed-off-by: Agata Dobrzyniewicz <adobrzyniewicz@habana.ai>

…on metrics (vllm-project#16665) Signed-off-by: Mark McLoughlin <markmc@redhat.com> Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>

…on metrics (vllm-project#16665) Signed-off-by: Mark McLoughlin <markmc@redhat.com> Signed-off-by: minpeter <kali2005611@gmail.com>

markmc requested review from WoosukKwon, robertgshaw2-redhat, njhill, ywang96, comaniac and alexm-redhat as code owners April 15, 2025 14:08

mergify bot added the v1 label Apr 15, 2025

markmc mentioned this pull request Apr 15, 2025

[V1] Add request-level, per-step acceptance counts tracking for spec dec. #16367

Closed

markmc added the speculative-decoding label Apr 15, 2025

luyuzhe111 reviewed Apr 15, 2025

View reviewed changes

vllm/v1/metrics/loggers.py Show resolved Hide resolved

wwl2755 reviewed Apr 16, 2025

View reviewed changes

markmc force-pushed the metrics-v1-spec-decoding-next branch from 2795078 to b56cd97 Compare April 17, 2025 15:12

markmc added the ready ONLY add when PR is ready to merge/full CI is needed label Apr 17, 2025

mergify bot added the needs-rebase label Apr 17, 2025

markmc force-pushed the metrics-v1-spec-decoding-next branch from b56cd97 to 71aa2b6 Compare April 18, 2025 07:44

mergify bot removed the needs-rebase label Apr 18, 2025

markmc force-pushed the metrics-v1-spec-decoding-next branch from 71aa2b6 to ae3608a Compare April 18, 2025 09:37

mergify bot added the needs-rebase label Apr 22, 2025

markmc force-pushed the metrics-v1-spec-decoding-next branch from ae3608a to 0d91a93 Compare April 22, 2025 06:29

mergify bot removed the needs-rebase label Apr 22, 2025

This was referenced Apr 22, 2025

[V1][Metrics] Add API for accessing in-memory Prometheus metrics #17010

Merged

[V1][Spec Decode] EAGLE-3 Support #16937

Merged

WoosukKwon reviewed Apr 24, 2025

View reviewed changes

tests/v1/core/test_scheduler.py Show resolved Hide resolved

markmc commented Apr 24, 2025

View reviewed changes

tests/v1/core/test_scheduler.py Show resolved Hide resolved

vllm/v1/core/sched/scheduler.py Show resolved Hide resolved

tests/v1/core/test_scheduler.py Show resolved Hide resolved

tests/v1/core/test_scheduler.py Show resolved Hide resolved

WoosukKwon reviewed Apr 24, 2025

View reviewed changes

vllm/v1/spec_decode/metrics.py Outdated Show resolved Hide resolved

WoosukKwon approved these changes Apr 24, 2025

View reviewed changes

vllm/v1/spec_decode/metrics.py Outdated Show resolved Hide resolved

markmc commented Apr 24, 2025

View reviewed changes

vllm/v1/spec_decode/metrics.py Outdated Show resolved Hide resolved

markmc force-pushed the metrics-v1-spec-decoding-next branch from 26731a3 to 2682063 Compare April 24, 2025 09:44

markmc added 6 commits April 24, 2025 05:53

[V1][Spec Decoding] Refactor Prometheus code into v1.spec_decode

8f046b5

Signed-off-by: Mark McLoughlin <markmc@redhat.com>

[V1][Spec Decoding] Add comments to SpecDecodingStats/Metrics

0eaa7ea

Signed-off-by: Mark McLoughlin <markmc@redhat.com>

[V1][Spec Decoding] Remove unused methods

6dfe25b

These must have been added in an earlier version of the PR where the SpecDecodingStats instance persisted. Signed-off-by: Mark McLoughlin <markmc@redhat.com>

[V1][Spec Decoding] Add scheduler test for num_drafts and num_accepte…

abab824

…d_tokens_per_pos Signed-off-by: Mark McLoughlin <markmc@redhat.com>

markmc force-pushed the metrics-v1-spec-decoding-next branch from 2682063 to abab824 Compare April 24, 2025 09:53

WoosukKwon merged commit 340d7b1 into vllm-project:main Apr 24, 2025
68 of 73 checks passed

jikunshang pushed a commit to jikunshang/vllm that referenced this pull request Apr 29, 2025

[V1][Spec Decoding] Add num_drafts and num_accepted_tokens_per_positi…

c30bdc4

…on metrics (vllm-project#16665) Signed-off-by: Mark McLoughlin <markmc@redhat.com>

lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Apr 29, 2025

[V1][Spec Decoding] Add num_drafts and num_accepted_tokens_per_positi…

88f8f1e

…on metrics (vllm-project#16665) Signed-off-by: Mark McLoughlin <markmc@redhat.com>

ckhordiasma mentioned this pull request May 14, 2025

nm vllm ent 0.8.5 sync red-hat-data-services/vllm#139

Merged

Uh oh!

[V1][Spec Decoding] Add num_drafts and num_accepted_tokens_per_position metrics #16665

[V1][Spec Decoding] Add num_drafts and num_accepted_tokens_per_position metrics #16665

Conversation

markmc commented Apr 15, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 15, 2025

Uh oh!

markmc commented Apr 15, 2025

Uh oh!

Uh oh!

wwl2755 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markmc commented Apr 16, 2025

Uh oh!

markmc commented Apr 16, 2025

Uh oh!

markmc commented Apr 17, 2025

Uh oh!

mergify bot commented Apr 17, 2025

Uh oh!

mergify bot commented Apr 22, 2025

Uh oh!

WoosukKwon commented Apr 24, 2025

Uh oh!

markmc commented Apr 24, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

WoosukKwon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

davidxia commented Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

markmc commented Apr 29, 2025

Uh oh!

davidxia commented Apr 29, 2025

Uh oh!

Uh oh!

markmc commented Apr 15, 2025 •

edited by github-actions bot

Loading

wwl2755 left a comment •

edited

Loading

davidxia commented Apr 29, 2025 •

edited

Loading