[BugFix][V1] Fix int32 token index overflow when preparing input ids #16806

sarckk · 2025-04-17T22:23:15Z

The issue can be reproduced when setting max-model-len to a large value (e.g. 3600000) and sending requests without limiting the max number of requests (e.g. not restricting --max-num-seqs, sending requests without --max-concurrency) and/or not limiting max. number of tokens to be processed in a single iteration (--max-num-batched-tokens).

The error is IndexError: index out of range in self which happens in _prepare_inputs:

        torch.index_select(self.input_batch.token_ids_cpu_tensor.flatten(),
                           0,
                           torch.from_numpy(token_indices),
                           out=self.input_ids_cpu[:total_num_scheduled_tokens])

When it fails, the last several items of token_indices contains indices that are negative, due to int32 overflow. token_indices is given by:

        token_indices = (positions_np +
                         req_indices * self.input_batch.token_ids_cpu.shape[1])

Here, req_indices is a np.int32 array, e.g. np.array([0,1,2,...598]), and self.input_batch.token_ids_cpu.shape[1] is the max model len, e.g. 3,600,000. As an example, multiplying 598 * 3,600,000 gives us 2,152,800,000 which is greater np.int32 max value of 2,147,483,647 so it overflows and gives us a negative number for token index.

To avoid overflow, we can cast to int64 before the multiplication. the max token index can be given by max_req_index * max_model_len - 1.

To verify, I checked that below completes successfully on H200 (previously it would fail with index error after ~1000 requests). Without this fix, the max no. of concurrent requests we can serve with 3.6M context length would be 596.

VLLM_DISABLE_COMPILE_CACHE=1 vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct --disable-log-requests --gpu-memory-utilization=0.95 --max-model-len=3600000 -tp 8

python3 benchmarks/benchmark_serving.py --port=8000 --dataset-name=sharegpt --dataset-path=ShareGPT_V3_unfiltered_cleaned_split.json --tokenizer=meta-llama/Llama-4-Scout-17B-16E-Instruct --request-rate=64 --backend=vllm --num-prompts=19200 --model=meta-llama/Llama-4-Scout-17B-16E-Instruct --sharegpt-output-len=1024

github-actions · 2025-04-17T22:23:25Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

luccafong · 2025-04-18T19:58:42Z

vllm/v1/worker/tpu_model_runner.py

        token_indices = (positions_np +
-                         req_indices * self.input_batch.token_ids_cpu.shape[1])
+                         req_indices.astype(self.token_indices_dtype) * 


shall we just create req_indices as token_indices_dtype instead of upcasting here?

req_indices may be used in the following logic.

Another idea is always having int64 index for such use cases. But probably require a bigger change / refactor, thoughts, @WoosukKwon ? Shall we just land this fix, and think about a more thorough check in the following PR?

I think we can create self.arange_np with dtype self.token_indices_dtype here. Increase in host memory is negligible even at 10M and we don't pay the cast at each scheduler step.
Computation is in i64 anyways as self.positions_cpu is already at i64, so that wouldn't be the only instance.

makes sense, maybe we can just keep self.arange_np dtype in int64 similar to self.positions_cpu?

houseroad · 2025-04-22T00:40:12Z

Btw, @sarckk , could you lint the code?

NickLucche

Thanks for fixing this!

NickLucche · 2025-04-22T08:50:53Z

vllm/v1/worker/tpu_model_runner.py

        token_indices = (positions_np +
-                         req_indices * self.input_batch.token_ids_cpu.shape[1])
+                         req_indices.astype(self.token_indices_dtype) * 


I think we can create self.arange_np with dtype self.token_indices_dtype here. Increase in host memory is negligible even at 10M and we don't pay the cast at each scheduler step.
Computation is in i64 anyways as self.positions_cpu is already at i64, so that wouldn't be the only instance.

sarckk · 2025-04-22T16:57:25Z

Btw, @sarckk , could you lint the code?

ran lint with pre-commit

WoosukKwon

LGTM! Thanks for the PR!

houseroad

Looks good. If it becomes a perf bottleneck, we can apply the optimization later.

houseroad · 2025-04-23T16:45:03Z

Rerunning the CI and see how it goes.

NickLucche

Looks clean, thanks!

NickLucche · 2025-04-23T17:54:47Z

PS I think there's some commits missing a sign-off.

* [BugFix] Remove default multiproc executor `collective_rpc` timeout (vllm-project#17000) Signed-off-by: Nick Hill <nhill@redhat.com> * [Core][V1][TPU] Enable structured decoding on TPU V1 (vllm-project#16499) Signed-off-by: Chenyaaang <chenyangli@google.com> * [Bugfix] validate urls object for multimodal content parts (vllm-project#16990) Signed-off-by: Guillaume Calmettes <gcalmettes@scaleway.com> * add Dockerfile build vllm against torch nightly (vllm-project#16936) Signed-off-by: Yang Wang <elainewy@meta.com> * [Kernel][ROCM] Upstream prefix prefill speed up for vLLM V1 (vllm-project#13305) Signed-off-by: Sage Moore <sage@neuralmagic.com> Signed-off-by: root <root@banff-cyxtera-s73-5.ctr.dcgpu> Signed-off-by: Aleksandr Malyshev <maleksan@amd.com> Signed-off-by: root <root@banff-cyxtera-s65-4.amd.com> Signed-off-by: maleksan85 <maleksan@amd.com> Signed-off-by: <> Co-authored-by: Sage Moore <sage@neuralmagic.com> Co-authored-by: root <root@banff-cyxtera-s73-5.ctr.dcgpu> Co-authored-by: Aleksandr Malyshev <maleksan@amd.com> Co-authored-by: qli88 <qiang.li2@amd.com> Co-authored-by: root <root@banff-cyxtera-s65-4.amd.com> * [V1][DP] More robust DP/EP dummy request coordination (vllm-project#16277) Signed-off-by: Nick Hill <nhill@redhat.com> * [BugFix] Revert ROCm Custom Paged Attention Env Flag Check (vllm-project#17022) Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com> * Revert "[Misc] Add S3 environment variables for better support of MinIO." (vllm-project#17021) * [misc] tune some env vars for GB200 (vllm-project#16992) Signed-off-by: youkaichao <youkaichao@gmail.com> * [INTEL-HPU][v0] Port delayed sampling to upstream (vllm-project#16949) Signed-off-by: Michal Adamczyk <michal.adamczyk@intel.com> Signed-off-by: Chendi Xue <chendi.xue@intel.com> Co-authored-by: Michal Adamczyk <madamczyk@habana.ai> * [doc] add download path tips (vllm-project#17013) Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com> * [Bugfix] Triton FA function takes no keyword arguments (vllm-project#16902) Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com> * [V1] Avoid socket errors during shutdown when requests are in in-flight (vllm-project#16807) Signed-off-by: Nick Hill <nhill@redhat.com> * [BugFix] llama4 fa3 fix - RuntimeError: scheduler_metadata must have shape (metadata_size) (vllm-project#16998) Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> * [Misc] Improve readability of get_open_port function. (vllm-project#17024) Signed-off-by: gitover22 <qidizou88@gmail.com> * [Bugfix] Fix AssertionError: skip_special_tokens=False is not supported for Mistral tokenizers (vllm-project#16964) Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com> * [CI] Run v1/test_serial_utils.py in CI (vllm-project#16996) Signed-off-by: Russell Bryant <rbryant@redhat.com> * Mistral-format support for compressed-tensors (vllm-project#16803) Signed-off-by: mgoin <mgoin64@gmail.com> * Categorize `tests/kernels/` based on kernel type (vllm-project#16799) Signed-off-by: mgoin <mgoin64@gmail.com> * [Doc] Add top anchor and a note to quantization/bitblas.md (vllm-project#17042) Signed-off-by: windsonsea <haifeng.yao@daocloud.io> * Ensure that `pid` passed to `kill_process_tree` is `int` for `mypy` (vllm-project#17051) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [CI] Update structured-output label automation (vllm-project#17055) Signed-off-by: Russell Bryant <rbryant@redhat.com> * Improve Transformers backend model loading QoL (vllm-project#17039) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * `CacheConfig.block_size` should always be `int` when used (vllm-project#17052) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * Use `@property` and private field for `data_parallel_rank_local` (vllm-project#17053) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [Frontend] Support guidance:no-additional-properties for compatibility with xgrammar (vllm-project#15949) Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com> * [BugFix][V1] Fix int32 token index overflow when preparing input ids (vllm-project#16806) * [V1][Spec Decode] Always use argmax for sampling draft tokens (vllm-project#16899) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu> * [CI/Build] workaround for CI build failure (vllm-project#17070) Signed-off-by: csy1204 <josang1204@gmail.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> * [Quantization]add prefix for commandA quantized model (vllm-project#17017) * [Minor] Use larger batch sizes for A100/B100/B200/MI300x (vllm-project#17073) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu> * [Bugfix] Enable V1 usage stats (vllm-project#16986) Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: Nick Hill <nhill@redhat.com> Co-authored-by: Nick Hill <nhill@redhat.com> * More informative error when using Transformers backend (vllm-project#16988) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * Addendum Fix to support FIPS enabled machines with MD5 hashing (vllm-project#17043) Signed-off-by: sydarb <areebsyed237@gmail.com> * [Bugfix][Core] add seq_id_to_seq_group clearing to avoid memory leak when s… (vllm-project#16472) Signed-off-by: 开哲 <kaizhe.zy@alibaba-inc.com> Co-authored-by: 开哲 <kaizhe.zy@alibaba-inc.com> * [V1] Update structured output (vllm-project#16812) Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com> * [doc] update to hyperlink (vllm-project#17096) Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com> * Add docs for runai_streamer_sharded (vllm-project#17093) Signed-off-by: Omer Dayan (SW-GPU) <omer@run.ai> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> * [Chore] Remove Sampler from Model Code (vllm-project#17084) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu> * Disable enforce_eager for V1 TPU sampler and structured output tests (vllm-project#17016) Signed-off-by: mgoin <mgoin64@gmail.com> * Simplify `TokenizerGroup` (vllm-project#16790) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * Fix OOT registration test (vllm-project#17099) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [V1][PP] Optimization: continue scheduling prefill chunks (vllm-project#17080) Signed-off-by: Rui Qiao <ruisearch42@gmail.com> * [Misc] Remove OLMo2 config copy (vllm-project#17066) Signed-off-by: Isotr0py <2037008807@qq.com> * Improve static type checking in `LoRAModelRunnerMixin` (vllm-project#17104) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [V1][Structured Output] Clear xgrammar compiler object when engine core shut down to avoid nanobind leaked warning (vllm-project#16954) Signed-off-by: shen-shanshan <467638484@qq.com> * [Frontend] Using matryoshka_dimensions control the allowed output dimensions. (vllm-project#16970) * Add missing rocm_skinny_gemms kernel test to CI (vllm-project#17060) Signed-off-by: mgoin <mgoin64@gmail.com> * [Misc] refactor example series - structured outputs (vllm-project#17040) Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com> * [V1][Spec Decoding] Add num_drafts and num_accepted_tokens_per_position metrics (vllm-project#16665) Signed-off-by: Mark McLoughlin <markmc@redhat.com> * [CI] Add automation for the `tool-calling` github label (vllm-project#17118) Signed-off-by: Russell Bryant <rbryant@redhat.com> * Updating builkite job for IBM Power (vllm-project#17111) Signed-off-by: Aaruni Aggarwal <aaruniagg@gmail.com> * existing torch installation pip command fix for docs (vllm-project#17059) * Molmo Requirements (vllm-project#17026) Signed-off-by: Eyshika Agarwal <eyshikaengineer@gmail.com> Signed-off-by: eyshika <eyshikaengineer@gmail.com> * Add `:markdownhelp:` to `EngineArgs` docs so markdown docstrings render properly (vllm-project#17124) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * Improve configs - `LoRAConfig` + `PromptAdapterConfig` (vllm-project#16980) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [Docs] Generate correct github links for decorated functions (vllm-project#17125) Signed-off-by: Russell Bryant <rbryant@redhat.com> * Add collective_rpc to llm engine (vllm-project#16999) Signed-off-by: Yinghai Lu <yinghai@thinkingmachines.ai> * Add chat template for Llama 4 models (vllm-project#16428) Signed-off-by: Max de Bayser <mbayser@br.ibm.com> * [Misc] Add example to run DeepSeek with Ray Serve LLM (vllm-project#17134) Signed-off-by: Rui Qiao <ruisearch42@gmail.com> * Better error message for missing mistral params.json (vllm-project#17132) Signed-off-by: mgoin <mgoin64@gmail.com> * Use custom address for listening socket (vllm-project#15988) Signed-off-by: Jens Glaser <glaserj@ornl.gov> * [FEAT] [ROCm]: AITER Fused MOE V1 Support (vllm-project#16752) Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com> Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com> * [Attention] FA3 decode perf improvement - single mma warp group support for head dim 128 (vllm-project#16864) Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> * fix float16 support for kimi-vl (vllm-project#17156) Co-authored-by: zhouzaida <zhouzaida@msh.team> * [Doc] V1 : Update LoRA status (vllm-project#17133) Signed-off-by: varun sundar rabindranath <vsundarr@redhat.com> Co-authored-by: varun sundar rabindranath <vsundarr@redhat.com> * [Docs] Fix True->true in supported_models.md (vllm-project#17141) * Move missed `SchedulerConfig` args into scheduler config group in `EngineArgs` (vllm-project#17131) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> * [Misc] Clean up redundant code in uniproc_executor.py (vllm-project#16762) Signed-off-by: Lifu Huang <lifu.hlf@gmail.com> * [Bugfix][Misc] Use TritonPlaceholderModule to defensively import triton (vllm-project#15099) Signed-off-by: Mengqing Cao <cmq0113@163.com> * [Misc] Benchmark Serving Script Support Appending Results (vllm-project#17028) Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> * [Perf]Optimize rotary_emb implementation to use Triton operator for improved inference performance (vllm-project#16457) Signed-off-by: cynthieye <yexin93@qq.com> Co-authored-by: MagnetoWang <magnetowang@outlook.com> * [Bugfix] remove fallback in guided_json (int range, patterns) (vllm-project#16725) Signed-off-by: csy1204 <josang1204@gmail.com> Co-authored-by: 조상연[플레이스 AI] <sang-yeon.cho@navercorp.com> * [Quantization][FP8] Add support for FP8 models with input_scale for output projection and QK quantization (vllm-project#15734) Signed-off-by: Randall Smith <Randall.Smith@amd.com> Signed-off-by: Luka Govedič <lgovedic@redhat.com> Co-authored-by: Luka Govedič <lgovedic@redhat.com> * [Doc] Add headings to improve gptqmodel.md (vllm-project#17164) Signed-off-by: windsonsea <haifeng.yao@daocloud.io> * Only turn on FastIncrementalDetokenizer when tokenizers >= 0.21.1 (vllm-project#17158) * [Doc] Add two links to disagg_prefill.md (vllm-project#17168) Signed-off-by: windsonsea <haifeng.yao@daocloud.io> * [Doc] Move todo out of beam search docstring (vllm-project#17183) Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com> * [Bugfix] Fix mistral model tests (vllm-project#17181) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> * [Bugfix] Fix Mistral ChatCompletionRequest Body Exception (vllm-project#16769) Signed-off-by: Jasmond Loh <Jasmond.Loh@hotmail.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> * Fix API typo and remove FP8 on V1 restriction --------- Signed-off-by: Nick Hill <nhill@redhat.com> Signed-off-by: Chenyaaang <chenyangli@google.com> Signed-off-by: Guillaume Calmettes <gcalmettes@scaleway.com> Signed-off-by: Yang Wang <elainewy@meta.com> Signed-off-by: Sage Moore <sage@neuralmagic.com> Signed-off-by: root <root@banff-cyxtera-s73-5.ctr.dcgpu> Signed-off-by: Aleksandr Malyshev <maleksan@amd.com> Signed-off-by: root <root@banff-cyxtera-s65-4.amd.com> Signed-off-by: maleksan85 <maleksan@amd.com> Signed-off-by: <> Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com> Signed-off-by: youkaichao <youkaichao@gmail.com> Signed-off-by: Michal Adamczyk <michal.adamczyk@intel.com> Signed-off-by: Chendi Xue <chendi.xue@intel.com> Signed-off-by: reidliu41 <reid201711@gmail.com> Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> Signed-off-by: gitover22 <qidizou88@gmail.com> Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com> Signed-off-by: Russell Bryant <rbryant@redhat.com> Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: windsonsea <haifeng.yao@daocloud.io> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com> Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu> Signed-off-by: csy1204 <josang1204@gmail.com> Signed-off-by: sydarb <areebsyed237@gmail.com> Signed-off-by: 开哲 <kaizhe.zy@alibaba-inc.com> Signed-off-by: Omer Dayan (SW-GPU) <omer@run.ai> Signed-off-by: Rui Qiao <ruisearch42@gmail.com> Signed-off-by: Isotr0py <2037008807@qq.com> Signed-off-by: shen-shanshan <467638484@qq.com> Signed-off-by: Mark McLoughlin <markmc@redhat.com> Signed-off-by: Aaruni Aggarwal <aaruniagg@gmail.com> Signed-off-by: Eyshika Agarwal <eyshikaengineer@gmail.com> Signed-off-by: eyshika <eyshikaengineer@gmail.com> Signed-off-by: Yinghai Lu <yinghai@thinkingmachines.ai> Signed-off-by: Max de Bayser <mbayser@br.ibm.com> Signed-off-by: Jens Glaser <glaserj@ornl.gov> Signed-off-by: varun sundar rabindranath <vsundarr@redhat.com> Signed-off-by: Lifu Huang <lifu.hlf@gmail.com> Signed-off-by: Mengqing Cao <cmq0113@163.com> Signed-off-by: cynthieye <yexin93@qq.com> Signed-off-by: Randall Smith <Randall.Smith@amd.com> Signed-off-by: Luka Govedič <lgovedic@redhat.com> Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com> Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: Jasmond Loh <Jasmond.Loh@hotmail.com> Co-authored-by: Nick Hill <nhill@redhat.com> Co-authored-by: Chenyaaang <42742451+Chenyaaang@users.noreply.github.com> Co-authored-by: Guillaume Calmettes <gcalmettes@scaleway.com> Co-authored-by: Yang Wang <elainewy@meta.com> Co-authored-by: Aleksandr Malyshev <164964928+maleksan85@users.noreply.github.com> Co-authored-by: Sage Moore <sage@neuralmagic.com> Co-authored-by: root <root@banff-cyxtera-s73-5.ctr.dcgpu> Co-authored-by: Aleksandr Malyshev <maleksan@amd.com> Co-authored-by: qli88 <qiang.li2@amd.com> Co-authored-by: root <root@banff-cyxtera-s65-4.amd.com> Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com> Co-authored-by: Chauncey <chaunceyjiang@gmail.com> Co-authored-by: youkaichao <youkaichao@gmail.com> Co-authored-by: Chendi.Xue <chendi.xue@intel.com> Co-authored-by: Michal Adamczyk <madamczyk@habana.ai> Co-authored-by: Reid <61492567+reidliu41@users.noreply.github.com> Co-authored-by: reidliu41 <reid201711@gmail.com> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Co-authored-by: huafeng <qidizou88@gmail.com> Co-authored-by: Russell Bryant <rbryant@redhat.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> Co-authored-by: Michael Yao <haifeng.yao@daocloud.io> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Travis Johnson <tsjohnso@us.ibm.com> Co-authored-by: Yong Hoon Shin <48474650+sarckk@users.noreply.github.com> Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu> Co-authored-by: Sangyeon Cho <josang1204@gmail.com> Co-authored-by: Chen Xia <cxia0209@gmail.com> Co-authored-by: Areeb Syed <areebsyed237@gmail.com> Co-authored-by: 张宇 <zhangyuygss@outlook.com> Co-authored-by: 开哲 <kaizhe.zy@alibaba-inc.com> Co-authored-by: omer-dayan <omdayan@nvidia.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Rui Qiao <161574667+ruisearch42@users.noreply.github.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Shanshan Shen <467638484@qq.com> Co-authored-by: wang.yuqi <noooop@126.com> Co-authored-by: Mark McLoughlin <markmc@redhat.com> Co-authored-by: Aaruni Aggarwal <47731267+AaruniAggarwal@users.noreply.github.com> Co-authored-by: Atilla <48064466+atilla00@users.noreply.github.com> Co-authored-by: Eyshika Agarwal <eyshikaengineer@gmail.com> Co-authored-by: Yinghai Lu <yinghai@thinkingmachines.ai> Co-authored-by: Maximilien de Bayser <mbayser@br.ibm.com> Co-authored-by: jglaser <glaserj@ornl.gov> Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com> Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com> Co-authored-by: zhouzaida <zhouzaida@msh.team> Co-authored-by: Varun Sundar Rabindranath <varunsundar08@gmail.com> Co-authored-by: varun sundar rabindranath <vsundarr@redhat.com> Co-authored-by: Lifu Huang <lifu.hlf@gmail.com> Co-authored-by: Mengqing Cao <cmq0113@163.com> Co-authored-by: yexin(叶鑫) <yexin93@qq.com> Co-authored-by: MagnetoWang <magnetowang@outlook.com> Co-authored-by: 조상연[플레이스 AI] <sang-yeon.cho@navercorp.com> Co-authored-by: rasmith <Randall.Smith@amd.com> Co-authored-by: Luka Govedič <lgovedic@redhat.com> Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com> Co-authored-by: Alex Brooks <alex.brooks@ibm.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk> Co-authored-by: Jasmond L <120363110+JasmondL@users.noreply.github.com>

…llm-project#16806)

…llm-project#16806) Signed-off-by: Agata Dobrzyniewicz <adobrzyniewicz@habana.ai>

…llm-project#16806) Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>

…llm-project#16806) Signed-off-by: minpeter <kali2005611@gmail.com>

Fix int32 token index overflow for long context

fe38f9c

mergify bot added the v1 label Apr 17, 2025

sarckk added 2 commits April 17, 2025 15:29

Fix max token index formula

3725309

Lint fixes

86c7260

sarckk force-pushed the fix-long-context-overflow branch from 0a289b7 to 86c7260 Compare April 17, 2025 23:09

sarckk marked this pull request as ready for review April 17, 2025 23:09

sarckk requested review from WoosukKwon, robertgshaw2-redhat, njhill, ywang96, comaniac and alexm-redhat as code owners April 17, 2025 23:09

sarckk marked this pull request as draft April 17, 2025 23:25

sarckk mentioned this pull request Apr 17, 2025

[Bug]: Llama4 Scout fails on H200 #16414

Closed

1 task

Cast to avoid overflow instead of storing as int64

7c1a09c

sarckk force-pushed the fix-long-context-overflow branch from dda7860 to 7c1a09c Compare April 18, 2025 00:01

sarckk marked this pull request as ready for review April 18, 2025 00:42

mergify bot added the tpu Related to Google TPUs label Apr 18, 2025

Add int64 casting for TPU worker

fbc8544

sarckk force-pushed the fix-long-context-overflow branch from d02c4a1 to fbc8544 Compare April 18, 2025 00:55

sarckk changed the title ~~Fix int32 token index overflow when preparing input ids~~ [BugFix] Fix int32 token index overflow when preparing input ids Apr 18, 2025

sarckk changed the title ~~[BugFix] Fix int32 token index overflow when preparing input ids~~ [BugFix][V1] Fix int32 token index overflow when preparing input ids Apr 18, 2025

luccafong reviewed Apr 18, 2025

View reviewed changes

NickLucche suggested changes Apr 22, 2025

View reviewed changes

sarckk added 2 commits April 22, 2025 09:51

Keep arange_np in int64

f399dd4

Change comments

cffa67f

WoosukKwon added the ready ONLY add when PR is ready to merge/full CI is needed label Apr 22, 2025

WoosukKwon approved these changes Apr 22, 2025

View reviewed changes

houseroad approved these changes Apr 23, 2025

View reviewed changes

houseroad requested a review from NickLucche April 23, 2025 16:44

NickLucche approved these changes Apr 23, 2025

View reviewed changes

WoosukKwon merged commit 32d4b66 into vllm-project:main Apr 23, 2025
82 of 85 checks passed

jikunshang pushed a commit to jikunshang/vllm that referenced this pull request Apr 29, 2025

[BugFix][V1] Fix int32 token index overflow when preparing input ids (v…

0fb4690

…llm-project#16806)

lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Apr 29, 2025

[BugFix][V1] Fix int32 token index overflow when preparing input ids (v…

f8132a7

…llm-project#16806)

adobrzyn pushed a commit to HabanaAI/vllm-fork that referenced this pull request Apr 30, 2025

[BugFix][V1] Fix int32 token index overflow when preparing input ids (v…

fdd1ff9

…llm-project#16806) Signed-off-by: Agata Dobrzyniewicz <adobrzyniewicz@habana.ai>

sarckk deleted the fix-long-context-overflow branch May 9, 2025 19:36

RichardoMrMu pushed a commit to RichardoMrMu/vllm that referenced this pull request May 12, 2025

[BugFix][V1] Fix int32 token index overflow when preparing input ids (v…

3b935f6

…llm-project#16806) Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>

ckhordiasma mentioned this pull request May 14, 2025

nm vllm ent 0.8.5 sync red-hat-data-services/vllm#139

Merged

minpeter pushed a commit to minpeter/vllm that referenced this pull request Jun 24, 2025

[BugFix][V1] Fix int32 token index overflow when preparing input ids (v…

8b405cc

…llm-project#16806) Signed-off-by: minpeter <kali2005611@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[BugFix][V1] Fix int32 token index overflow when preparing input ids #16806

[BugFix][V1] Fix int32 token index overflow when preparing input ids #16806

Uh oh!

sarckk commented Apr 17, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Apr 17, 2025

Uh oh!

luccafong Apr 18, 2025

Uh oh!

houseroad Apr 21, 2025

Uh oh!

NickLucche Apr 22, 2025

Uh oh!

sarckk Apr 22, 2025 •

edited

Loading

Uh oh!

houseroad commented Apr 22, 2025

Uh oh!

NickLucche left a comment

Uh oh!

NickLucche Apr 22, 2025

Uh oh!

sarckk commented Apr 22, 2025

Uh oh!

WoosukKwon left a comment

Uh oh!

houseroad left a comment

Uh oh!

houseroad commented Apr 23, 2025

Uh oh!

NickLucche left a comment •

edited

Loading

Uh oh!

NickLucche commented Apr 23, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[BugFix][V1] Fix int32 token index overflow when preparing input ids #16806

[BugFix][V1] Fix int32 token index overflow when preparing input ids #16806

Uh oh!

Conversation

sarckk commented Apr 17, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 17, 2025

Uh oh!

luccafong Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

houseroad Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

NickLucche Apr 22, 2025

Choose a reason for hiding this comment

Uh oh!

sarckk Apr 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

houseroad commented Apr 22, 2025

Uh oh!

NickLucche left a comment

Choose a reason for hiding this comment

Uh oh!

NickLucche Apr 22, 2025

Choose a reason for hiding this comment

Uh oh!

sarckk commented Apr 22, 2025

Uh oh!

WoosukKwon left a comment

Choose a reason for hiding this comment

Uh oh!

houseroad left a comment

Choose a reason for hiding this comment

Uh oh!

houseroad commented Apr 23, 2025

Uh oh!

NickLucche left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NickLucche commented Apr 23, 2025

Uh oh!

Uh oh!

Uh oh!

sarckk commented Apr 17, 2025 •

edited by github-actions bot

Loading

sarckk Apr 22, 2025 •

edited

Loading

NickLucche left a comment •

edited

Loading