[Bug]: xgrammar crashes with speculative decoding #11484
Description
Your current environment
The output of `python collect_env.py`
$python collect_env.py
Collecting environment information...
PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A
OS: Alibaba Group Enterprise Linux Server 7.2 (Paladin) (x86_64)
GCC version: (GCC) 10.2.1 20200825 (Alibaba 10.2.1-3 2.17)
Clang version: Could not collect
CMake version: version 3.26.4
Libc version: glibc-2.32
Python version: 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-4.9.151-015.ali3000.alios7.x86_64-x86_64-with-glibc2.32
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA L20
GPU 1: NVIDIA L20
Nvidia driver version: 535.161.08
cuDNN version: Probably one of the following:
/usr/local/cuda/targets/x86_64-linux/lib/libcudnn.so.8.9.3
/usr/local/cuda/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.9.3
/usr/local/cuda/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.9.3
/usr/local/cuda/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.9.3
/usr/local/cuda/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.9.3
/usr/local/cuda/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.9.3
/usr/local/cuda/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.9.3
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 128
On-line CPU(s) list: 0-127
Thread(s) per core: 2
Core(s) per socket: 32
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 143
Model name: Intel(R) Xeon(R) Gold 6462C
Stepping: 8
CPU MHz: 3899.816
CPU max MHz: 3900.0000
CPU min MHz: 800.0000
BogoMIPS: 6600.00
Virtualization: VT-x
L1d cache: 48K
L1i cache: 32K
L2 cache: 2048K
L3 cache: 61440K
NUMA node0 CPU(s): 0-31,64-95
NUMA node1 CPU(s): 32-63,96-127
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single cat_l2 cdp_l3 ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req avx512vbmi pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq pconfig flush_l1d arch_capabilities
Versions of relevant libraries:
[pip3] flake8==7.1.1
[pip3] flashinfer==0.1.6+cu121torch2.4
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.0
[pip3] torch==2.5.1
[pip3] torchaudio==2.1.0+cu121
[pip3] torchvision==0.20.1
[pip3] transformers==4.47.0
[pip3] triton==3.1.0
[conda] flashinfer 0.1.6+cu121torch2.4 pypi_0 pypi
[conda] numpy 1.26.4 pypi_0 pypi
[conda] nvidia-cublas-cu12 12.4.5.8 pypi_0 pypi
[conda] nvidia-cuda-cupti-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cuda-nvrtc-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cuda-runtime-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cudnn-cu12 9.1.0.70 pypi_0 pypi
[conda] nvidia-cufft-cu12 11.2.1.3 pypi_0 pypi
[conda] nvidia-curand-cu12 10.3.5.147 pypi_0 pypi
[conda] nvidia-cusolver-cu12 11.6.1.9 pypi_0 pypi
[conda] nvidia-cusparse-cu12 12.3.1.170 pypi_0 pypi
[conda] nvidia-ml-py 12.560.30 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.21.5 pypi_0 pypi
[conda] nvidia-nvjitlink-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-nvtx-cu12 12.4.127 pypi_0 pypi
[conda] pyzmq 26.2.0 pypi_0 pypi
[conda] torch 2.5.1 pypi_0 pypi
[conda] torchaudio 2.1.0+cu121 pypi_0 pypi
[conda] torchvision 0.20.1 pypi_0 pypi
[conda] transformers 4.47.0 pypi_0 pypi
[conda] triton 3.1.0 pypi_0 pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.5
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X SYS 0-31,64-95 0 N/A
GPU1 SYS X 32-63,96-127 1 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
NVIDIA_VISIBLE_DEVICES=GPU-4fc988d5-d6ed-a0c6-4c45-3b7dccb2e7a2,GPU-e671ef65-d6ae-ecc4-bf99-777fe77a5207
LD_LIBRARY_PATH=/opt/conda/lib/python3.10/site-packages/cv2/../../lib64:/opt/conda/lib/python3.10/site-packages/nvidia/nvjitlink/lib::/opt/conda/lib/python3.10/site-packages/aistudio_common/reader/libs/:/opt/taobao/java/jre/lib/amd64/server/:/usr/local/cuda/lib64:/opt/conda/lib/python3.10/site-packages/aistudio_common/reader/libs/:/opt/taobao/java/jre/lib/amd64/server/:/usr/local/cuda/lib64:/opt/conda/lib/python3.10/site-packages/aistudio_common/reader/libs/:/opt/taobao/java/jre/lib/amd64/server/:/usr/local/cuda/lib64:/opt/conda/lib/python3.10/site-packages/aistudio_common/reader/libs/:/opt/taobao/java/jre/lib/amd64/server/:/usr/local/cuda/lib64:/opt/conda/lib/python3.10/site-packages/aistudio_common/reader/libs/:/opt/taobao/java/jre/lib/amd64/server/:/usr/local/cuda/lib64
NVIDIA_DRIVER_CAPABILITIES=all
CUDA_MODULE_LOADING=LAZY
Model Input Dumps
No response
🐛 Describe the bug
When I use xgrammer as guided decoding backend, it crashes with speculative decoding. It works well without speculative decoding.
shell script:
export VLLM_ATTENTION_BACKEND=FLASHINFER
export TOKENIZERS_PARALLELISM=false
model_path="./Qwen2-72B-Instruct-GPTQ-Int4/"
speculative_model_path="./Qwen2-7B-Instruct-GPTQ-Int4/"
max_model_len=8192
port=8000
max_num_seqs=32
max_num_batched_tokens=4096
python -m vllm.entrypoints.openai.api_server \
--model ${model_path} \
--served-model-name model \
--max-model-len ${max_model_len} \
--max-seq-len-to-capture ${max_model_len} \
--distributed-executor-backend mp \
--disable-log-requests \
--tensor-parallel-size 2 \
--pipeline-parallel-size 1 \
--disable-custom-all-reduce \
--uvicorn-log-level error \
--port ${port} \
--max-num-seqs ${max_num_seqs} \
--gpu-memory-utilization 0.95 \
--enable-prefix-caching \
--enable-chunked-prefill \
--max-num-batched-tokens ${max_num_batched_tokens} \
--speculative-model ${speculative_model_path} \
--speculative-draft-tensor-parallel-size 2 \
--num-speculative-tokens 3 \
--speculative-disable-by-batch-size 10 \
--compilation-config 3 \output:
INFO 12-25 15:08:42 chat_utils.py:333] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
INFO 12-25 15:08:47 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20241225-150847.pkl...
WARNING 12-25 15:08:47 model_runner_base.py:143] Failed to pickle inputs of failed execution: cannot pickle 'flashinfer._prefill.BatchPrefillWithPagedKVCachePyTorchWrapper' object
ERROR 12-25 15:08:47 engine.py:135] IndexError('Error in model execution: tuple index out of range')
ERROR 12-25 15:08:47 engine.py:135] Traceback (most recent call last):
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/vllm/worker/model_runner_base.py", line 116, in _wrapper
ERROR 12-25 15:08:47 engine.py:135] return func(*args, **kwargs)
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1729, in execute_model
ERROR 12-25 15:08:47 engine.py:135] logits = self.model.compute_logits(hidden_or_intermediate_states,
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 487, in compute_logits
ERROR 12-25 15:08:47 engine.py:135] logits = self.logits_processor(self.lm_head, hidden_states,
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 12-25 15:08:47 engine.py:135] return self._call_impl(*args, **kwargs)
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 12-25 15:08:47 engine.py:135] return forward_call(*args, **kwargs)
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/logits_processor.py", line 77, in forward
ERROR 12-25 15:08:47 engine.py:135] logits = _apply_logits_processors(logits, sampling_metadata)
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/logits_processor.py", line 153, in _apply_logits_processors
ERROR 12-25 15:08:47 engine.py:135] logits_row = logits_processor(past_tokens_ids,
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/guided_decoding/xgrammar_decoding.py", line 258, in __call__
ERROR 12-25 15:08:47 engine.py:135] sampled_token = input_ids[-1]
ERROR 12-25 15:08:47 engine.py:135] IndexError: tuple index out of range
ERROR 12-25 15:08:47 engine.py:135]
ERROR 12-25 15:08:47 engine.py:135] The above exception was the direct cause of the following exception:
ERROR 12-25 15:08:47 engine.py:135]
ERROR 12-25 15:08:47 engine.py:135] Traceback (most recent call last):
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 133, in start
ERROR 12-25 15:08:47 engine.py:135] self.run_engine_loop()
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 196, in run_engine_loop
ERROR 12-25 15:08:47 engine.py:135] request_outputs = self.engine_step()
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 214, in engine_step
ERROR 12-25 15:08:47 engine.py:135] raise e
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 205, in engine_step
ERROR 12-25 15:08:47 engine.py:135] return self.engine.step()
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 1405, in step
ERROR 12-25 15:08:47 engine.py:135] outputs = self.model_executor.execute_model(
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 82, in execute_model
ERROR 12-25 15:08:47 engine.py:135] driver_outputs = self._driver_execute_model(execute_model_req)
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 120, in _driver_execute_model
ERROR 12-25 15:08:47 engine.py:135] return self.driver_worker.execute_model(execute_model_req)
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 12-25 15:08:47 engine.py:135] return func(*args, **kwargs)
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/vllm/spec_decode/spec_decode_worker.py", line 492, in execute_model
ERROR 12-25 15:08:47 engine.py:135] return self._run_no_spec(execute_model_req,
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
ERROR 12-25 15:08:47 engine.py:135] return func(*args, **kwds)
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/vllm/spec_decode/spec_decode_worker.py", line 649, in _run_no_spec
ERROR 12-25 15:08:47 engine.py:135] self.proposer_worker.execute_model(execute_model_req)
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/vllm/spec_decode/multi_step_worker.py", line 69, in execute_model
ERROR 12-25 15:08:47 engine.py:135] return self.worker.execute_model(*args, **kwargs)
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 343, in execute_model
ERROR 12-25 15:08:47 engine.py:135] output = self.model_runner.execute_model(
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 12-25 15:08:47 engine.py:135] return func(*args, **kwargs)
ERROR 12-25 15:08:47 engine.py:135] File "/opt/conda/lib/python3.10/site-packages/vllm/worker/model_runner_base.py", line 146, in _wrapper
ERROR 12-25 15:08:47 engine.py:135] raise type(err)(f"Error in model execution: "
ERROR 12-25 15:08:47 engine.py:135] IndexError: Error in model execution: tuple index out of range
CRITICAL 12-25 15:08:47 launcher.py:99] MQLLMEngine is already dead, terminating server process
CRITICAL 12-25 15:08:47 launcher.py:99] MQLLMEngine is already dead, terminating server process
CRITICAL 12-25 15:08:47 launcher.py:99] MQLLMEngine is already dead, terminating server process
CRITICAL 12-25 15:08:47 launcher.py:99] MQLLMEngine is already dead, terminating server process
CRITICAL 12-25 15:08:47 launcher.py:99] MQLLMEngine is already dead, terminating server process
CRITICAL 12-25 15:08:47 launcher.py:99] MQLLMEngine is already dead, terminating server process
CRITICAL 12-25 15:08:47 launcher.py:99] MQLLMEngine is already dead, terminating server process
CRITICAL 12-25 15:08:47 launcher.py:99] MQLLMEngine is already dead, terminating server process
Process SpawnProcess-1:
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 317, in _bootstrap
util._exit_function()
File "/opt/conda/lib/python3.10/multiprocessing/util.py", line 357, in _exit_function
p.join()
File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 149, in join
res = self._popen.wait(timeout)
File "/opt/conda/lib/python3.10/multiprocessing/popen_fork.py", line 43, in wait
return self.poll(os.WNOHANG if timeout == 0.0 else 0)
File "/opt/conda/lib/python3.10/multiprocessing/popen_fork.py", line 27, in poll
pid, sts = os.waitpid(self.pid, flag)
File "/opt/conda/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 351, in signal_handler
raise KeyboardInterrupt("MQLLMEngine terminated")
KeyboardInterrupt: MQLLMEngine terminated
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.