vllm serve meta-llama/Llama-3.1-8B-Instruct --enforce-eager

vllm bench serve --model meta-llama/Llama-3.1-8B-Instruct --dataset-name hf --dataset-path mgoin/mlperf-inference-llama2-data --num-prompts 24576

INFO 07-01 21:24:17 [__init__.py:244] Automatically detected platform cuda.
Namespace(subparser='bench', bench_type='serve', dispatch_function=<function BenchmarkServingSubcommand.cmd at 0x7fcfb44493a0>, seed=0, num_prompts=24576, dataset_name='hf', dataset_path='mgoin/mlperf-inference-llama2-data', custom_output_len=256, custom_skip_chat_template=False, sonnet_input_len=550, sonnet_output_len=150, sonnet_prefix_len=200, sharegpt_output_len=None, random_input_len=1024, random_output_len=128, random_range_ratio=0.0, random_prefix_len=0, hf_subset=None, hf_split=None, hf_output_len=None, endpoint_type='openai', label=None, backend='vllm', base_url=None, host='127.0.0.1', port=8000, endpoint='/v1/completions', max_concurrency=None, model='meta-llama/Llama-3.1-8B-Instruct', tokenizer=None, use_beam_search=False, logprobs=None, request_rate=inf, burstiness=1.0, trust_remote_code=False, disable_tqdm=False, profile=False, save_result=False, save_detailed=False, append_result=False, metadata=None, result_dir=None, result_filename=None, ignore_eos=False, percentile_metrics='ttft,tpot,itl', metric_percentiles='99', goodput=None, top_p=None, top_k=None, min_p=None, temperature=None, tokenizer_mode='auto', served_model_name=None, lora_modules=None, ramp_up_strategy=None, ramp_up_start_rps=None, ramp_up_end_rps=None)
README.md: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 541/541 [00:00<00:00, 7.27MB/s]
INFO 07-01 21:24:43 [datasets.py:219] Oversampled requests to reach 24576 total samples.
Starting initial single prompt test run...
Initial test run completed. Starting main benchmark run...
Traffic request rate: inf
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: None
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24576/24576 [06:27<00:00, 63.38it/s]
============ Serving Benchmark Result ============
Successful requests:                     24576     
Benchmark duration (s):                  387.77    
Total input tokens:                      5147391   
Total generated tokens:                  4178056   
Request throughput (req/s):              63.38     
Output token throughput (tok/s):         10774.59  
Total Token throughput (tok/s):          24048.96  
---------------Time to First Token----------------
Mean TTFT (ms):                          206251.00 
Median TTFT (ms):                        186350.21 
P99 TTFT (ms):                           366451.02 
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          67.24     
Median TPOT (ms):                        76.55     
P99 TPOT (ms):                           102.22    
---------------Inter-token Latency----------------
Mean ITL (ms):                           68.51     
Median ITL (ms):                         67.93     
P99 ITL (ms):                            177.94    
==================================================