Improve profiler and integrate profiler in bench_one_batch_server #6787

merrymercy · 2025-05-31T02:33:42Z

We can use python3 -m sglang.bench_one_batch_server to generate profile files for prefill/decode on various batch sizes.

Launch a server

python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct

Benchmark

python3 -m sglang.bench_one_batch_server --model meta-llama/Llama-3.1-8B-Instruct --base-url http://localhost:30000 --batch-size 8 64 --input-len 1024 --output-len 8 --show-report --profile --profile-by-stage

Report

Input lens: (1024,). Output lens: (8,).

batch size	latency (s)	input throughput (tok/s)	output throughput (tok/s)	acc length	ITL (ms)	input cost ($/1M)	output cost ($/1M)	profile
8	0.25	42065.69	1243.03	n/a	6.44	0.02	0.45	Profile
64	1.68	41054.69	6375.42	n/a	10.04	0.02	0.09	Profile

Co-authored-by: SangBin Cho <rkooo567@gmail.com>
Co-authored-by: Leon Gao <leongao@x.ai>

gemini-code-assist

Hello @merrymercy, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

Summary of Changes

Hello team,

Gemini here, providing a summary of this pull request to help everyone get up to speed quickly.

This PR focuses on enhancing the profiling capabilities within the project and integrating these improvements into the bench_one_batch_server script. The primary goal is to allow users to profile the server's performance with more granularity, specifically enabling profiling by different stages of request processing (like prefill and decode), and making this functionality easily accessible through the benchmark tool.

Highlights

Enhanced Profiling Script: A new standalone script (sglang.profiler) is introduced, providing a command-line interface to trigger profiling on a running SGLang server. It supports specifying the number of steps, activities (CPU, GPU, MEM, RPD), output directory, and profile name.
Stage-Based Profiling: The profiling mechanism is updated to support profiling specific stages of the forward pass (prefill and decode). This allows for more targeted performance analysis. This is controlled via a new --profile-by-stage flag.
RPD Profiler Integration: Support for the RPD profiler (for ROCm/AMD GPUs) is added, including automatic conversion of RPD traces to the standard Chrome trace format for easier visualization.
Integration into Benchmark Script: The bench_one_batch_server.py script is updated to include new --profile and --profile-by-stage arguments. When profiling is enabled, the benchmark run will trigger the profiler and include links to the generated trace files in the final report table.
Documentation and Logging Updates: Minor updates were made to the documentation regarding hyperparameter tuning and a logging message related to KV cache full errors was clarified.

Changelog

Click here to see the changelog

docs/backend/hyperparameter_tuning.md
- Updated error message text from 'decode out of memory happened' to 'KV cache pool is full. Retract requests.'
python/sglang/bench_one_batch_server.py
- Added example command line demonstrating new profiling flags.
- Imported run_profile from the new profiler module.
- Added --profile and --profile-by-stage arguments to BenchArgs and its CLI parser.
- Changed time.perf_counter() to time.time() in the server launch wait loop.
- Added profile and profile_by_stage parameters to run_one_case function.
- Integrated run_profile call within run_one_case when --profile is enabled.
- Modified throughput print statements to use lowercase ('last generation throughput', 'input throughput').
- Included the generated profile trace link in the return value of run_one_case.
- Updated run_benchmark to handle the new return value and store profile links.
- Modified the benchmark report summary table to include a 'profile' column with links when profiling is enabled.
- Simplified the GitHub step summary output.
python/sglang/profiler.py
- New file implementing the standalone profiler script and the core run_profile logic.
- Defines _run_profile to interact with the server's profiling API, handle output directories, and dump server info.
- Provides a command-line interface for running profiling with various options (URL, output dir, steps, activities, stage-based profiling).
python/sglang/srt/entrypoints/http_server.py
- Updated the /start_profile API endpoint handler (start_profile_async) to accept and pass the new profile_by_stage parameter to the scheduler.
python/sglang/srt/managers/expert_location.py
- Removed a redundant logger.info call.
python/sglang/srt/managers/io_struct.py
- Added profile_by_stage field to ProfileReqInput and ProfileReq models.
- Generalized the type hint for the activities list in profile request models from a Literal union to List[str].
python/sglang/srt/managers/scheduler.py
- Added new attributes to track stage-based profiling state (profiler_target_prefill_ct, profiler_decode_ct, profiler_prefill_ct, profiler_decode_ct, profile_by_stage, profile_steps, profile_in_progress).
- Corrected the placement of self.forward_sleep_time = None initialization.
- Updated the log message for KV cache full from 'Decode out of memory happened' to 'KV cache pool is full. Retract requests.'.
- Replaced the old profiler check in run_batch with a call to the new _profile_batch_predicate method.
- Renamed the start_profile method to init_profile and changed its behavior to initialize profiling parameters.
- Added a new start_profile method that takes an optional stage argument and handles the actual starting of the torch profiler or RPD profiler.
- Implemented RPD profiler setup, start, stop, flush, and conversion to chrome trace format.
- Modified the stop_profile method to handle stopping based on stage and export traces with stage suffixes.
- Added the _profile_batch_predicate method to manage starting/stopping profiling based on stage and step counts.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Code runs fast, or slow,
Profiling helps us know,
Where cycles go.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces significant improvements to the profiling capabilities and integrates them into bench_one_batch_server.py. The addition of stage-by-stage profiling and RPD profiler support is a valuable enhancement.

Overall, the changes are well-structured, especially the new profiler.py and the refactoring in scheduler.py to handle different profiling modes. However, there are a few areas that need attention, particularly regarding the efficiency of the benchmarking script when profiling is enabled, and potential issues with the RPD profiler's intermediate file handling.

No explicit style guide was provided, so feedback is based on PEP 8 for Python and general best practices.

Summary of Findings

Efficiency of Benchmarking with Profiling: The bench_one_batch_server.py script runs benchmark cases twice when profiling is enabled, potentially doubling execution time. Profiling should ideally be integrated into a single run.
RPD Profiler Intermediate File Handling: The RPD profiler in scheduler.py uses a non-namespaced trace.rpd file in the current working directory, which could lead to conflicts if multiple instances run concurrently on the same host with the same CWD.
Profiler Configuration in Benchmark Script: In bench_one_batch_server.py, num_steps and activities for profiling are hardcoded. Making them configurable via BenchArgs would improve flexibility.
Stage-Based Profiling Step Count: The logic in _profile_batch_predicate for stage-based profiling might profile num_steps + 1 iterations instead of the intended num_steps due to the stopping condition.
Potential Typo in Scheduler: There's a potential typo in scheduler.py when resetting RPD profiler path attribute (self.rpd_profiler_path = None vs self.rpd_profile_path = None).

Merge Readiness

This pull request introduces valuable profiling enhancements. However, there are several high and medium severity issues identified, particularly concerning the efficiency of the benchmark script when profiling is active and the handling of intermediate files by the RPD profiler. I recommend addressing these issues before merging. I am unable to approve pull requests, so please ensure these changes are reviewed and approved by a maintainer.

python/sglang/bench_one_batch_server.py

python/sglang/srt/managers/scheduler.py

python/sglang/bench_one_batch_server.py

python/sglang/srt/managers/scheduler.py

…l-project#6787)

Merge branch 'sgl_20250610_sync_tag047 of git@code.alipay.com:Theta/SGLang.git into main https://code.alipay.com/Theta/SGLang/pull_requests/52 Reviewed-by: 剑川 <jianchuan.gys@antgroup.com> * [Bugfix] Fix slice operation when chunk size mismatch (sgl-project#6697) * [Bugfix] Fix ChatCompletion endpoint of mini_lb when stream is set (sgl-project#6703) * [CI] Fix setup of disaggregation with different tp (sgl-project#6706) * [PD] Remove Unnecessary Exception Handling for FastQueue.get() (sgl-project#6712) * Fuse routed_scaling_factor in DeepSeek (sgl-project#6710) * Overlap two kernels in DeepSeek with communication (sgl-project#6711) * Minor refactor two-batch overlap (sgl-project#6682) * Speed up when having padding tokens two-batch overlap (sgl-project#6668) * [Feature] Support Flashinfer fp8 blockwise GEMM kernel on Blackwell (sgl-project#6479) * Fix LoRA bench (sgl-project#6719) * temp * Fix PP for Qwen3 MoE (sgl-project#6709) * [feat] triton kernel for get_last_loc (sgl-project#6676) * [fix] more mem for draft_extend cuda_graph (sgl-project#6726) * [PD] bug fix: Update status if nixl receiver send a a dummy req. (sgl-project#6720) * Tune memory arguments on B200 (sgl-project#6718) * Add DeepSeek-R1-0528 function call chat template (sgl-project#6725) * refactor(tool call): Fix BaseFormatDetector tool_index issue and refactor `parse_streaming_increment` (sgl-project#6715) * Add draft extend CUDA graph for Triton backend (sgl-project#6705) * refactor apply_w8a8_block_fp8_linear in fp (sgl-project#6545) * [PD] Support completion endpoint (sgl-project#6729) * PD Rust LB (PO2) (sgl-project#6437) * Super tiny enable sole usage of expert distribution metrics and update doc (sgl-project#6680) * Support picking variants of EPLB algorithms (sgl-project#6728) * Support tuning DeepEP configs (sgl-project#6742) * [test] add ut and bm for get_last_loc (sgl-project#6746) * Fix mem_fraction_static for AMD CI (sgl-project#6748) * [fix][RL] Fix DeepSeekV3ForCausalLM.post_load_weights for multiple update weight (sgl-project#6265) * Improve EPLB logical to physical dispatch map (sgl-project#6727) * Update DeepSeek-R1-0528 function call chat template (sgl-project#6765) * [PD] Optimize time out logic and add env var doc for mooncake (sgl-project#6761) * Fix aiohttp 'Chunk too big' in bench_serving (sgl-project#6737) * Support sliding window in triton backend (sgl-project#6509) * Fix shared experts fusion error (sgl-project#6289) * Fix one bug in the grouped-gemm triton kernel (sgl-project#6772) * update llama4 chat template and pythonic parser (sgl-project#6679) * feat(tool call): Enhance Llama32Detector for improved JSON parsing in non-stream (sgl-project#6784) * Support token-level quantization for EP MoE (sgl-project#6782) * Temporarily lower mmlu threshold for triton sliding window backend (sgl-project#6785) * ci: relax test_function_call_required (sgl-project#6786) * Add intel_amx backend for Radix Attention for CPU (sgl-project#6408) * Fix incorrect LoRA weight loading for fused gate_up_proj (sgl-project#6734) * fix(PD-disaggregation): Can not get local ip (sgl-project#6792) * [FIX] mmmu bench serving result display error (sgl-project#6525) (sgl-project#6791) * Bump torch to 2.7.0 (sgl-project#6788) * chore: bump sgl-kernel v0.1.5 (sgl-project#6794) * Improve profiler and integrate profiler in bench_one_batch_server (sgl-project#6787) * chore: upgrade sgl-kernel v0.1.5 (sgl-project#6795) * [Minor] Always append newline after image token when parsing chat message (sgl-project#6797) * Update CI tests for Llama4 models (sgl-project#6421) * [Feat] Enable PDL automatically on Hopper architecture (sgl-project#5981) * chore: update blackwell docker (sgl-project#6800) * misc: cache is_hopper_arch (sgl-project#6799) * Remove contiguous before Flashinfer groupwise fp8 gemm (sgl-project#6804) * Correctly abort the failed grammar requests & Improve the handling of abort (sgl-project#6803) * [EP] Add cuda kernel for moe_ep_pre_reorder (sgl-project#6699) * Add draft extend CUDA graph for flashinfer backend (sgl-project#6805) * Refactor CustomOp to avoid confusing bugs (sgl-project#5382) * Tiny log prefill time (sgl-project#6780) * Tiny fix EPLB assertion about rebalancing period and recorder window size (sgl-project#6813) * Add simple utility to dump tensors for debugging (sgl-project#6815) * Fix profiles do not have consistent names (sgl-project#6811) * Speed up rebalancing when using non-static dispatch algorithms (sgl-project#6812) * [1/2] Add Kernel support for Cutlass based Fused FP4 MoE (sgl-project#6093) * [Router] Fix k8s Service Discovery (sgl-project#6766) * Add CPU optimized kernels for topk and rope fusions (sgl-project#6456) * fix new_page_count_next_decode (sgl-project#6671) * Fix wrong weight reference in dynamic EPLB (sgl-project#6818) * Minor add metrics to expert location updater (sgl-project#6816) * [Refactor] Rename `n_share_experts_fusion` as `num_fused_shared_experts` (sgl-project#6735) * [FEAT] Add transformers backend support (sgl-project#5929) * [fix] recover auto-dispatch for rmsnorm and rope (sgl-project#6745) * fix ep_moe_reorder kernel bugs (sgl-project#6858) * [Refactor] Multimodal data processing for VLM (sgl-project#6659) * Decoder-only Scoring API (sgl-project#6460) * feat: add dp-rank to KV events (sgl-project#6852) * Set `num_fused_shared_experts` as `num_shared_experts` when shared_experts fusion is not disabled (sgl-project#6736) * Fix one missing arg in DeepEP (sgl-project#6878) * Support LoRA in TestOpenAIVisionServer and fix fused kv_proj loading bug. (sgl-project#6861) * support 1 shot allreduce in 1-node and 2-node using mscclpp (sgl-project#6277) * Fix Qwen3MoE missing token padding optimization (sgl-project#6820) * Tiny update error hints (sgl-project#6846) * Support layerwise rebalancing experts (sgl-project#6851) * Tiny allow profiler API to auto create directory (sgl-project#6865) * Support Blackwell DeepEP docker images (sgl-project#6868) * [EP] Add cuda kernel for moe_ep_post_reorder (sgl-project#6837) * [theta]merge 0605 * oai: fix openAI client error with single request via batch api (sgl-project#6170) * [PD] Fix potential perf spike caused by tracker gc and optimize doc (sgl-project#6764) * Use deepgemm instead of triton for fused_qkv_a_proj_with_mqa (sgl-project#6890) * [CUTLASS-FP4-MOE] Introduce CutlassMoEParams class for easy initialization of Cutlass Grouped Gems Metadata (sgl-project#6887) * bugfix(OAI): Fix image_data processing for jinja chat templates (sgl-project#6877) * [CPU] enable CI for PRs, add Dockerfile and auto build task (sgl-project#6458) * AITER backend extension and workload optimizations (sgl-project#6838) * [theta]merge * [theta]merge * [Feature] Support Flashinfer fmha on Blackwell (sgl-project#6930) * Fix a bug in abort & Improve docstrings for abort (sgl-project#6931) * Tiny support customize DeepEP max dispatch tokens per rank (sgl-project#6934) * Sync the changes on cuda graph runners (sgl-project#6932) * [PD] Optimize transfer queue forward logic for dummy rank (sgl-project#6922) * [Refactor] image data process in bench_serving (sgl-project#6879) * [fix] logical_to_all_physical_map index 256 is out of bounds in EP parallel. (sgl-project#6767) * Add triton fused moe kernel config for E=257 on B200 (sgl-project#6939) * [sgl-kernel] update deepgemm (sgl-project#6942) * chore: bump sgl-kernel v0.1.6 (sgl-project#6943) * Minor compile fused topk (sgl-project#6944) * [Bugfix] pipeline parallelism and Eagle Qwen2 (sgl-project#6910) * Tiny re-introduce profile id logging (sgl-project#6912) * Add triton version as a fused_moe_triton config search key to avoid performace decrease in different Triton version (sgl-project#5955) * reduce torch.zeros overhead in moe align block size kernel (sgl-project#6369) * chore: upgrade sgl-kernel v0.1.6 (sgl-project#6945) * add fbgemm moe grouped gemm kernel benchmark (sgl-project#6924) * [Docker] Add docker file for SGL Router (sgl-project#6915) * Disabling mixed chunked prefill when eagle is enabled (sgl-project#6874) * Add canary for EPLB rebalancing (sgl-project#6895) * Refactor global_server_args_dict (sgl-project#6866) * Fuse routed scaling factor in topk_reduce kernel (sgl-project#6220) * Update server timeout time in AMD CI. (sgl-project#6953) * [misc] add is_cpu() (sgl-project#6950) * Add H20 fused MoE kernel tuning configs for DeepSeek-R1/V3 (sgl-project#6885) * Add a CUDA kernel for fusing mapping and weighted sum for MoE. (sgl-project#6916) * chore: bump sgl-kernel v0.1.6.post1 (sgl-project#6955) * chore: upgrade sgl-kernel v0.1.6.post1 (sgl-project#6957) * [DeepseekR1-FP4] Add Support for nvidia/DeepSeekR1-FP4 model (sgl-project#6853) * Revert "Fuse routed scaling factor in topk_reduce kernel (sgl-project#6220)" (sgl-project#6968) * [AMD] Add more tests to per-commit-amd (sgl-project#6926) * chore: bump sgl-kernel v0.1.7 (sgl-project#6963) * Slightly improve the sampler to skip unnecessary steps (sgl-project#6956) * rebase h20 fused_moe config (sgl-project#6966) * Fix CI and triton moe Configs (sgl-project#6974) * Remove unnecessary kernels of num_token_non_padded (sgl-project#6965) * Extend cuda graph capture bs for B200 (sgl-project#6937) * Fuse routed scaling factor in deepseek (sgl-project#6970) * Sync cuda graph runners (sgl-project#6976) * Fix draft extend ut stability with flush cache (sgl-project#6979) * Fix triton sliding window test case (sgl-project#6981) * Fix expert distribution dumping causes OOM (sgl-project#6967) * Minor remove one kernel for DeepSeek (sgl-project#6977) * [perf][sgl-kernel] extend cutlass_mla_decode to support num_head < 128 (sgl-project#6929) * Enable more unit tests for AMD CI. (sgl-project#6983) * Use torch.compile to fuse flash attention decode metadata preparation (sgl-project#6973) * Eliminate stream sync to speed up LoRA batch init (sgl-project#6960) * support qwen3 emebedding (sgl-project#6990) * Fix torch profiler bugs for bench_offline_throughput.py (sgl-project#6557) * chore: upgrade flashinfer v0.2.6.post1 jit (sgl-project#6958) * cleanup tmp dir (sgl-project#7007) * chore: update pr test xeon (sgl-project#7008) * Fix cutlass MLA gets almost zero accuracy (sgl-project#6998) * Update amd nightly models CI. (sgl-project#6992) * feat: add direct routing strategy to DP worker (sgl-project#6884) * Fallback to lower triton version for unfound fused moe configs (sgl-project#7013) * Fix torchvision version for Blackwell (sgl-project#7015) * Simplify prepare_extend_after_decode (sgl-project#6987) * Migrate to assertEqual (sgl-project#6741) * Fix torch version in blackwell dockerfile (sgl-project#7017) * chore: update pr test xeon (sgl-project#7018) * Update default settings for blackwell (sgl-project#7023) * Support both approximate and exact expert distribution collection (sgl-project#6964) * Add decode req pool (sgl-project#6980) * [theta]merge 0610 * [theta]merge 0610 * [CI] Add CI workflow for sgl-router docker build (sgl-project#7027) * Fix fused_moe triton configs (sgl-project#7029) * CPU: map changes from developing branch in sgl-kernel (sgl-project#6833) * chore: bump v0.4.7 (sgl-project#7038) * Update README.md (sgl-project#7040)

merrymercy added 4 commits May 30, 2025 16:44

Add profiler

af7d234

improve profiler

33cd2bf

profile by stage

0a162cc

Fix

387789a

merrymercy requested review from Ying1123, hnyls2002, xiezhq-hermann and zhaochenyang20 as code owners May 31, 2025 02:33

gemini-code-assist bot reviewed May 31, 2025

View reviewed changes

gemini-code-assist bot suggested changes May 31, 2025

View reviewed changes

Fix

a1653b1

merrymercy requested review from zhyncs and ispobock as code owners May 31, 2025 02:54

merrymercy added 2 commits May 30, 2025 21:14

fix

7b0f4c2

Simplify tests

bd1774d

merrymercy requested a review from ByronHsu as a code owner May 31, 2025 04:44

merrymercy added 6 commits May 31, 2025 13:40

open

ef0e4f3

Simplify test_pp

1869901

Fix flakytests

1a616b6

Simplify ci

20ec0f7

Fix

6efe964

Fix calibration

82784b5

merrymercy merged commit 2d72fc4 into main May 31, 2025
5 of 36 checks passed

merrymercy deleted the lianmin/sync-code branch May 31, 2025 22:53

Edenzzzz pushed a commit to Edenzzzz/sglang that referenced this pull request Jun 2, 2025

Improve profiler and integrate profiler in bench_one_batch_server (sg…

5403644

…l-project#6787)

Layssy pushed a commit to Layssy/sglang-iaas that referenced this pull request Jun 9, 2025

Improve profiler and integrate profiler in bench_one_batch_server (sg…

09f082d

…l-project#6787)

hubertlu-tw mentioned this pull request Jun 12, 2025

[AMD] Update profiling doc for rpd instructions #7110

Draft

6 tasks

xwu-intel pushed a commit to xwu-intel/sglang that referenced this pull request Jun 17, 2025

Improve profiler and integrate profiler in bench_one_batch_server (sg…

225d7b8

…l-project#6787)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Improve profiler and integrate profiler in bench_one_batch_server #6787

Improve profiler and integrate profiler in bench_one_batch_server #6787

Uh oh!

merrymercy commented May 31, 2025 •

edited

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Improve profiler and integrate profiler in bench_one_batch_server #6787

Improve profiler and integrate profiler in bench_one_batch_server #6787

Uh oh!

Conversation

merrymercy commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Changelog

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Summary of Findings

Merge Readiness

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

merrymercy commented May 31, 2025 •

edited

Loading