chore: update blackwell docker #6800
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Hello @zhyncs, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
Summary of Changes
Hello team, gemini-code-assist here to provide a summary of this pull request. The intent of this PR, based on the title and the changes observed in the Dockerfile.blackwell, is to update the base Docker image and associated dependencies for the Blackwell environment. The primary change involves switching from a PyTorch manylinux builder image to an NVIDIA Triton Server image, which necessitates updating the package management system from yum to apt and adjusting the installation commands accordingly. Several Python package installations have also been updated, including the addition of the flashinfer library and the inclusion of the --break-system-packages flag to avoid potential conflicts.
Highlights
- Base Image Change: The Dockerfile now uses
nvcr.io/nvidia/tritonserver:25.05-py3-minas the base image instead ofpytorch/manylinux2_28-builder:cuda12.8. - Package Manager Switch: The system package installation commands have been updated to use
aptinstead ofyum, reflecting the change in the base operating system environment. - Dependency Updates: Several system and Python dependencies are installed via
aptandpip, including common development tools and libraries likelibibverbs-dev,rdma-core,infiniband-diags,openssh-server,perftest,lsof,zsh,ccache,tmux,htop,git-lfs,tree. TheflashinferPython package has also been added. - Pip Install Flags: The
--break-system-packagesflag has been added to mostpip3 installcommands to handle potential conflicts with system-installed Python packages in the new base image.
Changelog
- docker/Dockerfile.blackwell
- Changed base image from
pytorch/manylinux2_28-builder:cuda12.8tonvcr.io/nvidia/tritonserver:25.05-py3-min(line 1). - Replaced
yum installblock with anapt installblock, adding and updating various system dependencies (lines 7-18, replacing lines 19-22). - Added
--break-system-packagesflag topip3 installcommands for torch, sgl_kernel, setuptools/wheel/scikit-build-core, sglang, and nvidia-nccl-cu12 (lines 21, 23-24, 27, 29). - Added a new
pip3 installcommand for theflashinferPython package (line 31). - Added
--break-system-packagesflag to the finalpip3 installcommand for minimal Python packages (line 42).
- Changed base image from
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Base image shifts,
From builder to Triton,
New tools installed.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request updates the Dockerfile.blackwell by changing the base image to nvcr.io/nvidia/tritonserver:25.05-py3-min and adjusting package installations accordingly (from yum to apt, adding --break-system-packages for pip). This aligns the Dockerfile with a Debian-based environment, likely targeting NVIDIA Triton for deployment.
While the changes are generally moving in the right direction for a Triton-based environment, there are a few points regarding Python package compatibility, sudo usage, and apt command structure that could be improved.
It would also be beneficial if the PR description included a brief note on the motivation for changing the base image, for better context to reviewers.
Summary of Findings
- Python Wheel Compatibility: The installed
sgl_kernel(cp39-abi3) andflashinfer_python(cp38-abi3) wheels are being used with a Python 3.10 base image. Whileabi3provides forward compatibility, it's recommended to verify if Python 3.10 specific wheels are available or if thisabi3usage is confirmed to be optimal and stable. (Commented) sudoPackage Installation: Thesudopackage is installed. Its necessity in the Docker container should be clarified, as it can increase the attack surface if not strictly required. (Commented)- APT Package Installation Command: The
apt installblock can be improved by usingapt-getand--no-install-recommendsfor better scripting practice and potentially smaller image size. (Commented) - Unpinned
wheelDependency: Thewheelpackage is installed without a pinned version (changed fromwheel==0.41.0), whilesetuptoolsremains pinned. For consistency in build reproducibility, consider pinningwheelas well. (Not commented due to severity settings) - Necessity of
software-properties-common: Thesoftware-properties-commonpackage is installed. It's typically used for managing PPAs. If no PPAs are added, this package might be unnecessary and could be removed to reduce image size. (Not commented due to severity settings)
Merge Readiness
This pull request makes significant updates to the Dockerfile.blackwell. Before merging, I recommend addressing the comments regarding Python wheel compatibility and the use of sudo. The suggestion for refining the apt-get command should also be considered for improved maintainability and image optimization. Given these points, I am requesting changes. I am unable to approve this pull request; please ensure other reviewers approve these changes before merging.
Comment on lines
+7
to
+18
| RUN echo 'tzdata tzdata/Areas select America' | debconf-set-selections \ | ||
| && echo 'tzdata tzdata/Zones/America select Los_Angeles' | debconf-set-selections \ | ||
| && apt update -y \ | ||
| && apt install software-properties-common -y \ | ||
| && apt install python3 python3-pip -y \ | ||
| && apt install curl git sudo libibverbs-dev -y \ | ||
| && apt install rdma-core infiniband-diags openssh-server perftest -y \ | ||
| && apt install lsof zsh ccache tmux htop git-lfs tree -y \ | ||
| && python3 --version \ | ||
| && python3 -m pip --version \ | ||
| && rm -rf /var/lib/apt/lists/* \ | ||
| && apt clean |
There was a problem hiding this comment.
The apt package installation block is functional, but could be slightly improved for maintainability and potentially image size:
- It's common practice to use
apt-getin scripts rather thanaptfor more stable behavior across versions. - Adding the
--no-install-recommendsflag toapt-get installcan help reduce the size of the Docker image by avoiding installation of unnecessary recommended packages. - Listing all packages in a single
apt-get install -y --no-install-recommends ...command (afterapt-get update) can improve readability within this multi-lineRUNinstruction.
Could you consider these adjustments?
RUN echo 'tzdata tzdata/Areas select America' | debconf-set-selections \
&& echo 'tzdata tzdata/Zones/America select Los_Angeles' | debconf-set-selections \
&& apt-get update -y \
&& apt-get install -y --no-install-recommends \
software-properties-common \
python3 \
python3-pip \
curl \
git \
sudo \
libibverbs-dev \
rdma-core \
infiniband-diags \
openssh-server \
perftest \
lsof \
zsh \
ccache \
tmux \
htop \
git-lfs \
tree \
&& python3 --version \
&& python3 -m pip --version \
&& rm -rf /var/lib/apt/lists/* \
&& apt-get clean
| && apt update -y \ | ||
| && apt install software-properties-common -y \ | ||
| && apt install python3 python3-pip -y \ | ||
| && apt install curl git sudo libibverbs-dev -y \ |
There was a problem hiding this comment.
The sudo package is being installed. Could you clarify the need for sudo within this Docker container? If the container is intended to run as root, sudo is generally unnecessary. If it's for a non-root user, its inclusion should be carefully considered as it can increase the potential attack surface.
Comment on lines
+23
to
+31
| RUN pip3 install https://github.com/sgl-project/whl/releases/download/v0.1.5/sgl_kernel-0.1.5+cu128-cp39-abi3-manylinux2014_x86_64.whl --break-system-packages \ | ||
| && pip3 install setuptools==75.0.0 wheel scikit-build-core --break-system-packages | ||
|
|
||
| RUN git clone --depth=1 https://github.com/sgl-project/sglang.git \ | ||
| && cd sglang && pip3 install -e "python[blackwell]" | ||
| && cd sglang && pip3 install -e "python[blackwell]" --break-system-packages | ||
|
|
||
| RUN pip3 install nvidia-nccl-cu12==2.26.2.post1 --force-reinstall --no-deps | ||
| RUN pip3 install nvidia-nccl-cu12==2.26.2.post1 --force-reinstall --no-deps --break-system-packages | ||
|
|
||
| ENV DEBIAN_FRONTEND=interactive | ||
| RUN pip3 install https://download.pytorch.org/whl/cu128/flashinfer/flashinfer_python-0.2.5%2Bcu128torch2.7-cp38-abi3-linux_x86_64.whl --break-system-packages |
There was a problem hiding this comment.
The new base image (nvcr.io/nvidia/tritonserver:25.05-py3-min) uses Python 3.10. The sgl_kernel wheel being installed is cp39-abi3 and flashinfer_python is cp38-abi3.
While abi3 wheels are designed for cross-version compatibility (so these should work with Python 3.10), it's worth confirming:
- Are Python 3.10-specific wheels (
cp310) available for these packages? If so, they might offer better optimization or avoid potential subtle compatibility issues. - If
cp310wheels are not available, has theabi3compatibility of these specificcp38/cp39wheels been thoroughly vetted with the Python 3.10 environment provided by this Triton base image?
This ensures optimal performance and stability.
Edenzzzz
pushed a commit
to Edenzzzz/sglang
that referenced
this pull request
Jun 2, 2025
Layssy
pushed a commit
to Layssy/sglang-iaas
that referenced
this pull request
Jun 9, 2025
xwu-intel
pushed a commit
to xwu-intel/sglang
that referenced
this pull request
Jun 17, 2025
walker-ai
pushed a commit
to walker-ai/sglang
that referenced
this pull request
Jul 8, 2025
Merge branch 'sgl_20250610_sync_tag047 of git@code.alipay.com:Theta/SGLang.git into main https://code.alipay.com/Theta/SGLang/pull_requests/52 Reviewed-by: 剑川 <jianchuan.gys@antgroup.com> * [Bugfix] Fix slice operation when chunk size mismatch (sgl-project#6697) * [Bugfix] Fix ChatCompletion endpoint of mini_lb when stream is set (sgl-project#6703) * [CI] Fix setup of disaggregation with different tp (sgl-project#6706) * [PD] Remove Unnecessary Exception Handling for FastQueue.get() (sgl-project#6712) * Fuse routed_scaling_factor in DeepSeek (sgl-project#6710) * Overlap two kernels in DeepSeek with communication (sgl-project#6711) * Minor refactor two-batch overlap (sgl-project#6682) * Speed up when having padding tokens two-batch overlap (sgl-project#6668) * [Feature] Support Flashinfer fp8 blockwise GEMM kernel on Blackwell (sgl-project#6479) * Fix LoRA bench (sgl-project#6719) * temp * Fix PP for Qwen3 MoE (sgl-project#6709) * [feat] triton kernel for get_last_loc (sgl-project#6676) * [fix] more mem for draft_extend cuda_graph (sgl-project#6726) * [PD] bug fix: Update status if nixl receiver send a a dummy req. (sgl-project#6720) * Tune memory arguments on B200 (sgl-project#6718) * Add DeepSeek-R1-0528 function call chat template (sgl-project#6725) * refactor(tool call): Fix BaseFormatDetector tool_index issue and refactor `parse_streaming_increment` (sgl-project#6715) * Add draft extend CUDA graph for Triton backend (sgl-project#6705) * refactor apply_w8a8_block_fp8_linear in fp (sgl-project#6545) * [PD] Support completion endpoint (sgl-project#6729) * PD Rust LB (PO2) (sgl-project#6437) * Super tiny enable sole usage of expert distribution metrics and update doc (sgl-project#6680) * Support picking variants of EPLB algorithms (sgl-project#6728) * Support tuning DeepEP configs (sgl-project#6742) * [test] add ut and bm for get_last_loc (sgl-project#6746) * Fix mem_fraction_static for AMD CI (sgl-project#6748) * [fix][RL] Fix DeepSeekV3ForCausalLM.post_load_weights for multiple update weight (sgl-project#6265) * Improve EPLB logical to physical dispatch map (sgl-project#6727) * Update DeepSeek-R1-0528 function call chat template (sgl-project#6765) * [PD] Optimize time out logic and add env var doc for mooncake (sgl-project#6761) * Fix aiohttp 'Chunk too big' in bench_serving (sgl-project#6737) * Support sliding window in triton backend (sgl-project#6509) * Fix shared experts fusion error (sgl-project#6289) * Fix one bug in the grouped-gemm triton kernel (sgl-project#6772) * update llama4 chat template and pythonic parser (sgl-project#6679) * feat(tool call): Enhance Llama32Detector for improved JSON parsing in non-stream (sgl-project#6784) * Support token-level quantization for EP MoE (sgl-project#6782) * Temporarily lower mmlu threshold for triton sliding window backend (sgl-project#6785) * ci: relax test_function_call_required (sgl-project#6786) * Add intel_amx backend for Radix Attention for CPU (sgl-project#6408) * Fix incorrect LoRA weight loading for fused gate_up_proj (sgl-project#6734) * fix(PD-disaggregation): Can not get local ip (sgl-project#6792) * [FIX] mmmu bench serving result display error (sgl-project#6525) (sgl-project#6791) * Bump torch to 2.7.0 (sgl-project#6788) * chore: bump sgl-kernel v0.1.5 (sgl-project#6794) * Improve profiler and integrate profiler in bench_one_batch_server (sgl-project#6787) * chore: upgrade sgl-kernel v0.1.5 (sgl-project#6795) * [Minor] Always append newline after image token when parsing chat message (sgl-project#6797) * Update CI tests for Llama4 models (sgl-project#6421) * [Feat] Enable PDL automatically on Hopper architecture (sgl-project#5981) * chore: update blackwell docker (sgl-project#6800) * misc: cache is_hopper_arch (sgl-project#6799) * Remove contiguous before Flashinfer groupwise fp8 gemm (sgl-project#6804) * Correctly abort the failed grammar requests & Improve the handling of abort (sgl-project#6803) * [EP] Add cuda kernel for moe_ep_pre_reorder (sgl-project#6699) * Add draft extend CUDA graph for flashinfer backend (sgl-project#6805) * Refactor CustomOp to avoid confusing bugs (sgl-project#5382) * Tiny log prefill time (sgl-project#6780) * Tiny fix EPLB assertion about rebalancing period and recorder window size (sgl-project#6813) * Add simple utility to dump tensors for debugging (sgl-project#6815) * Fix profiles do not have consistent names (sgl-project#6811) * Speed up rebalancing when using non-static dispatch algorithms (sgl-project#6812) * [1/2] Add Kernel support for Cutlass based Fused FP4 MoE (sgl-project#6093) * [Router] Fix k8s Service Discovery (sgl-project#6766) * Add CPU optimized kernels for topk and rope fusions (sgl-project#6456) * fix new_page_count_next_decode (sgl-project#6671) * Fix wrong weight reference in dynamic EPLB (sgl-project#6818) * Minor add metrics to expert location updater (sgl-project#6816) * [Refactor] Rename `n_share_experts_fusion` as `num_fused_shared_experts` (sgl-project#6735) * [FEAT] Add transformers backend support (sgl-project#5929) * [fix] recover auto-dispatch for rmsnorm and rope (sgl-project#6745) * fix ep_moe_reorder kernel bugs (sgl-project#6858) * [Refactor] Multimodal data processing for VLM (sgl-project#6659) * Decoder-only Scoring API (sgl-project#6460) * feat: add dp-rank to KV events (sgl-project#6852) * Set `num_fused_shared_experts` as `num_shared_experts` when shared_experts fusion is not disabled (sgl-project#6736) * Fix one missing arg in DeepEP (sgl-project#6878) * Support LoRA in TestOpenAIVisionServer and fix fused kv_proj loading bug. (sgl-project#6861) * support 1 shot allreduce in 1-node and 2-node using mscclpp (sgl-project#6277) * Fix Qwen3MoE missing token padding optimization (sgl-project#6820) * Tiny update error hints (sgl-project#6846) * Support layerwise rebalancing experts (sgl-project#6851) * Tiny allow profiler API to auto create directory (sgl-project#6865) * Support Blackwell DeepEP docker images (sgl-project#6868) * [EP] Add cuda kernel for moe_ep_post_reorder (sgl-project#6837) * [theta]merge 0605 * oai: fix openAI client error with single request via batch api (sgl-project#6170) * [PD] Fix potential perf spike caused by tracker gc and optimize doc (sgl-project#6764) * Use deepgemm instead of triton for fused_qkv_a_proj_with_mqa (sgl-project#6890) * [CUTLASS-FP4-MOE] Introduce CutlassMoEParams class for easy initialization of Cutlass Grouped Gems Metadata (sgl-project#6887) * bugfix(OAI): Fix image_data processing for jinja chat templates (sgl-project#6877) * [CPU] enable CI for PRs, add Dockerfile and auto build task (sgl-project#6458) * AITER backend extension and workload optimizations (sgl-project#6838) * [theta]merge * [theta]merge * [Feature] Support Flashinfer fmha on Blackwell (sgl-project#6930) * Fix a bug in abort & Improve docstrings for abort (sgl-project#6931) * Tiny support customize DeepEP max dispatch tokens per rank (sgl-project#6934) * Sync the changes on cuda graph runners (sgl-project#6932) * [PD] Optimize transfer queue forward logic for dummy rank (sgl-project#6922) * [Refactor] image data process in bench_serving (sgl-project#6879) * [fix] logical_to_all_physical_map index 256 is out of bounds in EP parallel. (sgl-project#6767) * Add triton fused moe kernel config for E=257 on B200 (sgl-project#6939) * [sgl-kernel] update deepgemm (sgl-project#6942) * chore: bump sgl-kernel v0.1.6 (sgl-project#6943) * Minor compile fused topk (sgl-project#6944) * [Bugfix] pipeline parallelism and Eagle Qwen2 (sgl-project#6910) * Tiny re-introduce profile id logging (sgl-project#6912) * Add triton version as a fused_moe_triton config search key to avoid performace decrease in different Triton version (sgl-project#5955) * reduce torch.zeros overhead in moe align block size kernel (sgl-project#6369) * chore: upgrade sgl-kernel v0.1.6 (sgl-project#6945) * add fbgemm moe grouped gemm kernel benchmark (sgl-project#6924) * [Docker] Add docker file for SGL Router (sgl-project#6915) * Disabling mixed chunked prefill when eagle is enabled (sgl-project#6874) * Add canary for EPLB rebalancing (sgl-project#6895) * Refactor global_server_args_dict (sgl-project#6866) * Fuse routed scaling factor in topk_reduce kernel (sgl-project#6220) * Update server timeout time in AMD CI. (sgl-project#6953) * [misc] add is_cpu() (sgl-project#6950) * Add H20 fused MoE kernel tuning configs for DeepSeek-R1/V3 (sgl-project#6885) * Add a CUDA kernel for fusing mapping and weighted sum for MoE. (sgl-project#6916) * chore: bump sgl-kernel v0.1.6.post1 (sgl-project#6955) * chore: upgrade sgl-kernel v0.1.6.post1 (sgl-project#6957) * [DeepseekR1-FP4] Add Support for nvidia/DeepSeekR1-FP4 model (sgl-project#6853) * Revert "Fuse routed scaling factor in topk_reduce kernel (sgl-project#6220)" (sgl-project#6968) * [AMD] Add more tests to per-commit-amd (sgl-project#6926) * chore: bump sgl-kernel v0.1.7 (sgl-project#6963) * Slightly improve the sampler to skip unnecessary steps (sgl-project#6956) * rebase h20 fused_moe config (sgl-project#6966) * Fix CI and triton moe Configs (sgl-project#6974) * Remove unnecessary kernels of num_token_non_padded (sgl-project#6965) * Extend cuda graph capture bs for B200 (sgl-project#6937) * Fuse routed scaling factor in deepseek (sgl-project#6970) * Sync cuda graph runners (sgl-project#6976) * Fix draft extend ut stability with flush cache (sgl-project#6979) * Fix triton sliding window test case (sgl-project#6981) * Fix expert distribution dumping causes OOM (sgl-project#6967) * Minor remove one kernel for DeepSeek (sgl-project#6977) * [perf][sgl-kernel] extend cutlass_mla_decode to support num_head < 128 (sgl-project#6929) * Enable more unit tests for AMD CI. (sgl-project#6983) * Use torch.compile to fuse flash attention decode metadata preparation (sgl-project#6973) * Eliminate stream sync to speed up LoRA batch init (sgl-project#6960) * support qwen3 emebedding (sgl-project#6990) * Fix torch profiler bugs for bench_offline_throughput.py (sgl-project#6557) * chore: upgrade flashinfer v0.2.6.post1 jit (sgl-project#6958) * cleanup tmp dir (sgl-project#7007) * chore: update pr test xeon (sgl-project#7008) * Fix cutlass MLA gets almost zero accuracy (sgl-project#6998) * Update amd nightly models CI. (sgl-project#6992) * feat: add direct routing strategy to DP worker (sgl-project#6884) * Fallback to lower triton version for unfound fused moe configs (sgl-project#7013) * Fix torchvision version for Blackwell (sgl-project#7015) * Simplify prepare_extend_after_decode (sgl-project#6987) * Migrate to assertEqual (sgl-project#6741) * Fix torch version in blackwell dockerfile (sgl-project#7017) * chore: update pr test xeon (sgl-project#7018) * Update default settings for blackwell (sgl-project#7023) * Support both approximate and exact expert distribution collection (sgl-project#6964) * Add decode req pool (sgl-project#6980) * [theta]merge 0610 * [theta]merge 0610 * [CI] Add CI workflow for sgl-router docker build (sgl-project#7027) * Fix fused_moe triton configs (sgl-project#7029) * CPU: map changes from developing branch in sgl-kernel (sgl-project#6833) * chore: bump v0.4.7 (sgl-project#7038) * Update README.md (sgl-project#7040)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Modifications
Checklist