Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

I used the following command to load the deepseek v3 bf16 model, but the following error occurred.

python3 -m sglang.launch_server --model-path /path/DeepSeek-V3-Base-bf16 --tp 32 --dist-init-addr ipv6:6000 --nnodes 4 --node-rank 0 --trust-remote-code --host "::" --port 12345 --dtype bfloat16

ERROR:
[2025-05-06 11:38:39 TP0] Scheduler hit an exception: Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/sglang/srt/managers/scheduler.py", line 2215, in run_scheduler_process
scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, pp_rank, dp_rank)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/sglang/srt/managers/scheduler.py", line 268, in init
self.tp_worker = TpWorkerClass(
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 64, in init
self.worker = TpModelWorker(
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/sglang/srt/managers/tp_worker.py", line 81, in init
self.model_runner = ModelRunner(
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/sglang/srt/model_executor/model_runner.py", line 190, in init
self.initialize(min_per_gpu_memory)
File "/usr/local/lib/python3.11/dist-packages/sglang/srt/model_executor/model_runner.py", line 205, in initialize
self.load_model()
File "/usr/local/lib/python3.11/dist-packages/sglang/srt/model_executor/model_runner.py", line 458, in load_model
self.model = get_model(
^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/sglang/srt/model_loader/init.py", line 22, in get_model
return loader.load_model(
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/sglang/srt/model_loader/loader.py", line 377, in load_model
model.load_weights(self._get_all_weights(model_config, model))
File "/usr/local/lib/python3.11/dist-packages/sglang/srt/models/deepseek_v2.py", line 1695, in load_weights
weights_dict[shared_expert_weight_name],
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'model.layers.3.mlp.shared_experts.down_proj.weight_scale'

I think this is because I used a model in bf16 format. Since models in bf16 format do not include weight_scale or weight_scale_inv, it caused an error when loading the model.

I cannot use the fp8 method to load the model because I used the deepseek v3 model after SFT. After SFT, I cannot quantize the model back to its original fp8 format.

Thank you.

Reproduction

deepseek v3 bf16

python3 -m sglang.launch_server --model-path /path/DeepSeek-V3-Base-bf16 --tp 32 --dist-init-addr ipv6:6000 --nnodes 4 --node-rank 0 --trust-remote-code --host "::" --port 12345 --dtype bfloat16

Environment

Python: 3.11.2 (main, May 2 2024, 11:59:08) [GCC 12.2.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA H100 80GB HBM3
GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.4, V12.4.131
CUDA Driver Version: 535.161.08
PyTorch: 2.6.0+cu124
sglang: 0.4.6.post2
sgl_kernel: 0.1.1
flashinfer_python: 0.2.5
triton: 3.2.0
transformers: 4.51.1
torchao: 0.10.0
numpy: 1.26.4
aiohttp: 3.11.12
fastapi: 0.115.8
hf_transfer: 0.1.9
huggingface_hub: 0.30.2
interegular: 0.3.3
modelscope: 1.22.3
orjson: 3.10.15
outlines: 0.1.11
packaging: 24.1
psutil: 7.0.0
pydantic: 2.10.6
python-multipart: 0.0.20
pyzmq: 26.2.1
uvicorn: 0.34.0
uvloop: 0.21.0
vllm: 0.7.2
xgrammar: 0.1.17
openai: 1.63.0
tiktoken: 0.9.0
anthropic: 0.45.2
litellm: 1.61.5
decord: 0.6.0
NVIDIA Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 NIC10 NIC11 NIC12 NIC13 NIC14 NIC15 NIC16 NIC17 NIC18 NIC19 NIC20 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PXB PXB NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS 0-55,112-167 0 N/A
GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE NODE NODE PXB PXB NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS 0-55,112-167 0 N/A
GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE NODE NODE NODE NODE PXB PXB NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS 0-55,112-167 0 N/A
GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE NODE NODE NODE NODE NODE NODE PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS 0-55,112-167 0 N/A
GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PXB PXB NODE NODE NODE NODE NODE NODE NODE NODE NODE 56-111,168-223 1 N/A
GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE PXB PXB NODE NODE NODE NODE NODE 56-111,168-223 1 N/A
GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE PXB PXB NODE NODE NODE 56-111,168-223 1 N/A
GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE NODE NODE PXB PXB NODE 56-111,168-223 1 N/A
NIC0 PXB NODE NODE NODE SYS SYS SYS SYS X PIX NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC1 PXB NODE NODE NODE SYS SYS SYS SYS PIX X NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC2 NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE X PIX NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC3 NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE PIX X NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC4 NODE PXB NODE NODE SYS SYS SYS SYS NODE NODE NODE NODE X PIX NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC5 NODE PXB NODE NODE SYS SYS SYS SYS NODE NODE NODE NODE PIX X NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC6 NODE NODE PXB NODE SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE X PIX NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC7 NODE NODE PXB NODE SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE PIX X NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC8 NODE NODE NODE PXB SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE NODE NODE X PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC9 NODE NODE NODE PXB SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE NODE NODE PIX X SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC10 SYS SYS SYS SYS PXB NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS X PIX NODE NODE NODE NODE NODE NODE NODE NODE NODE
NIC11 SYS SYS SYS SYS PXB NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX X NODE NODE NODE NODE NODE NODE NODE NODE NODE
NIC12 SYS SYS SYS SYS NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE X PIX NODE NODE NODE NODE NODE NODE PIX
NIC13 SYS SYS SYS SYS NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE PIX X NODE NODE NODE NODE NODE NODE PIX
NIC14 SYS SYS SYS SYS NODE PXB NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE X PIX NODE NODE NODE NODE NODE
NIC15 SYS SYS SYS SYS NODE PXB NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE PIX X NODE NODE NODE NODE NODE
NIC16 SYS SYS SYS SYS NODE NODE PXB NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE X PIX NODE NODE NODE
NIC17 SYS SYS SYS SYS NODE NODE PXB NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE PIX X NODE NODE NODE
NIC18 SYS SYS SYS SYS NODE NODE NODE PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE NODE NODE X PIX NODE
NIC19 SYS SYS SYS SYS NODE NODE NODE PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE NODE NODE PIX X NODE
NIC20 SYS SYS SYS SYS NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE PIX PIX NODE NODE NODE NODE NODE NODE X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
NIC4: mlx5_4
NIC5: mlx5_5
NIC6: mlx5_6
NIC7: mlx5_7
NIC8: mlx5_8
NIC9: mlx5_9
NIC10: mlx5_10
NIC11: mlx5_11
NIC12: mlx5_12
NIC13: mlx5_13
NIC14: mlx5_14
NIC15: mlx5_15
NIC16: mlx5_16
NIC17: mlx5_17
NIC18: mlx5_18
NIC19: mlx5_19
NIC20: mlx5_20