Skip to content

update llama4 chat template and pythonic parser #6679

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
May 31, 2025
Merged

update llama4 chat template and pythonic parser #6679

merged 5 commits into from
May 31, 2025

Conversation

upfixer
Copy link
Contributor

@upfixer upfixer commented May 28, 2025 •
edited
Loading

update llama4 chat template and pythonic parser to remove <|python_start|> and <|python_end|>

Motivation

Modifications

  • updated llama4 chat template
  • updated pythonic parser to remove <|python_start|> and <|python_end|>

Checklist

gemini-code-assist[bot]
gemini-code-assist bot reviewed May 28, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @upfixer, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

Summary of Changes

Hello team, gemini-code-assist here to provide a summary of this pull request. This PR, authored by upfixer, aims to update the Llama4 chat template and its corresponding pythonic parser. The primary motivation is to align with the latest Llama4 documentation and address an issue where the model sometimes outputs unexpected <|python_start|> and <|python_end|> tokens, which interfere with the parser.

Highlights

  • Llama4 Chat Template Update: The Jinja template for Llama4 tool calling (examples/chat_template/tool_chat_template_llama4_pythonic.jinja) has been significantly revised. This includes updating the source reference, changing the logic for handling system messages (adopting a new default tool system message from the official Llama documentation), removing the tools_in_user_message handling, and adjusting the formatting for tool calls and tool responses to match the expected structure (adding brackets [] around calls, quoting parameter values, and using <|eot|> instead of <|eom|>).
  • Handle Unexpected Python Tokens: A key change in the pythonic parser (python/sglang/srt/function_call/pythonic_detector.py) is the addition of a helper method _text_strip to explicitly remove the <|python_start|> and <|python_end|> tokens. This stripping is applied before checking for tool calls and before parsing the text, ensuring these extraneous tokens do not break the parsing logic.
  • Add Unit Tests for Token Handling: New unit tests have been added in test/srt/test_function_call_parser.py to specifically verify that the parser correctly handles and removes the <|python_start|> and <|python_end|> tokens during both full text parsing (detect_and_parse) and streaming parsing (parse_streaming_increment).

Changelog

Click here to see the changelog
  • examples/chat_template/tool_chat_template_llama4_pythonic.jinja
    • Updated the source URL reference in the comment (line 1).
    • Revised the logic for setting the system message, prioritizing user-provided messages but using a new, more detailed default tool system message if tools are present and no system message is provided (lines 14-43).
    • Removed the tools_in_user_message flag and associated logic (lines 6-8, 39, 56-79).
    • Modified the formatting of tool calls to wrap them in square brackets [] and quote parameter values (lines 78-93).
    • Changed the end-of-message token from <|eom|> to <|eot|> for tool calls and tool responses (lines 93, 107).
  • python/sglang/srt/function_call/pythonic_detector.py
    • Added a static method _text_strip to remove <|python_start|> and <|python_end|> tokens (lines 35-41).
    • Applied _text_strip to the input text in has_tool_call (line 44).
    • Applied _text_strip to the input text in detect_and_parse (line 49).
    • Applied _text_strip to the input text in parse_streaming_increment before adding it to the buffer (line 117).
  • test/srt/test_function_call_parser.py
    • Added test_parse_streaming_with_python_start_and_end_token to test streaming parsing with the new tokens (lines 267-291).
    • Added test_detect_and_parse_with_python_start_and_end_token to test full parsing with the new tokens (lines 293-307).
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in issue comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Template changed, parser refined,
Tokens stripped, no longer bind.
Llama speaks clear,
No Python fear,
Function calls, precisely aligned.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist[bot]
gemini-code-assist bot suggested changes May 28, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request effectively updates the Llama4 chat template to align with new specifications and addresses an issue with unexpected tokens in the Pythonic parser. The changes to the pythonic_detector.py and the corresponding tests in test_function_call_parser.py are well-implemented and directly solve the described problem.

The Jinja template tool_chat_template_llama4_pythonic.jinja has been significantly refactored. While most changes improve clarity and align with the Llama4 documentation, there's a critical aspect regarding the formatting of tool call arguments that needs attention to ensure correctness.

Summary of Findings

  • Tool Call Argument Formatting in Jinja Template: The method for formatting tool call arguments in tool_chat_template_llama4_pythonic.jinja (lines 84-88) does not correctly escape double quotes within string arguments and universally quotes all argument values. This can lead to syntax errors when parsing tool calls with certain string inputs and deviates from Pythonic representation for non-string types. A more robust approach that handles type-specific formatting and escaping is recommended.
  • Pythonic Parser Update: The changes in python/sglang/srt/function_call/pythonic_detector.py to strip <|python_start|> and <|python_end|> tokens are clear and effectively address the reported issue. This was not commented on directly due to review settings (positive change).
  • Test Coverage: The new tests in test/srt/test_function_call_parser.py adequately cover the scenarios involving the unwanted <|python_start|> and <|python_end|> tokens for both streaming and non-streaming parsing. This was not commented on directly due to review settings (positive change).

Merge Readiness

The pull request makes valuable updates. However, due to the identified high-severity issue with tool call argument formatting in the Jinja template, I recommend that these changes be addressed before merging. The Python parser updates and new tests are excellent. I am not authorized to approve pull requests, so please ensure further review and approval from other maintainers after addressing the feedback.

@CatherineSue CatherineSue self-assigned this May 28, 2025
CatherineSue
CatherineSue reviewed May 28, 2025
View reviewed changes
CatherineSue
CatherineSue reviewed May 28, 2025
View reviewed changes
@upfixer
Copy link
Contributor Author

upfixer commented May 29, 2025 •
edited by CatherineSue
Loading

CatherineSue@16a1ed9#diff-428bc0919d595fad48874e3c86e5fb7dde7b91f5891b9b00a274ed2368af985d

🧪 Starting tool_choice functionality tests...

================================================================================
📋 TEST SUITE 1: Single Tool Choice Tests
================================================================================

============================================================
Testing NON-STREAMING: Tool choice AUTO
tool_choice: auto
============================================================
✅ Request successful
Content: To answer the question about the weather in the top 5 most populated cities in the US, I first need to identify what those cities are. Then, I can use the `get_weather` tool to find the current weather for each city.

The top 5 most populated cities in the US are:
1. New York City, NY
2. Los Angeles, CA
3. Chicago, IL
4. Houston, TX
5. Phoenix, AZ

Next, I will use the `get_weather` tool for each of these cities.

[get_weather(city="New York City"), get_weather(city="Los Angeles"), get_weather(city="Chicago"), get_weather(city="Houston"), get_weather(city="Phoenix")]
Tool calls: None
ℹ️  INFO: Auto mode - tool calls: False

============================================================
Testing STREAMING: Tool choice AUTO
tool_choice: auto
============================================================
✅ Request successful
Content: To answer the question about the weather in the top 5 most populated cities in the US, I first need to identify what those cities are. Then, I can proceed to find the current weather for each city.

The top 5 most populated cities in the US, based on general knowledge, are:
1. New York City
2. Los Angeles
3. Chicago
4. Houston
5. Phoenix

Given the available tools, I can use the `get_weather` function to find the weather for each of these cities.

## Step 1: Get Weather for New York City


## Step 2: Get Weather for Los Angeles


## Step 3: Get Weather for Chicago


## Step 4: Get Weather for Houston


## Step 5: Get Weather for Phoenix

Tool call chunks: 5
ℹ️  INFO: Auto mode - tool call chunks: 5

============================================================
Testing NON-STREAMING: Tool choice REQUIRED
tool_choice: required
============================================================
✅ Request successful
Content: None
Tool calls: [ChatCompletionMessageToolCall(id='call_Nq2dtCiVT_KZciSxZ6sBpw', function=Function(arguments='{"decision": "TOOL", "content": "To answer the question about the weather in the top 5 most populated cities in the US, I first need to identify what those cities are. Then, I can use the get_weather tool to find the current weather for each city."}', name='make_next_step_decision'), type='function', index=0), ChatCompletionMessageToolCall(id='call_3-vmUupmTSGK1EI8zSqK7Q', function=Function(arguments='{"city": "New York"}', name='get_weather'), type='function', index=1), ChatCompletionMessageToolCall(id='call_4UoU0CouTvSz9g85rYqdmw', function=Function(arguments='{"city": "Los Angeles"}', name='get_weather'), type='function', index=2), ChatCompletionMessageToolCall(id='call_yANWRZinSCugnVSYxdxMBg', function=Function(arguments='{"city": "Chicago"}', name='get_weather'), type='function', index=3), ChatCompletionMessageToolCall(id='call_7srsQQI-QwGrwoCHzmJzZw', function=Function(arguments='{"city": "Houston"}', name='get_weather'), type='function', index=4), ChatCompletionMessageToolCall(id='call_ZXTjf3PoRMql52pKECmpOQ', function=Function(arguments='{"city": "Phoenix"}', name='get_weather'), type='function', index=5)]
✅ PASS: Tool calls present when required

============================================================
Testing STREAMING: Tool choice REQUIRED
tool_choice: required
============================================================
✅ Request successful
Content:
Tool call chunks: 1
✅ PASS: Tool call chunks present when required

============================================================
Testing NON-STREAMING: Tool choice SPECIFIC: get_weather
tool_choice: {'type': 'function', 'function': {'name': 'get_weather'}}
============================================================
✅ Request successful
Content: None
Tool calls: [ChatCompletionMessageToolCall(id='call_XHw7u_SGReC2BsMt7Lt0fQ', function=Function(arguments='{"city": "New York"}', name='get_weather'), type='function', index=0), ChatCompletionMessageToolCall(id='call_3Ja3eLKkSkCz5Vsu7RtfJQ', function=Function(arguments='{"city": "Los Angeles"}', name='get_weather'), type='function', index=1), ChatCompletionMessageToolCall(id='call_4i0WQoYNTsacZNfWF3LQgA', function=Function(arguments='{"city": "Chicago"}', name='get_weather'), type='function', index=2), ChatCompletionMessageToolCall(id='call_7-h6cF9DTdiMEpCQ8nugVQ', function=Function(arguments='{"city": "Houston"}', name='get_weather'), type='function', index=3), ChatCompletionMessageToolCall(id='call_2fkpy3LzQguCzKc7uKBeUg', function=Function(arguments='{"city": "Phoenix"}', name='get_weather'), type='function', index=4)]
✅ PASS: Correct function called: get_weather

============================================================
Testing STREAMING: Tool choice SPECIFIC: get_weather
tool_choice: {'type': 'function', 'function': {'name': 'get_weather'}}
============================================================
✅ Request successful
Content:
Tool call chunks: 5
✅ PASS: Correct function called: get_weather

============================================================
Testing NON-STREAMING: Tool choice SPECIFIC: get_pokemon_info
tool_choice: {'type': 'function', 'function': {'name': 'get_pokemon_info'}}
============================================================
✅ Request successful
Content: None
Tool calls: [ChatCompletionMessageToolCall(id='call_i4IPSrfYQiOH5grNFl5BWQ', function=Function(arguments='{"name": " Pikachu"}', name='get_pokemon_info'), type='function', index=0)]
✅ PASS: Correct function called: get_pokemon_info

============================================================
Testing STREAMING: Tool choice SPECIFIC: get_pokemon_info
tool_choice: {'type': 'function', 'function': {'name': 'get_pokemon_info'}}
============================================================
✅ Request successful
Content:
Tool call chunks: 1
✅ PASS: Correct function called: get_pokemon_info

============================================================
Testing NON-STREAMING: Tool choice SPECIFIC: make_next_step_decision
tool_choice: {'type': 'function', 'function': {'name': 'make_next_step_decision'}}
============================================================
✅ Request successful
Content: None
Tool calls: [ChatCompletionMessageToolCall(id='call_li8C8gHbRJqHsCAFtPsTdA', function=Function(arguments='{"decision": "TOOL", "content": "To answer the question about the weather in the top 5 most populated cities in the US, I first need to identify what those cities are and then find their current weather conditions. The tool I need for this task is not specified, but typically, I would use a geolocation or weather API. Since I don\'t have a predefined function list to work with, I\'ll assume I need to find this information through internal knowledge or external research."}', name='make_next_step_decision'), type='function', index=0)]
✅ PASS: Correct function called: make_next_step_decision

============================================================
Testing STREAMING: Tool choice SPECIFIC: make_next_step_decision
tool_choice: {'type': 'function', 'function': {'name': 'make_next_step_decision'}}
============================================================
✅ Request successful
Content:
Tool call chunks: 1
✅ PASS: Correct function called: make_next_step_decision

================================================================================
📋 TEST SUITE 2: Multi-Tool Travel Assistant Tests
================================================================================

============================================================
Testing MULTI-TOOL NON-STREAMING: Multi-Tool AUTO (Travel Assistant)
tool_choice: auto
============================================================
✅ Request successful
Content: None
Tool calls: [ChatCompletionMessageToolCall(id='call_UEttNA_sR7yFJp8nW9LSSg', function=Function(arguments='{"location": "Tokyo", "unit": "celsius"}', name='get_weather'), type='function', index=0), ChatCompletionMessageToolCall(id='call_4LRjsr-4QXiyJeL6Zoj0Bw', function=Function(arguments='{"city": "Tokyo"}', name='get_tourist_attractions'), type='function', index=1)]
  - Called: get_weather with args: {"location": "Tokyo", "unit": "celsius"}
  - Called: get_tourist_attractions with args: {"city": "Tokyo"}
ℹ️  INFO: Auto mode - got 2 tool calls
✅ BONUS: Both expected functions called!

============================================================
Testing MULTI-TOOL STREAMING: Multi-Tool AUTO (Travel Assistant)
tool_choice: auto
============================================================
✅ Request successful
Content:
Tool call chunks: 2
  - Found function: get_weather
  - Found function: get_tourist_attractions
ℹ️  INFO: Auto mode - got 2 tool call chunks
✅ BONUS: Both expected functions found in chunks!

============================================================
Testing MULTI-TOOL NON-STREAMING: Multi-Tool REQUIRED (Travel Assistant)
tool_choice: required
============================================================
✅ Request successful
Content: None
Tool calls: [ChatCompletionMessageToolCall(id='call_ZfWia8qDRYKdmN-5WN5djA', function=Function(arguments='{"location": "Tokyo", "unit": "celsius"}', name='get_weather'), type='function', index=0), ChatCompletionMessageToolCall(id='call_g1aEwbRNTj6nKo1Vv3CDtg', function=Function(arguments='{"city": "Tokyo"}', name='get_tourist_attractions'), type='function', index=1)]
  - Called: get_weather with args: {"location": "Tokyo", "unit": "celsius"}
  - Called: get_tourist_attractions with args: {"city": "Tokyo"}
✅ PASS: Tool calls present when required (2 calls)
✅ BONUS: Both expected functions called!

============================================================
Testing MULTI-TOOL STREAMING: Multi-Tool REQUIRED (Travel Assistant)
tool_choice: required
============================================================
✅ Request successful
Content:
Tool call chunks: 2
  - Found function: get_weather
  - Found function: get_tourist_attractions
✅ PASS: Tool call chunks present when required (2 chunks)
✅ BONUS: Both expected functions found in chunks!

================================================================================
📊 COMPREHENSIVE TEST SUMMARY
================================================================================

🗳️  Model: /models/llama4/meta/llama-4-scout-17b-16e-instruct/

🔸 Single Tool Choice Tests:
  Tool choice AUTO
    Non-streaming: ✅ PASS
    Streaming: ✅ PASS
  Tool choice REQUIRED
    Non-streaming: ✅ PASS
    Streaming: ✅ PASS
  Tool choice SPECIFIC: get_weather
    Non-streaming: ✅ PASS
    Streaming: ✅ PASS
  Tool choice SPECIFIC: get_pokemon_info
    Non-streaming: ✅ PASS
    Streaming: ✅ PASS
  Tool choice SPECIFIC: make_next_step_decision
    Non-streaming: ✅ PASS
    Streaming: ✅ PASS

🔸 Multi-Tool Travel Assistant Tests:
  Multi-Tool AUTO (Travel Assistant)
    Non-streaming: ✅ PASS
    Streaming: ✅ PASS
  Multi-Tool REQUIRED (Travel Assistant)
    Non-streaming: ✅ PASS
    Streaming: ✅ PASS

Overall: 14/14 tests passed
🎉 All tests passed!

@CatherineSue
Copy link
Collaborator 

============================================================
Testing NON-STREAMING: Tool choice AUTO
tool_choice: auto
============================================================
✅ Request successful
Content: To answer the question about the weather in the top 5 most populated cities in the US, I first need to identify what those cities are. Then, I can use the `get_weather` tool to find the current weather for each city.

The top 5 most populated cities in the US are:
1. New York City, NY
2. Los Angeles, CA
3. Chicago, IL
4. Houston, TX
5. Phoenix, AZ

Next, I will use the `get_weather` tool for each of these cities.

[get_weather(city="New York City"), get_weather(city="Los Angeles"), get_weather(city="Chicago"), get_weather(city="Houston"), get_weather(city="Phoenix")]
Tool calls: None
ℹ️  INFO: Auto mode - tool calls: False

This is related to the comment I left above regarding match vs search. Please address them and retest this one. Thanks!

@CatherineSue
Resolve comments
8e8c48d 
- Updated `has_tool_call` method to use `search` instead of `match` for regex.
- Introduced `_strip_and_split_buffer` method to manage special tokens and partial tokens.
- Enhanced `parse_streaming_increment` to correctly handle and retain special tokens around tool calls.
- Improved unit tests for parsing with special tokens to ensure accurate detection and parameter extraction.
@CatherineSue
Copy link
Collaborator

Updated result:

============================================================
Testing NON-STREAMING: Tool choice AUTO
tool_choice: auto
============================================================
✅ Request successful
Content: To determine the weather in the top 5 most populated cities in the US, we first need to identify those cities and then retrieve their current weather information.

The top 5 most populated cities in the US are typically New York City, Los Angeles, Chicago, Houston, and Phoenix.

1. **Identify the top 5 cities**: 
   - New York City
   - Los Angeles
   - Chicago
   - Houston
   - Phoenix

2. **Get the weather for each city**:
   - We need to use the `get_weather` function for each of these cities.

Let's start by getting the weather for the first city, New York City.


Tool calls: [ChatCompletionMessageToolCall(id='call_rX_4PJH6RJWfOMmvnw6wDw', function=Function(arguments='{"decision": "TOOL", "content": "Get the weather for New York City using get_weather function"}', name='make_next_step_decision'), type='function', index=None), ChatCompletionMessageToolCall(id='call_xeiv4bY-TfCh5mbZqH2-UA', function=Function(arguments='{"city": "New York City"}', name='get_weather'), type='function', index=None)]
ℹ️  INFO: Auto mode - tool calls: True

Tool calls successfully detected

@CatherineSue
Copy link
Collaborator

Screenshot 2025-05-30 at 1 24 10 PM

@zhyncs zhyncs merged commit 4fac524 into sgl-project:main May 31, 2025
Edenzzzz pushed a commit to Edenzzzz/sglang that referenced this pull request Jun 2, 2025
@upfixer @CatherineSue @Edenzzzz
update llama4 chat template and pythonic parser (sgl-project#6679)
199a7e4 
Co-authored-by: Chang Su <chang.s.su@oracle.com>
Layssy pushed a commit to Layssy/sglang-iaas that referenced this pull request Jun 9, 2025
@upfixer @CatherineSue @Layssy
update llama4 chat template and pythonic parser (sgl-project#6679)
cc8f816 
Co-authored-by: Chang Su <chang.s.su@oracle.com>
xwu-intel pushed a commit to xwu-intel/sglang that referenced this pull request Jun 17, 2025
@upfixer @CatherineSue @xwu-intel
update llama4 chat template and pythonic parser (sgl-project#6679)
57e5f79 
Co-authored-by: Chang Su <chang.s.su@oracle.com>
walker-ai pushed a commit to walker-ai/sglang that referenced this pull request Jul 8, 2025
@yudian0504
PullRequest: 52 sgl_20250610_sync_tag047
a19e2e2 
Merge branch 'sgl_20250610_sync_tag047 of git@code.alipay.com:Theta/SGLang.git into main

https://code.alipay.com/Theta/SGLang/pull_requests/52


Reviewed-by: 剑川 <jianchuan.gys@antgroup.com>


* [Bugfix] Fix slice operation when chunk size mismatch (sgl-project#6697)
* [Bugfix] Fix ChatCompletion endpoint of mini_lb when stream is set (sgl-project#6703)
* [CI] Fix setup of disaggregation with different tp (sgl-project#6706)
* [PD] Remove Unnecessary Exception Handling for FastQueue.get() (sgl-project#6712)
* Fuse routed_scaling_factor in DeepSeek (sgl-project#6710)
* Overlap two kernels in DeepSeek with communication (sgl-project#6711)
* Minor refactor two-batch overlap (sgl-project#6682)
* Speed up when having padding tokens two-batch overlap (sgl-project#6668)
* [Feature] Support Flashinfer fp8 blockwise GEMM kernel on Blackwell (sgl-project#6479)
* Fix LoRA bench (sgl-project#6719)
* temp
* Fix PP for Qwen3 MoE (sgl-project#6709)
* [feat] triton kernel for get_last_loc (sgl-project#6676)
* [fix] more mem for draft_extend cuda_graph (sgl-project#6726)
* [PD] bug fix:  Update status if nixl receiver send a a dummy req. (sgl-project#6720)
* Tune memory arguments on B200 (sgl-project#6718)
* Add DeepSeek-R1-0528 function call chat template (sgl-project#6725)
* refactor(tool call): Fix BaseFormatDetector tool_index issue and refactor `parse_streaming_increment` (sgl-project#6715)
* Add draft extend CUDA graph for Triton backend (sgl-project#6705)
* refactor apply_w8a8_block_fp8_linear in fp (sgl-project#6545)
* [PD] Support completion endpoint (sgl-project#6729)
* PD Rust LB (PO2) (sgl-project#6437)
* Super tiny enable sole usage of expert distribution metrics and update doc (sgl-project#6680)
* Support picking variants of EPLB algorithms (sgl-project#6728)
* Support tuning DeepEP configs (sgl-project#6742)
* [test] add ut and bm for get_last_loc (sgl-project#6746)
* Fix mem_fraction_static for AMD CI (sgl-project#6748)
* [fix][RL] Fix DeepSeekV3ForCausalLM.post_load_weights for multiple update weight (sgl-project#6265)
* Improve EPLB logical to physical dispatch map (sgl-project#6727)
* Update DeepSeek-R1-0528 function call chat template (sgl-project#6765)
* [PD] Optimize time out logic and add env var doc for mooncake (sgl-project#6761)
* Fix aiohttp 'Chunk too big' in bench_serving (sgl-project#6737)
* Support sliding window in triton backend (sgl-project#6509)
* Fix shared experts fusion error (sgl-project#6289)
* Fix one bug in the grouped-gemm triton kernel (sgl-project#6772)
* update llama4 chat template and pythonic parser (sgl-project#6679)
* feat(tool call): Enhance Llama32Detector for improved JSON parsing in non-stream (sgl-project#6784)
* Support token-level quantization for EP MoE (sgl-project#6782)
* Temporarily lower mmlu threshold for triton sliding window backend (sgl-project#6785)
* ci: relax test_function_call_required (sgl-project#6786)
* Add intel_amx backend for Radix Attention for CPU (sgl-project#6408)
* Fix incorrect LoRA weight loading for fused gate_up_proj (sgl-project#6734)
* fix(PD-disaggregation): Can not get local ip (sgl-project#6792)
* [FIX] mmmu bench serving result display error (sgl-project#6525) (sgl-project#6791)
* Bump torch to 2.7.0 (sgl-project#6788)
* chore: bump sgl-kernel v0.1.5 (sgl-project#6794)
* Improve profiler and integrate profiler in bench_one_batch_server (sgl-project#6787)
* chore: upgrade sgl-kernel v0.1.5 (sgl-project#6795)
* [Minor] Always append newline after image token when parsing chat message (sgl-project#6797)
* Update CI tests for Llama4 models (sgl-project#6421)
* [Feat] Enable PDL automatically on Hopper architecture (sgl-project#5981)
* chore: update blackwell docker (sgl-project#6800)
* misc: cache is_hopper_arch (sgl-project#6799)
* Remove contiguous before Flashinfer groupwise fp8 gemm (sgl-project#6804)
* Correctly abort the failed grammar requests & Improve the handling of abort (sgl-project#6803)
* [EP] Add cuda kernel for moe_ep_pre_reorder (sgl-project#6699)
* Add draft extend CUDA graph for flashinfer backend  (sgl-project#6805)
* Refactor CustomOp to avoid confusing bugs (sgl-project#5382)
* Tiny log prefill time (sgl-project#6780)
* Tiny fix EPLB assertion about rebalancing period and recorder window size (sgl-project#6813)
* Add simple utility to dump tensors for debugging (sgl-project#6815)
* Fix profiles do not have consistent names (sgl-project#6811)
* Speed up rebalancing when using non-static dispatch algorithms (sgl-project#6812)
* [1/2] Add Kernel support for Cutlass based Fused FP4 MoE (sgl-project#6093)
* [Router] Fix k8s Service Discovery (sgl-project#6766)
* Add CPU optimized kernels for topk and rope fusions  (sgl-project#6456)
* fix new_page_count_next_decode (sgl-project#6671)
* Fix wrong weight reference in dynamic EPLB (sgl-project#6818)
* Minor add metrics to expert location updater (sgl-project#6816)
* [Refactor] Rename `n_share_experts_fusion` as `num_fused_shared_experts` (sgl-project#6735)
* [FEAT] Add transformers backend support  (sgl-project#5929)
* [fix] recover auto-dispatch for rmsnorm and rope (sgl-project#6745)
* fix ep_moe_reorder kernel bugs (sgl-project#6858)
* [Refactor] Multimodal data processing for VLM (sgl-project#6659)
* Decoder-only Scoring API (sgl-project#6460)
* feat: add dp-rank to KV events (sgl-project#6852)
* Set `num_fused_shared_experts` as `num_shared_experts` when shared_experts fusion is not disabled (sgl-project#6736)
* Fix one missing arg in DeepEP (sgl-project#6878)
* Support LoRA in TestOpenAIVisionServer and fix fused kv_proj loading bug. (sgl-project#6861)
* support 1 shot allreduce  in 1-node and 2-node using mscclpp (sgl-project#6277)
* Fix Qwen3MoE missing token padding optimization (sgl-project#6820)
* Tiny update error hints (sgl-project#6846)
* Support layerwise rebalancing experts (sgl-project#6851)
* Tiny allow profiler API to auto create directory (sgl-project#6865)
* Support Blackwell DeepEP docker images (sgl-project#6868)
* [EP] Add cuda kernel for moe_ep_post_reorder (sgl-project#6837)
* [theta]merge 0605
* oai: fix openAI client error with single request via batch api (sgl-project#6170)
* [PD] Fix potential perf spike caused by tracker gc and optimize doc (sgl-project#6764)
* Use deepgemm instead of triton for fused_qkv_a_proj_with_mqa (sgl-project#6890)
* [CUTLASS-FP4-MOE]  Introduce CutlassMoEParams class for easy initialization of Cutlass Grouped Gems Metadata (sgl-project#6887)
* bugfix(OAI): Fix image_data processing for jinja chat templates (sgl-project#6877)
* [CPU] enable CI for PRs, add Dockerfile and auto build task (sgl-project#6458)
* AITER backend extension and workload optimizations (sgl-project#6838)
* [theta]merge
* [theta]merge
* [Feature] Support Flashinfer fmha on Blackwell (sgl-project#6930)
* Fix a bug in abort & Improve docstrings for abort (sgl-project#6931)
* Tiny support customize DeepEP max dispatch tokens per rank (sgl-project#6934)
* Sync the changes on cuda graph runners (sgl-project#6932)
* [PD] Optimize transfer queue forward logic for dummy rank (sgl-project#6922)
* [Refactor] image data process in bench_serving (sgl-project#6879)
* [fix] logical_to_all_physical_map index 256 is out of bounds in EP parallel. (sgl-project#6767)
* Add triton fused moe kernel config for E=257 on B200 (sgl-project#6939)
* [sgl-kernel] update deepgemm (sgl-project#6942)
* chore: bump sgl-kernel v0.1.6 (sgl-project#6943)
* Minor compile fused topk (sgl-project#6944)
* [Bugfix] pipeline parallelism and Eagle Qwen2 (sgl-project#6910)
* Tiny re-introduce profile id logging (sgl-project#6912)
* Add triton version as a fused_moe_triton config search key to avoid performace decrease in different Triton version (sgl-project#5955)
* reduce torch.zeros overhead in moe align block size kernel (sgl-project#6369)
* chore: upgrade sgl-kernel v0.1.6 (sgl-project#6945)
* add fbgemm moe grouped gemm kernel benchmark (sgl-project#6924)
* [Docker] Add docker file for SGL Router (sgl-project#6915)
* Disabling mixed chunked prefill when eagle is enabled (sgl-project#6874)
* Add canary for EPLB rebalancing (sgl-project#6895)
* Refactor global_server_args_dict (sgl-project#6866)
* Fuse routed scaling factor in topk_reduce kernel (sgl-project#6220)
* Update server timeout time in AMD CI. (sgl-project#6953)
* [misc] add is_cpu() (sgl-project#6950)
* Add H20 fused MoE kernel tuning configs for DeepSeek-R1/V3 (sgl-project#6885)
* Add a CUDA kernel for fusing mapping and weighted sum for MoE. (sgl-project#6916)
* chore: bump sgl-kernel v0.1.6.post1 (sgl-project#6955)
* chore: upgrade sgl-kernel v0.1.6.post1 (sgl-project#6957)
* [DeepseekR1-FP4] Add Support for nvidia/DeepSeekR1-FP4 model (sgl-project#6853)
* Revert "Fuse routed scaling factor in topk_reduce kernel (sgl-project#6220)" (sgl-project#6968)
* [AMD] Add more tests to per-commit-amd (sgl-project#6926)
* chore: bump sgl-kernel v0.1.7 (sgl-project#6963)
* Slightly improve the sampler to skip unnecessary steps (sgl-project#6956)
* rebase h20 fused_moe config (sgl-project#6966)
* Fix CI and triton moe Configs (sgl-project#6974)
* Remove unnecessary kernels of num_token_non_padded (sgl-project#6965)
* Extend cuda graph capture bs for B200 (sgl-project#6937)
* Fuse routed scaling factor in deepseek (sgl-project#6970)
* Sync cuda graph runners (sgl-project#6976)
* Fix draft extend ut stability with flush cache (sgl-project#6979)
* Fix triton sliding window test case (sgl-project#6981)
* Fix expert distribution dumping causes OOM (sgl-project#6967)
* Minor remove one kernel for DeepSeek (sgl-project#6977)
* [perf][sgl-kernel] extend cutlass_mla_decode to support num_head < 128 (sgl-project#6929)
* Enable more unit tests for AMD CI. (sgl-project#6983)
* Use torch.compile to fuse flash attention decode metadata preparation (sgl-project#6973)
* Eliminate stream sync to speed up LoRA batch init  (sgl-project#6960)
* support qwen3 emebedding (sgl-project#6990)
* Fix torch profiler bugs for bench_offline_throughput.py (sgl-project#6557)
* chore: upgrade flashinfer v0.2.6.post1 jit (sgl-project#6958)
* cleanup tmp dir (sgl-project#7007)
* chore: update pr test xeon (sgl-project#7008)
* Fix cutlass MLA gets almost zero accuracy (sgl-project#6998)
* Update amd nightly models CI. (sgl-project#6992)
* feat: add direct routing strategy to DP worker (sgl-project#6884)
* Fallback to lower triton version for unfound fused moe configs (sgl-project#7013)
* Fix torchvision version for Blackwell (sgl-project#7015)
* Simplify prepare_extend_after_decode (sgl-project#6987)
* Migrate to assertEqual (sgl-project#6741)
* Fix torch version in blackwell dockerfile (sgl-project#7017)
* chore: update pr test xeon (sgl-project#7018)
* Update default settings for blackwell (sgl-project#7023)
* Support both approximate and exact expert distribution collection (sgl-project#6964)
* Add decode req pool (sgl-project#6980)
* [theta]merge 0610
* [theta]merge 0610
* [CI] Add CI workflow for sgl-router docker build (sgl-project#7027)
* Fix fused_moe triton configs (sgl-project#7029)
* CPU: map changes from developing branch in sgl-kernel (sgl-project#6833)
* chore: bump v0.4.7 (sgl-project#7038)
* Update README.md (sgl-project#7040)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CatherineSue CatherineSue CatherineSue left review comments

@zhyncs zhyncs zhyncs approved these changes

@merrymercy merrymercy Awaiting requested review from merrymercy

@Ying1123 Ying1123 Awaiting requested review from Ying1123

@hnyls2002 hnyls2002 Awaiting requested review from hnyls2002

@ispobock ispobock Awaiting requested review from ispobock

@ByronHsu ByronHsu Awaiting requested review from ByronHsu

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] requested changes

Reviewers whose approvals may not affect merge requirements
Assignees

@CatherineSue CatherineSue

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@upfixer @CatherineSue @zhyncs