Skip to content

[Bug] incorret tool_calls index when there are multi tool_calls and in stream mode #6310

New issue
New issue
Closed
#6715
Closed
[Bug] incorret tool_calls index when there are multi tool_calls and in stream mode#6310
#6715
@CyanCherry

Description

@CyanCherry
CyanCherry
opened on May 15, 2025

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
  • 5. Please use English, otherwise it will be closed.

Describe the bug

for request with multi tool_calls, and at stream mode (response in event-stream mode), tool_calls indexes are different among the all tool_calls chunk
seems first tool_calls index is the index of the tools in the tools of request,
but following tool_calls index is the index of the tools called by LLM in this response

the tool_calls fragment is as (full response is in the reproduce block):

data: {"id":"27b2495d4c3a4511a5c8403f5a43db78","object":"chat.completion.chunk","created":1747280110,"model":"QWen","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":null,"tool_calls":[{"id":"call_kiRj2rWPTJqX3yLwcZmHew","index":0,"type":"function","function":{"name":"list_tasks","arguments":""}}]},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"27b2495d4c3a4511a5c8403f5a43db78","object":"chat.completion.chunk","created":1747280110,"model":"QWen","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":null,"tool_calls":[{"id":null,"index":0,"type":"function","function":{"name":null,"arguments":"{\"skip\": 0"}}]},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"27b2495d4c3a4511a5c8403f5a43db78","object":"chat.completion.chunk","created":1747280110,"model":"QWen","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":null,"tool_calls":[{"id":null,"index":0,"type":"function","function":{"name":null,"arguments":", \"limit\": 2"}}]},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"27b2495d4c3a4511a5c8403f5a43db78","object":"chat.completion.chunk","created":1747280110,"model":"QWen","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":null,"tool_calls":[{"id":null,"index":0,"type":"function","function":{"name":null,"arguments":"0"}}]},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"27b2495d4c3a4511a5c8403f5a43db78","object":"chat.completion.chunk","created":1747280110,"model":"QWen","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":null,"tool_calls":[{"id":null,"index":0,"type":"function","function":{"name":null,"arguments":", \"active_only\": true}"}}]},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

Reproduction

start command:

python3 -m sglang.launch_server --model-path ./Qwen3-32B --tensor-parallel-size 8 --reasoning-parser qwen3 --tool-call-parser qwen25 --host 0.0.0.0

full request:

{
    "model": "Qwen/Qwen3",
    "messages": [
        {
            "role": "user",
            "content": "list tasks in XML format"
        }
    ],
    "stream": true,
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "list_tasks_json",
                "description": "List tasks in JSON fromat",
                "parameters": {
                    "properties": {
                        "page": {
                            "default": 0,
                            "type": "interger"
                        },
                        "page_size": {
                            "default": 10,
                            "type": "integer"
                        }
                    },
                    "type": "object"
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "list_tasks_md",
                "description": "List tasks in markdown format",
                "parameters": {
                    "properties": {
                        "page": {
                            "default": 0,
                            "type": "interger"
                        },
                        "page_size": {
                            "default": 10,
                            "type": "integer"
                        }
                    },
                    "type": "object"
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "list_tasks_xml",
                "description": "List tasks in XML format",
                "parameters": {
                    "properties": {
                        "page": {
                            "default": 0,
                            "type": "interger"
                        },
                        "page_size": {
                            "default": 10,
                            "type": "integer"
                        }
                    },
                    "type": "object"
                }
            }
        }
    ]
}

full response:

HTTP/1.1 200 OK
Content-Type: text/event-stream; charset=utf-8
Date: Thu, 15 May 2025 03:51:40 GMT
Server: uvicorn
Transfer-Encoding: chunked

Response code: 200 (OK); Time: 9ms (9 ms)

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":"assistant","content":null,"reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":"\n","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":"Okay","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":",","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" the","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" user","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" wants","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" to","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" list","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" tasks","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" in","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" XML","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" format","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":".","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" Let","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" me","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" check","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" the","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" available","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" tools","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":".","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" There","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":"'s","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" a","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" function","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" called","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" list","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":"_tasks","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":"_xml","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":".","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" The","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" parameters","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" are","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" page","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" and","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" page","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":"_size","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":",","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" both","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" integers","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" with","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" defaults","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" ","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":"0","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" and","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" ","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":"1","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":"0","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":".","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" Since","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" the","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" user","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" didn","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":"'t","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" specify","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" page","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" or","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" page","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":"_size","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":",","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" I","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" should","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" use","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" the","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" defaults","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":".","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" So","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" I","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" need","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" to","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" call","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" list","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":"_tasks","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":"_xml","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" with","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" page","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" ","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":"0","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" and","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" page","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":"_size","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" ","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":"1","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":"0","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":".","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" Let","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" me","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" make","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" sure","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" the","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" JSON","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" is","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" correctly","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" formatted","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" with","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" those","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":" values","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":".\n","tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":"\n\n","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":null,"tool_calls":[{"id":"call_m3PPycQ5TUSUZO7AAaBrQw","index":2,"type":"function","function":{"name":"list_tasks_xml","arguments":""}}]},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":null,"tool_calls":[{"id":null,"index":0,"type":"function","function":{"name":null,"arguments":"{\"page\": 0"}}]},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":null,"tool_calls":[{"id":null,"index":0,"type":"function","function":{"name":null,"arguments":", \"page_size\": 1"}}]},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":null,"tool_calls":[{"id":null,"index":0,"type":"function","function":{"name":null,"arguments":"0}"}}]},"logprobs":null,"finish_reason":null,"matched_stop":null}],"usage":null}

data: {"id":"fe1d2314cd94482a8820a506a5361d72","object":"chat.completion.chunk","created":1747281101,"model":"Qwen/Qwen3","choices":[{"index":0,"delta":{"role":null,"content":null,"reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"tool_calls","matched_stop":null}],"usage":null}

data: [DONE]

dockerfile:

FROM nvcr.io/nvidia/tritonserver:25.03-py3

RUN pip install -U "sglang[all]" -i https://nexus.deeproute.ai/repository/pypi-group/simple

Environment

lsb_release -a:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 24.04.1 LTS
Release:        24.04
Codename:       noble

nvidia-smi:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.135                Driver Version: 550.135        CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4080 ...    On  |   00000000:01:00.0 Off |                  N/A |
| 30%   30C    P8             16W /  320W |   15243MiB /  16376MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 4080 ...    On  |   00000000:21:00.0 Off |                  N/A |
| 30%   29C    P8             18W /  320W |   15243MiB /  16376MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA GeForce RTX 4080 ...    On  |   00000000:41:00.0 Off |                  N/A |
| 30%   28C    P8              4W /  320W |   15245MiB /  16376MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA GeForce RTX 4080 ...    On  |   00000000:61:00.0 Off |                  N/A |
| 30%   28C    P8             17W /  320W |   15243MiB /  16376MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA GeForce RTX 4080 ...    On  |   00000000:81:00.0 Off |                  N/A |
| 30%   28C    P8             22W /  320W |   15243MiB /  16376MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA GeForce RTX 4080 ...    On  |   00000000:A1:00.0 Off |                  N/A |
| 30%   29C    P8             12W /  320W |   15243MiB /  16376MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA GeForce RTX 4080 ...    On  |   00000000:C1:00.0 Off |                  N/A |
| 30%   29C    P8             19W /  320W |   15245MiB /  16376MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA GeForce RTX 4080 ...    On  |   00000000:E1:00.0 Off |                  N/A |
| 30%   28C    P8             21W /  320W |   15243MiB /  16376MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions