Skip to content

Conversation

CatherineSue
Copy link
Collaborator

@CatherineSue CatherineSue commented Mar 11, 2025

Motivation

Llama3.3 frequently fails to generate the <|python_tag|> token at higher temperatures, so additional checks are needed to properly detect tool calls.

Modifications

This commit refactors the tool call detection logic to allow each tool call parser to handle its own detection logic:

  1. Added a has_tool_call() method to BaseFormatDetector class
  2. Implemented has_tool_call() in Llama32Detector to check for both <|python_tag|> and JSON objects starting with "{"
  3. Implemented has_tool_call() in other detector classes (Qwen25Detector, MistralDetector)
  4. Updated FunctionCallParser to expose has_tool_call() and check all detectors
  5. Modified adapter.py to use the parser's has_tool_call() method instead of hardcoding detection logic

Checklist

Llama3.3 frequently fails to generate the <|python_tag|> token at higher temperatures,
so additional checks are needed to properly detect tool calls. This commit refactors the
tool call detection logic to allow each tool call parser to handle its own detection logic:

1. Added a has_tool_call() method to BaseFormatDetector class
2. Implemented has_tool_call() in Llama32Detector to check for both <|python_tag|>
   and JSON objects starting with "{"
3. Implemented has_tool_call() in other detector classes (Qwen25Detector, MistralDetector)
4. Updated FunctionCallParser to expose has_tool_call() and check all detectors
5. Modified adapter.py to use the parser's has_tool_call() method instead of hardcoding
   detection logic
@CatherineSue
Copy link
Collaborator Author

manual test:
Screenshot 2025-03-11 at 4 18 40 PM

@zhyncs zhyncs self-assigned this Mar 11, 2025
@zhyncs
Copy link
Member

zhyncs commented Mar 11, 2025

@shuaills @hnyls2002 @DarkSharpness

@zhyncs zhyncs requested a review from shuaills March 11, 2025 23:37
@CatherineSue CatherineSue force-pushed the fix-llama3-tool-call branch from ec44081 to 888a0fc Compare March 12, 2025 00:01
@zhyncs
Copy link
Member

zhyncs commented Mar 12, 2025

@Ubospica

@DarkSharpness
Copy link
Collaborator

LGTM!

@zhyncs zhyncs merged commit 5fe7960 into sgl-project:main Mar 13, 2025
2 of 19 checks passed
@CatherineSue CatherineSue deleted the fix-llama3-tool-call branch March 19, 2025 00:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants