Skip to content

Conversation

shuaills
Copy link
Collaborator

@shuaills shuaills commented Jan 1, 2025

Motivation

related to #2696

@zhaochenyang20
Copy link
Collaborator

zhaochenyang20 commented Jan 1, 2025

SGLang supports two grammar backends:

  • Outlines (default): Supports JSON schema and regular expression constraints.
  • XGrammar: Supports JSON schema and EBNF constraints and currently uses the GGML BNF format.

We suggest using XGrammar whenever possible for its better performance. For more details, see XGrammar technical overview.

To use Xgrammar, simply add --grammar-backend xgrammar when launching the server. If no backend is specified, Outlines will be used as the default.

@zhaochenyang20
Copy link
Collaborator

try:
response_content = response.choices[0].message.content
# validate the JSON response by the pydantic model
capital_info = CapitalInfo.model_validate_json(response_content)
print_highlight(f"Validated response: {capital_info.model_dump_json()}")
except ValueError as e:
print(f"Validation error: {e}")

Do not use try except, since we do not want the except to happen and pass CI.

@zhaochenyang20
Copy link
Collaborator

Define Schemas: Directly define a JSON schema or use Pydantic to define a schema.
Validate Responses: Use Pydantic to ensure the response matches expected structure and types.

Give example for both Json directly and Pydantic

@zhaochenyang20 zhaochenyang20 merged commit 062c48d into sgl-project:main Jan 1, 2025
2 checks passed
@shuaills shuaills deleted the Structured_Output_Format branch January 2, 2025 22:24
XiaotongJiang pushed a commit to XiaotongJiang/sglang that referenced this pull request Jan 3, 2025
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants