Description

This adds a provider for llama.cpp's llama-server, leveraging its recently introduced grammar-constrained, universal tool call support.

Structural changes:

• Refactored system.ts:
  • Optionally return tool definitions to pass to ApiHandler.createMessage (if ModelInfo.supportsTools)
  • Add provider-specific formatting of tool call examples (implemented w/ llama.cpp api calls - cached)

Limitations:

• No streaming yet, so you may wanna launch llama-server with the --verbose flag to see what's happening (especially when the KV cache isn't hot yet on first call)
  • llama-server doesn't support tool calls in streaming mode yet
  • I'm translating the tool_calls output back to the XML-ish format Cline expects, so this would also need a revamp to support streaming
• Calling llama-server's /apply-template endpoint w/ + fresh new add_generation_prompt param to compute tool call deltas. This is used in the system prompt to express the few-shot tool use examples with the call syntax native to the model, which helps it tremendously (esp. when small).
• Need an async call to know the model's max tokens, which makes the code slightly awkward (getModel may return default values before props are fetched; may want to make getModel async as follow up?)
• Qwen 2.5 Coder support needs tool-call: fix Qwen 2.5 Coder support, add micro benchmarks, support trigger patterns for lazy grammars ggml-org/llama.cpp#12034 (under review / not merged yet)

TODOs / follow ups:

• Merge server: support add_generation_prompt query param ggml-org/llama.cpp#12062
• Fix Context Window gauge
• Nicer error messages when llama-server is down (currently cryptic fetch failed)
• Update help text to heavily suggest disabling MCP to save context
• Merge tool-call: fix Qwen 2.5 Coder support, add micro benchmarks, support trigger patterns for lazy grammars ggml-org/llama.cpp#12034 (non-blocker)

Note that switching to native tool use for other providers should be possible (even for Claude, assuming it doesn't mess with its caching), but one would need to find a nice syntax to format tool call examples.

Test Procedure

I haven't tested this thoroughly yet, but did check that the system prompt didn't budge (only diff is in Example 6 where JSON arguments are now fully indented - incl. labels & assignees)

Mostly had to disabled MCP to save prompt, tested successfully w/ long context window models (128k tokens):

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
git remote add ochafik https://github.com/ochafik/llama.cpp
git fetch ochafik
git checkout ochafik/tool-bench-prod
cmake -B build -DLLAMA_CURL=1
cmake --build build -t llama-server --parallel --config Release
alias llama-server=./build/bin/llama-server

llama-server --jinja -fa -hf unsloth/phi-4-GGUF:Q6_K -c 131072 -ngl 999
llama-server --jinja -fa -hf bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q6_K_L -c 0 -ngl 999
llama-server --jinja -fa -hf unsloth/Qwen2.5-Coder-32B-Instruct-128K-GGUF -c 0 -ngl 999

Some notes:

• Qwen 2.5 Coder 32B 128k works best but requires this PR: tool-call: fix Qwen 2.5 Coder support, add micro benchmarks, support trigger patterns for lazy grammars ggml-org/llama.cpp#12034
• unsloth/phi-4-GGUF supports 128k context window but it's not set in its properties (the other models support -c 0 to mean "use the model's maximum")

Random prompts tested:

• write a web app that creates qr codes
• write a lisp parser in c++
• write a trie in c++
• build a todo list web app that persists in local storage

Type of Change

• 🐛 Bug fix (non-breaking change which fixes an issue)
• ✨ New feature (non-breaking change which adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
• 📚 Documentation update

Pre-flight Checklist

• Changes are limited to a single feature, bugfix or chore (split larger changes into separate PRs)
• happy to split this
• Tests are passing (npm test) and code is formatted and linted (npm run format && npm run lint)
• I have created a changeset using npm run changeset (required for user-facing changes)
• I have reviewed contributor guidelines

Screenshots

Additional Notes

Still some todos but could be done as follow ups.