Skip to content

Conversation

FlorianVal
Copy link
Contributor

From the Issue : #1343

Seems like that the api_base and api_keys args were forgot when setting stream_outputs=True in a CodeAgent.

I don't see any reason to not put them in case of a streamed output, so I added them.

Calling a ollama server with custom api_base and stream_outputs=True is now working. 👋

@FlorianVal
Copy link
Contributor Author

More information about that : Ollama support is still not fully implemented by Litellm. see this Issue.
When providing api_base key to completion() it is used to generate text with the right ollama server but some function might not get the api_base argument and fallback to localhost. So we can have that sort of log when setting the logger :

INFO:httpx:HTTP Request: POST http://XXX.XXX.XXX.XXX:11434/api/generate "HTTP/1.1 200 OK" (Right call to remote server)
00:26:08 - LiteLLM:INFO: cost_calculator.py:656 - selected model name for cost calculation: ollama/qwen3:8b
INFO:LiteLLM:selected model name for cost calculation: ollama/qwen3:8b
INFO:httpx:HTTP Request: POST http://localhost:11434/api/show "HTTP/1.1 200 OK"

As we can see there is still an http call to localhost, so the cost calculation (which is 0) is provided by the local ollama. I'm not sure if it affects other functionality.

Anyway the only way that it affects us is : if you have an ollama server locally and a remote ollama server, you might get an http request to your local ollama server. if we want to fix this on our side we could add

if api_base:
  os.environ["OLLAMA_API_BASE"] = api_base

In the LiteLLMModel init But might be better to just wait for a fix inside LiteLLM.

TL;DR: LiteLLM does not fully support Ollama, but the PR is still valid. 👋

Copy link
Member

@albertvillanova albertvillanova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good catch!

@albertvillanova albertvillanova merged commit d59f04f into huggingface:main May 19, 2025
3 checks passed
@FlorianVal FlorianVal deleted the fix/LiteLLM_stream_outputs_fix branch May 20, 2025 07:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] setting stream_outputs=True in a CodeAgent made the model ignore api_base
2 participants