Implement Agent and Model tracing using Langfuse #45
Conversation
* Integrate Langfuse SDK to trace - Agent runs - Model calls - Code executions * Create `tracing` package containing public API and implemenation * Adapt quickstart to use tracing * Add tracing to CLI
* Use `tracing.configure()` instead of a tracing context manager * Add observability documentation * TODO: API docs and screenshots for observability documentation is missing
- async tracer API - async langfuse impl - tracing data created in stream() methods of - CodeActAgentTurn - LiteLLMTurn - renamings
|
@cstub I experimented with some changes here for further discussion. See commit message for a high-level overview of changes. Regarding the async tracer API, I'm aware, that the langfuse Python SDK doesn't block for writing tracing data i.e.. uses a queue which shouldn't be an issue for async client code, but they still use a sync logger API which may cause troubles with some logging backends. I didn't look further into the langfuse async client. Seems to be even more lower-level, but this might be an alternative for a later change. I'll take a look at the user docs in a next step. |
There was a problem hiding this comment.
Pull Request Overview
This PR integrates the Langfuse SDK to enable tracing for agent turns, model calls, and code executions, while introducing a new tracing package and updating documentation and CLI configuration accordingly.
- Added test cases for tracing context, agent turns, and model interactions.
- Implemented new tracing classes (Langfuse, Noop, context) and updated the agent, model, CLI, and docs to support observability.
Reviewed Changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/tracing/* | Added tests to validate tracing configuration and behavior. |
| freeact/tracing/* | New tracing package with Langfuse and Noop implementations. |
| freeact/model/litellm.py | Modified to create spans and update trace data during streaming. |
| freeact/agent.py | Updated to include trace creation, span management for code execution and agent run. |
| pyproject.toml, mkdocs.yml, docs/* | Updated configuration and documentation for observability integration. |
…lity documentation
There was a problem hiding this comment.
No comments on the current changes in the PR, looks very good! However, I found an issue while experimenting with the observability.py example.
To reproduce:
Set model_name=gpt-4.1 and leave all other settings as-is. This will cause an error in litellm because gpt-4.1 doesn't support a reasoning effort which is expected. However, what I see first in the stack trace is
Task exception was never retrieved
future: <Task finished name='Task-170' coro=<<async_generator_athrow without __name__>()> exception=ValueError("<Token var=<ContextVar name='tracing_active_trace' default=None at 0x73b74d6fc310> at 0x73b74c2c7340> was created in a different Context")>
Traceback (most recent call last):
File "/home/martin/Development/gradion/freeact/freeact/tracing/context.py", line 149, in trace
yield active_trace
File "/home/martin/Development/gradion/freeact/freeact/agent.py", line 94, in stream
yield elem
GeneratorExit
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/martin/Development/gradion/freeact/freeact/agent.py", line 82, in stream
async with tracing.trace(
File "/home/martin/Applications/miniconda3/envs/gradion-freeact/lib/python3.11/contextlib.py", line 231, in __aexit__
await self.gen.athrow(typ, value, traceback)
File "/home/martin/Development/gradion/freeact/freeact/tracing/context.py", line 151, in trace
_active_trace_context.reset(token)
ValueError: <Token var=<ContextVar name='tracing_active_trace' default=None at 0x73b74d6fc310> at 0x73b74c2c7340> was created in a different Context
This is unexpected. Can you please investigate that?
|
I could fix the problem mentioned in this comment. |
tracingpackage containing public API and implementation