Working streaming Gradio chatbot outputs #1246
Conversation
| yield gr.ChatMessage(role="assistant", content="**Planning step**", metadata={"status": "done"}) | ||
| yield gr.ChatMessage(role="assistant", content=step_log.plan, metadata={"status": "done"}) | ||
| yield gr.ChatMessage( | ||
| role="assistant", content=get_step_footnote_content(step_log, "Planning step"), metadata={"status": "done"} |
There was a problem hiding this comment.
Using metadata field to note when a step is done, will help us in the gradio app when receiving a completion delta to differentiate between "previous message is complete: start a new one" or "append this completion delta's text to the previous pending message".
Maybe there's a simpler way thought, WDYT?
There was a problem hiding this comment.
Not clear to me how to use this param. However, it seems it requires a "title" key.
There was a problem hiding this comment.
Filling the "title" key makes a title show up, which wouldn't look very good in our case.
This title field does not seem mandatory in practice, since the previous implementaiton of gr.ChatMessage in smolagents comes from @yvrjsharma and does not have title fields in every message.
There was a problem hiding this comment.
So it appears to be a misalignment in the doc of this param.
| <a target="_blank" href="https://github.com/huggingface/smolagents"><b>huggingface/smolagents</b></a> | ||
| </div>""") | ||
| gr.HTML( | ||
| "<br><br><h4><center>Powered by <a target='_blank' href='https://github.com/huggingface/smolagents'><b>smolagents</b></a></center></h4>" |
There was a problem hiding this comment.
Making the branding a bit less present, because for now it's too big. Maybe could be removed altogether.
| yield gr.ChatMessage(role="assistant", content="**Planning step**", metadata={"status": "done"}) | ||
| yield gr.ChatMessage(role="assistant", content=step_log.plan, metadata={"status": "done"}) | ||
| yield gr.ChatMessage( | ||
| role="assistant", content=get_step_footnote_content(step_log, "Planning step"), metadata={"status": "done"} |
There was a problem hiding this comment.
Not clear to me how to use this param. However, it seems it requires a "title" key.
src/smolagents/agents.py
Outdated
| pass | ||
| def step(self, memory_step: ActionStep) -> Generator[Any]: | ||
| """To be implemented in children classes. Should yield either None if the step is not final, or the final answer.""" | ||
| yield None |
There was a problem hiding this comment.
I think the conversion of step to generator is a breaking change.
|
@yvrjsharma could you take a look at this comment? The way've I've modified the method |
src/smolagents/agents.py
Outdated
| @@ -1007,10 +1035,10 @@ def initialize_system_prompt(self) -> str: | |||
| ) | |||
| return system_prompt | |||
|
|
|||
| def step(self, memory_step: ActionStep) -> None | Any: | |||
| def _step(self, memory_step: ActionStep) -> Generator[Any]: | |||
There was a problem hiding this comment.
@albertvillanova here's how I modified the step method.
|
@albertvillanova I went for |
This finalizes the streaming refactoring: makes step functions generators, and adds some intermediate
yieldto yield intermediate completions.Also I've set all examples to stream outputs by default when the underlying Model class allows it. Progressively we should switch wherever we can to
stream_ouputsbecause it's much more user-friendly than waiting until the end of message generation.