Add streaming chat completions #31
Conversation
There was a problem hiding this comment.
✅ This pull request was sent to the PullRequest network.
@bakks you can click here to see the review status or cancel the code review job.
There was a problem hiding this comment.
Changes look good to add the ChatCompletion calls. One minor question noted, but it's more for future diagnostics rather than anything substantial.
Reviewed with ❤️ by PullRequest
| // make sure there isn't any extra whitespace before or after | ||
| line = bytes.TrimSpace(line) | ||
| // the completion API only returns data events | ||
| if !bytes.HasPrefix(line, dataPrefix) { |
There was a problem hiding this comment.
I believe this line is here partly to filter out empty lines in the streaming response, of which there are many, so I don't think it makes sense to add extra logging here.
Some background context - the OpenAI documentation on the streaming flag is:
Whether to stream back partial progress. If set, tokens will be sent as data-only server-sent events as they become available, with the stream terminated by a
data: [DONE]message.
What I saw when debugging the responses is that every bit of data indeed had the data: prefix, except for the completely empty lines. So this line is exercised during normal operation, and the JSON unmarshalling will take care of throwing an error if there's line corruption.
|
Please merge when ready, i don't have write access |
|
I've rebased and pushed after the merge of #30, so this branch should only show my own changes now. Merge when ready. |
Hey all - I'm just piggybacking on #30 and adding streaming support. Recommend that you merge @abatilo 's code first so that my additions are more clear. And please feel free to edit this PR yourself.