chore(openai): refactor streamed response handling to the tracing plugin #6107
Conversation
Overall package sizeSelf size: 9.68 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | @datadog/libdatadog | 0.7.0 | 35.02 MB | 35.02 MB | | @datadog/native-appsec | 10.0.0 | 20.3 MB | 20.31 MB | | @datadog/native-iast-taint-tracking | 4.0.0 | 11.72 MB | 11.73 MB | | @datadog/pprof | 5.9.0 | 9.77 MB | 10.14 MB | | @opentelemetry/core | 1.30.1 | 908.66 kB | 7.16 MB | | protobufjs | 7.5.3 | 2.95 MB | 5.6 MB | | @datadog/wasm-js-rewriter | 4.0.1 | 2.85 MB | 3.58 MB | | @datadog/native-metrics | 3.1.1 | 1.02 MB | 1.43 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | jsonpath-plus | 10.3.0 | 617.18 kB | 1.08 MB | | import-in-the-middle | 1.14.2 | 122.36 kB | 850.93 kB | | lru-cache | 10.4.3 | 804.3 kB | 804.3 kB | | source-map | 0.7.4 | 226 kB | 226 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.1 | 109.9 kB | 109.9 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | ignore | 7.0.5 | 63.38 kB | 63.38 kB | | istanbul-lib-coverage | 3.2.2 | 34.37 kB | 34.37 kB | | rfdc | 1.4.1 | 27.15 kB | 27.15 kB | | @isaacs/ttlcache | 1.4.1 | 25.2 kB | 25.2 kB | | dc-polyfill | 0.1.9 | 25.11 kB | 25.11 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | shell-quote | 1.8.3 | 23.74 kB | 23.74 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | semifies | 1.0.0 | 15.84 kB | 15.84 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | ttl-set | 1.0.0 | 4.61 kB | 9.69 kB | | mutexify | 1.4.0 | 5.71 kB | 8.74 kB | | path-to-regexp | 0.1.12 | 6.6 kB | 6.6 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | module-details-from-path | 1.0.4 | 3.96 kB | 3.96 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #6107 +/- ##
==========================================
+ Coverage 81.98% 82.79% +0.80%
==========================================
Files 475 475
Lines 19598 19600 +2
==========================================
+ Hits 16067 16227 +160
+ Misses 3531 3373 -158 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
BenchmarksBenchmark execution time: 2025-07-16 12:53:01 Comparing candidate commit 93021ce in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 1271 metrics, 52 unstable metrics. |
| * @param {number} n the number of choices to expect in the response | ||
| * @returns {Record<string, any>} | ||
| */ | ||
| function constructChatCompletionResponseFromStreamedChunks (chunks, n) { |
There was a problem hiding this comment.
We could consolidate these two methods and use a smaller part for the unique handling and just pass through that smaller method. That prevents code duplication :)
There was a problem hiding this comment.
i thought about that but just wanted to be explicit in how we're handling each case - if that's all good with you! i thought trying to extract it to pass something like a onIndexedChunk function to the shared handler function would hinder readability/initial understanding a bit, but if you feel strongly i can implement it!
…gin (#6107) * refactor to tracing plugin
…gin (#6107) * refactor to tracing plugin
…gin (#6107) * refactor to tracing plugin
What does this PR do?
Moves streamed response coalescing to the tracing plugin instead of the instrumentation. This allows us:
Motivation
Easier maintenance, developer experience, decoupling.
There's still a lot of cleanup needed for this integration that has unfortunately just been lower priority tech debt, but this is hopefully a good start. I figure a good amount of refactoring will come when we migrate this integration to use orchestrion instead.
MLOB-3331