feat(ai): add vercel ai integration #5858

sabrenner · 2025-06-09T17:52:57Z

What does this PR do?

Adds APM and LLM Observability support for ai@4.0.0 and greater.

DISCLAIMER: Most LOC are from "cassettes" added locally used to mock and play back locally-recorded responses from provider APIs. These are stripped of any sensitive information/headers in the ddapm-test-agent image.

The Vercel AI SDK provides OTel tracing of their operations under the hood. This gives us a nice "in" to patch the tracer used to intercept the startActiveSpan function, and various operations on the underlying span, and translate them into APM and LLM Observability spans.

This integration works by doing exactly that - patching the tracer passed in, and if none is passed in, using a default one and enabling experimental telemetry so that the underlying Vercel AI SDK automatically uses this tracer.

Some implementation details:

For APM spans, we are not concerned with setting tags from transformed attributes except the model name and provider from the operation, should the attributes exist.
For LLM Observability, we parse the existing attributes based on a mapping of the operation name to our definition of the different span kinds (workflow, llm, embedding, and tool are applicable)

Additional changes unrelated to the user-facing feature include:

Slightly better LLMObs dev-ex when writing tests by providing a useLlmobs hook that will provide a getEvents function to get APM spans and pre-encoded LLMObs span events. This is just a nice-to-have that can be used in the other integrations as well.

Motivation

Closes #5410

MLOB-2980

github-actions · 2025-06-09T17:53:28Z

Overall package size

Self size: 11.86 MB
Deduped: 111.51 MB
No deduping: 111.9 MB

Dependency sizes

| name | version | self size | total size | |------|---------|-----------|------------| | @datadog/libdatadog | 0.7.0 | 35.02 MB | 35.02 MB | | @datadog/native-appsec | 10.1.0 | 20.37 MB | 20.37 MB | | @datadog/native-iast-taint-tracking | 4.0.0 | 11.72 MB | 11.73 MB | | @datadog/pprof | 5.9.0 | 9.77 MB | 10.14 MB | | @opentelemetry/core | 1.30.1 | 908.66 kB | 7.16 MB | | protobufjs | 7.5.3 | 2.95 MB | 5.6 MB | | @datadog/wasm-js-rewriter | 4.0.1 | 2.85 MB | 3.58 MB | | @datadog/native-metrics | 3.1.1 | 1.02 MB | 1.43 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | jsonpath-plus | 10.3.0 | 617.18 kB | 1.08 MB | | import-in-the-middle | 1.14.2 | 122.36 kB | 850.93 kB | | lru-cache | 10.4.3 | 804.3 kB | 804.3 kB | | source-map | 0.7.4 | 226 kB | 226 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.1 | 109.9 kB | 109.9 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | ignore | 7.0.5 | 63.38 kB | 63.38 kB | | istanbul-lib-coverage | 3.2.2 | 34.37 kB | 34.37 kB | | rfdc | 1.4.1 | 27.15 kB | 27.15 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB | | @isaacs/ttlcache | 1.4.1 | 25.2 kB | 25.2 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | shell-quote | 1.8.3 | 23.74 kB | 23.74 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | semifies | 1.0.0 | 15.84 kB | 15.84 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | ttl-set | 1.0.0 | 4.61 kB | 9.69 kB | | mutexify | 1.4.0 | 5.71 kB | 8.74 kB | | path-to-regexp | 0.1.12 | 6.6 kB | 6.6 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | module-details-from-path | 1.0.4 | 3.96 kB | 3.96 kB |

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

codecov · 2025-06-09T17:54:03Z

Codecov Report

❌ Patch coverage is 8.13397% with 192 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.08%. Comparing base (e28e38d) to head (017d15e).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
packages/dd-trace/src/llmobs/plugins/ai/index.js	8.00%	138 Missing ⚠️
packages/dd-trace/src/llmobs/plugins/ai/util.js	3.63%	53 Missing ⚠️
packages/dd-trace/src/plugins/index.js	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5858      +/-   ##
==========================================
+ Coverage   81.90%   83.08%   +1.18%     
==========================================
  Files         376      392      +16     
  Lines       16739    17343     +604     
==========================================
+ Hits        13710    14410     +700     
+ Misses       3029     2933      -96

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

pr-commenter · 2025-06-09T17:59:29Z

Benchmarks

Benchmark execution time: 2025-08-14 15:30:41

Comparing candidate commit 017d15e in PR branch sabrenner/vercel-ai-sdk-integration with baseline commit e28e38d in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 1270 metrics, 53 unstable metrics.

datadog-datadog-prod-us1 · 2025-06-09T18:07:29Z

Datadog Report

Branch report: sabrenner/vercel-ai-sdk-integration
Commit report: 9b1dfdb
Test service: dd-trace-js-integration-tests

✅ 0 Failed, 1257 Passed, 0 Skipped, 20m 11.47s Total Time

packages/dd-trace/src/id.js

packages/datadog-instrumentations/src/helpers/hooks.js

packages/datadog-instrumentations/src/vercel-ai.js

packages/dd-trace/src/id.js

packages/dd-trace/src/llmobs/tagger.js

packages/dd-trace/src/llmobs/plugins/vercel-ai.js

packages/datadog-instrumentations/src/ai.js

…r/vercel-ai-sdk-integration

ctamulonis-glossy · 2025-07-31T09:12:42Z

Hey folks! I'm excited for this PR! When do you expect it will be merged? Thanks!

gabrielsch · 2025-07-31T18:24:10Z

same here, we're waiting to see this merged so we can start using Datadog's LLM product, thank you!

Are you expecting this is going to work on AI SDK v5 which is currently in Beta?

…r/vercel-ai-sdk-integration

sabrenner · 2025-08-05T13:42:19Z

Hi @ctamulonis-glossy @gabrielsch! Yes we are looking to get this merged soon (within the next couple of weeks), and it should support AI SDK v5 😄

…Dog/dd-trace-js into sabrenner/vercel-ai-sdk-integration

…r/vercel-ai-sdk-integration

* add vercel ai integration with otel processing * add some typedocs and comments * fix tagger test * rename to 'ai' * try doing with a custom tracer * change up implementation slightly * codeowners * undo id changes * get rid of otel span start/end publishes * revert llmobs tagger test change * add better noop default tracer and esm support * delete util file * add initial test skeleton * fix duplicate wrapping * simplify patching * apm tests * write some tests * add rest of llmobs tests * add ci job * fix node version and import issue with check and externally defined version * fix metadata tagging * handle tool rolls * remove import * add default return and docstring for formatMessage * fix tool message tests * add model name and provider tags to apm tracing * some self review * address some review comments * do not stub for tests, instead use dummy test agent * move cassettes to local directory to fix tests * configurable flush interval for tests * use separate image for ai tests that have local cassettes attached * use different port * move test flush interval back local * change in esm test * Revert "move test flush interval back local" This reverts commit 39ccf68. * Revert "use different port" This reverts commit ac4071c. * Revert "use separate image for ai tests that have local cassettes attached" This reverts commit 72eca83. * Revert "move cassettes to local directory to fix tests" This reverts commit 20c3c63. * Revert "change in esm test" This reverts commit 32d75a6. * remove env var from llmobs workflow * fix type hint for test util * add test cassettes * re-trigger ci * more review fixes * try removing configuration * revert supported config undoing * add ai to versions package.json * add @ai-sdk/openai to versions package.json * fix tests * remove unrelated change * clean up tests & versions more to not use zod directly * require withVersions directly * forgotten withversions * change ci for ai job to use node latest

sabrenner · 2025-08-18T13:35:35Z

hey folks - for all watching this PR, auto-instrumentation support for Vercel's AI SDK ai has been released in dd-trace@5.63.0 😄

* add vercel ai integration with otel processing * add some typedocs and comments * fix tagger test * rename to 'ai' * try doing with a custom tracer * change up implementation slightly * codeowners * undo id changes * get rid of otel span start/end publishes * revert llmobs tagger test change * add better noop default tracer and esm support * delete util file * add initial test skeleton * fix duplicate wrapping * simplify patching * apm tests * write some tests * add rest of llmobs tests * add ci job * fix node version and import issue with check and externally defined version * fix metadata tagging * handle tool rolls * remove import * add default return and docstring for formatMessage * fix tool message tests * add model name and provider tags to apm tracing * some self review * address some review comments * do not stub for tests, instead use dummy test agent * move cassettes to local directory to fix tests * configurable flush interval for tests * use separate image for ai tests that have local cassettes attached * use different port * move test flush interval back local * change in esm test * Revert "move test flush interval back local" This reverts commit 39ccf68. * Revert "use different port" This reverts commit ac4071c. * Revert "use separate image for ai tests that have local cassettes attached" This reverts commit 72eca83. * Revert "move cassettes to local directory to fix tests" This reverts commit 20c3c63. * Revert "change in esm test" This reverts commit 32d75a6. * remove env var from llmobs workflow * fix type hint for test util * add test cassettes * re-trigger ci * more review fixes * try removing configuration * revert supported config undoing * add ai to versions package.json * add @ai-sdk/openai to versions package.json * fix tests * remove unrelated change * clean up tests & versions more to not use zod directly * require withVersions directly * forgotten withversions * change ci for ai job to use node latest

sabrenner added 2 commits June 9, 2025 13:44

add vercel ai integration with otel processing

94a2b68

add some typedocs and comments

0ed7d4c

sabrenner added the semver-minor label Jun 9, 2025

fix tagger test

233beb8

sabrenner commented Jun 9, 2025

View reviewed changes

packages/dd-trace/src/id.js Outdated Show resolved Hide resolved

rochdev reviewed Jun 9, 2025

View reviewed changes

sabrenner added 3 commits June 9, 2025 15:56

rename to 'ai'

90f4f13

try doing with a custom tracer

a4014cc

change up implementation slightly

407d6dc

sabrenner commented Jun 11, 2025

View reviewed changes

packages/datadog-instrumentations/src/ai.js Outdated Show resolved Hide resolved

sabrenner added 16 commits June 17, 2025 09:59

codeowners

16c3d28

undo id changes

fbff2a3

get rid of otel span start/end publishes

d963226

revert llmobs tagger test change

067c596

add better noop default tracer and esm support

24f37a8

delete util file

563d56f

Merge branch 'master' of github.com:DataDog/dd-trace-js into sabrenne…

7e906a2

…r/vercel-ai-sdk-integration

add initial test skeleton

47056b9

fix duplicate wrapping

2da477f

Merge branch 'master' of github.com:DataDog/dd-trace-js into sabrenne…

a43db02

…r/vercel-ai-sdk-integration

simplify patching

cc9b031

apm tests

2015808

Merge branch 'master' of github.com:DataDog/dd-trace-js into sabrenne…

2ce1798

…r/vercel-ai-sdk-integration

Merge branch 'master' of github.com:DataDog/dd-trace-js into sabrenne…

5a5a23c

…r/vercel-ai-sdk-integration

write some tests

16c5da4

add rest of llmobs tests

0c2df21

Merge branch 'master' into sabrenner/vercel-ai-sdk-integration

2938177

sabrenner and others added 7 commits August 1, 2025 12:38

Merge branch 'master' into sabrenner/vercel-ai-sdk-integration

d2b83d6

add ai to versions package.json

fce9699

add @ai-sdk/openai to versions package.json

df8349a

Merge branch 'master' of github.com:DataDog/dd-trace-js into sabrenne…

77af982

…r/vercel-ai-sdk-integration

fix tests

cc6ef27

remove unrelated change

13a92c6

clean up tests & versions more to not use zod directly

69c1712

sabrenner and others added 6 commits August 5, 2025 09:42

Merge branch 'master' into sabrenner/vercel-ai-sdk-integration

dfbbcff

Merge branch 'master' into sabrenner/vercel-ai-sdk-integration

518ea21

require withVersions directly

986ceed

Merge branch 'sabrenner/vercel-ai-sdk-integration' of github.com:Data…

57e3fc5

…Dog/dd-trace-js into sabrenner/vercel-ai-sdk-integration

forgotten withversions

5d27dd4

Merge branch 'master' into sabrenner/vercel-ai-sdk-integration

9a5ca5c

Kyle-Verhoog previously approved these changes Aug 12, 2025

View reviewed changes

sabrenner added 2 commits August 14, 2025 11:19

Merge branch 'master' of github.com:DataDog/dd-trace-js into sabrenne…

fd1e9a9

…r/vercel-ai-sdk-integration

change ci for ai job to use node latest

017d15e

sabrenner dismissed Kyle-Verhoog’s stale review via 017d15e August 14, 2025 15:19

rochdev approved these changes Aug 14, 2025

View reviewed changes

sabrenner merged commit 6cac0b9 into master Aug 14, 2025
757 of 767 checks passed

sabrenner deleted the sabrenner/vercel-ai-sdk-integration branch August 14, 2025 20:39

dd-octo-sts bot mentioned this pull request Aug 14, 2025

v5.63.0 proposal #6240

Merged

sabrenner mentioned this pull request Aug 20, 2025

fix(ai): properly handle partial experimental_telemetry configurations to wrap the correct tracer #6319

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(ai): add vercel ai integration #5858

feat(ai): add vercel ai integration #5858

Uh oh!

sabrenner commented Jun 9, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jun 9, 2025 •

edited

Loading

Uh oh!

codecov bot commented Jun 9, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Jun 9, 2025 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 bot commented Jun 9, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ctamulonis-glossy commented Jul 31, 2025

Uh oh!

gabrielsch commented Jul 31, 2025 •

edited

Loading

Uh oh!

sabrenner commented Aug 5, 2025

Uh oh!

Uh oh!

sabrenner commented Aug 18, 2025

Uh oh!

Uh oh!

feat(ai): add vercel ai integration #5858

feat(ai): add vercel ai integration #5858

Uh oh!

Conversation

sabrenner commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Uh oh!

github-actions bot commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overall package size

Uh oh!

codecov bot commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pr-commenter bot commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Uh oh!

datadog-datadog-prod-us1 bot commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Datadog Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ctamulonis-glossy commented Jul 31, 2025

Uh oh!

gabrielsch commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sabrenner commented Aug 5, 2025

Uh oh!

Uh oh!

sabrenner commented Aug 18, 2025

Uh oh!

Uh oh!

sabrenner commented Jun 9, 2025 •

edited

Loading

github-actions bot commented Jun 9, 2025 •

edited

Loading

codecov bot commented Jun 9, 2025 •

edited

Loading

pr-commenter bot commented Jun 9, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented Jun 9, 2025 •

edited

Loading

gabrielsch commented Jul 31, 2025 •

edited

Loading