[cli] add telemetry tracking for vercel dev invocations #12349

trek · 2024-10-23T13:43:44Z

This one was a beast to get right because of how much filesystem/environment mocking needed to occur, how little already existed in the repo, and how little I knew about doing these in vitest. Some notes!

added memfs which is the library vitest recommends for mocking the file system. Their example involves mocking for every test which I'm not sure yet we'd want do. Figuring out the right incantation to fully mock the filesystem in just one test file was an adventure.
Updated vitest to latest so we can stub environment variables to undefined. Our version of vitest has unstubAllEnvs but that seemed like it would just paint us into a weird corner. We use process.env.__VERCEL_DEV_RUNNING as a lock to prevent someone from running vercel dev twice in one process but in tests this is exactly what occurs. Outside of tests we use rely on the process exiting to do the cleanup for us.
I mocked the entire DevServer class very basically. We don't want to actually run it but when someone comes back to add more unit tests they should add in more spys to ensure the server is being invoked with the arguments we expect and methods are called in the correct order.

We do have our own filesystem mocking in the repo, but not as part of cli. Extracting that seemed a bit too heavy and IMO we should use a third party library for this anyway.

changeset-bot · 2024-10-23T13:43:49Z

🦋 Changeset detected

Latest commit: 5261f9d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
vercel	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

…rcel-dev

packages/cli/test/unit/commands/dev/index.test.ts

Co-authored-by: Jeff See <jeffsee.55@gmail.com>

…f github.com:vercel/vercel into trek/zero-2656-add-telemetry-tracking-for-vercel-dev

packages/cli/package.json

EndangeredMassa · 2024-10-24T04:45:36Z

packages/cli/src/util/telemetry/commands/inspect/index.ts

+  }
+
+  trackCliOptionTimeout(timeout?: string) {
+    if (timeout) {


suggestion (soft-blocking): missing implementation?

Did you intend to include the inspect telemetry in this PR?
‏

Nope! Juggling too many branches while rerunning CI 22 times

I think you still want to remove this file, but that can be a follow-up.

I'm in such a mixed up state. This is not on my local branch at all and git claims I'm up to date.

When some of our test mocks intentionally respond with a 500, that response takes several seconds because the [client's `fetch` function](https://github.com/vercel/vercel/blob/7e8fbd20e01135513ad28936155b078650732d37/packages/cli/src/util/client.ts#L173-L174) automatically retries the request. But since we're mocking these requests we should just automatically fail.

Noticed that the comment wasn't strictly true, so pass in the actual value of the `--confirm` flag to the telemetry client.

This one required adding unit tests for the `--repo` flag, which was a bit tricky.

…f github.com:vercel/vercel into trek/zero-2656-add-telemetry-tracking-for-vercel-dev

This reverts commit 0de6791.

trek · 2024-10-24T15:32:08Z

packages/next/package.json

@@ -11,7 +11,8 @@
    "test-next-local": "pnpm test test/integration/*.test.js test/integration/*.test.ts",
    "test-next-local-legacy": "pnpm test test/integration/legacy/*.test.js",
    "test-next-local:middleware": "pnpm test test/integration/middleware.test.ts",
-    "test-e2e": "rm -f test/builder-info.json; pnpm test test/fixtures/**/*.test.js",


Flakey tests related to next@canary

trek · 2024-10-24T15:32:11Z

packages/next/test/fixtures/00-server-build/index.test.js

-        { urlPath: '/fallback-blocking/on-demand-2', query: '?another=value' },
-      ],
-    },
+    // {


Flakey tests related to next@canary

EndangeredMassa · 2024-10-24T15:35:02Z

packages/cli/src/util/telemetry/commands/inspect/index.ts

+  }
+
+  trackCliOptionTimeout(timeout?: string) {
+    if (timeout) {


I think you still want to remove this file, but that can be a follow-up.

…rcel-dev

socket-security · 2024-10-24T15:47:09Z

New dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher
npm/vitest@2.1.3	environment, eval Transitive: filesystem, shell, unsafe	`+23`	3.84 MB	vitestbot

View full report↗︎

e2e tests for the Next.js builder were disabled in #12349.

This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## vercel@37.13.0 ### Minor Changes - Add telemetry for `vercel logs` ([#12386](#12386)) - Add telemetry for `vercel env pull` ([#12368](#12368)) - add telemetry for `vercel env rm` ([#12384](#12384)) - [cli] add subcommand tracking for `dns` group ([#12375](#12375)) - [cli] add telemetry for `vercel certs ls` ([#12383](#12383)) - [cli] add telemetry tracking to vercel inspect ([#12367](#12367)) - Add telemetry for `vercel certs remove` ([#12394](#12394)) - Add telemetry for `vercel integration list` ([#12404](#12404)) - add telemetry tracking to `env add` ([#12357](#12357)) - track standard environments in `vc env add` ([#12385](#12385)) - [cli] add telemetry for `vercel target ls` ([#12352](#12352)) - [cli] add telemetry tracking for `vercel domains ls` ([#12400](#12400)) - [cli] add `VERCEL_ENV` and `VERCEL` to process to simulate runtime ([#12358](#12358)) - Add telemetry for `vercel domains move` ([#12411](#12411)) - [cli] add telemetry tracking to `git connect` and `git disconnect` ([#12373](#12373)) - Add telemetry to `vercel domains buy` ([#12413](#12413)) - [cli] add telemetry tracking to `project list` ([#12339](#12339)) - Add telemetry for `vercel domains add` ([#12409](#12409)) - Add telemetry to `vercel domains rm` ([#12410](#12410)) - Add telemetry for `vercel list` ([#12364](#12364)) - [cli] add subcommand tracking for `integration` group ([#12377](#12377)) - Add telemetry for `vercel certs add` ([#12391](#12391)) - [cli] add subcommand tracking for `domains` group ([#12374](#12374)) - [cli] add telemetry tracking to `project add` ([#12340](#12340)) - Add telemetry for `vercel integration open` ([#12408](#12408)) - Add telemetry for `vercel init` ([#12371](#12371)) - [cli] add subcommand tracking for `integration` group ([#12377](#12377)) - [cli] add telemetry tracking for `vercel dns import` ([#12379](#12379)) - [cli] add telemetry for `vercel dns ls` ([#12380](#12380)) - Add telemetry for `vercel env ls` ([#12392](#12392)) - Add telemetry for `vercel bisect` ([#12362](#12362)) - [cli] add telemetry for `vercel dns remove` ([#12381](#12381)) - Add telemetry for `vercel integration add` ([#12406](#12406)) - [cli] add telemetry tracking for vercel dev invocations ([#12349](#12349)) - [cli] add subcommand tracking for `certs` group ([#12376](#12376)) - [cli] add telemetry for `vercel redeploy` ([#12353](#12353)) - [cli] add telemetry for `vercel domains inspect` ([#12407](#12407)) - Add telemetry for `vercel certs issue` ([#12401](#12401)) - Add telemetry for `vercel link` ([#12360](#12360)) ### Patch Changes - Remove stray file ([#12363](#12363)) - add metrics to dns add ([#12414](#12414)) - Add telemetry to the `teams` subcommands. Telemetry collection is not currently enabled and when it is, will be a major version bump for the CLI. ([#12346](#12346)) - add telemetry to `vc pull` ([#12387](#12387)) - [cli] refactor Output usage ([#12305](#12305)) - extract pull invocation from build ([#12372](#12372)) - extract vc pull logic from entrypoint ([#12378](#12378)) - Updated dependencies \[[`13bb443f2b0791c7bd402447fcb540cffd0b6eae`](13bb443)]: - @vercel/remix-builder@2.2.13 ## @vercel/remix-builder@2.2.13 ### Patch Changes - Update `@remix-run/dev` fork to v2.13.1 ([#12334](#12334)) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## vercel@37.13.0 ### Minor Changes - Add telemetry for `vercel logs` ([#12386](vercel/vercel#12386)) - Add telemetry for `vercel env pull` ([#12368](vercel/vercel#12368)) - add telemetry for `vercel env rm` ([#12384](vercel/vercel#12384)) - [cli] add subcommand tracking for `dns` group ([#12375](vercel/vercel#12375)) - [cli] add telemetry for `vercel certs ls` ([#12383](vercel/vercel#12383)) - [cli] add telemetry tracking to vercel inspect ([#12367](vercel/vercel#12367)) - Add telemetry for `vercel certs remove` ([#12394](vercel/vercel#12394)) - Add telemetry for `vercel integration list` ([#12404](vercel/vercel#12404)) - add telemetry tracking to `env add` ([#12357](vercel/vercel#12357)) - track standard environments in `vc env add` ([#12385](vercel/vercel#12385)) - [cli] add telemetry for `vercel target ls` ([#12352](vercel/vercel#12352)) - [cli] add telemetry tracking for `vercel domains ls` ([#12400](vercel/vercel#12400)) - [cli] add `VERCEL_ENV` and `VERCEL` to process to simulate runtime ([#12358](vercel/vercel#12358)) - Add telemetry for `vercel domains move` ([#12411](vercel/vercel#12411)) - [cli] add telemetry tracking to `git connect` and `git disconnect` ([#12373](vercel/vercel#12373)) - Add telemetry to `vercel domains buy` ([#12413](vercel/vercel#12413)) - [cli] add telemetry tracking to `project list` ([#12339](vercel/vercel#12339)) - Add telemetry for `vercel domains add` ([#12409](vercel/vercel#12409)) - Add telemetry to `vercel domains rm` ([#12410](vercel/vercel#12410)) - Add telemetry for `vercel list` ([#12364](vercel/vercel#12364)) - [cli] add subcommand tracking for `integration` group ([#12377](vercel/vercel#12377)) - Add telemetry for `vercel certs add` ([#12391](vercel/vercel#12391)) - [cli] add subcommand tracking for `domains` group ([#12374](vercel/vercel#12374)) - [cli] add telemetry tracking to `project add` ([#12340](vercel/vercel#12340)) - Add telemetry for `vercel integration open` ([#12408](vercel/vercel#12408)) - Add telemetry for `vercel init` ([#12371](vercel/vercel#12371)) - [cli] add subcommand tracking for `integration` group ([#12377](vercel/vercel#12377)) - [cli] add telemetry tracking for `vercel dns import` ([#12379](vercel/vercel#12379)) - [cli] add telemetry for `vercel dns ls` ([#12380](vercel/vercel#12380)) - Add telemetry for `vercel env ls` ([#12392](vercel/vercel#12392)) - Add telemetry for `vercel bisect` ([#12362](vercel/vercel#12362)) - [cli] add telemetry for `vercel dns remove` ([#12381](vercel/vercel#12381)) - Add telemetry for `vercel integration add` ([#12406](vercel/vercel#12406)) - [cli] add telemetry tracking for vercel dev invocations ([#12349](vercel/vercel#12349)) - [cli] add subcommand tracking for `certs` group ([#12376](vercel/vercel#12376)) - [cli] add telemetry for `vercel redeploy` ([#12353](vercel/vercel#12353)) - [cli] add telemetry for `vercel domains inspect` ([#12407](vercel/vercel#12407)) - Add telemetry for `vercel certs issue` ([#12401](vercel/vercel#12401)) - Add telemetry for `vercel link` ([#12360](vercel/vercel#12360)) ### Patch Changes - Remove stray file ([#12363](vercel/vercel#12363)) - add metrics to dns add ([#12414](vercel/vercel#12414)) - Add telemetry to the `teams` subcommands. Telemetry collection is not currently enabled and when it is, will be a major version bump for the CLI. ([#12346](vercel/vercel#12346)) - add telemetry to `vc pull` ([#12387](vercel/vercel#12387)) - [cli] refactor Output usage ([#12305](vercel/vercel#12305)) - extract pull invocation from build ([#12372](vercel/vercel#12372)) - extract vc pull logic from entrypoint ([#12378](vercel/vercel#12378)) - Updated dependencies \[[`e4304c4526f6062d23b7fdb381d9bb24c7431a0d`](vercel/vercel@e4304c4)]: - @vercel/remix-builder@2.2.13 ## @vercel/remix-builder@2.2.13 ### Patch Changes - Update `@remix-run/dev` fork to v2.13.1 ([#12334](vercel/vercel#12334)) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

[cli] add telemetry tracking for vercel dev invocations

d073b2e

Create strong-spiders-protect.md

80866c1

vercel bot deployed to Preview October 23, 2024 13:48 View deployment

trek marked this pull request as ready for review October 23, 2024 15:45

trek requested review from TooTallNate, EndangeredMassa, onsclom, jeffsee55 and erikareads as code owners October 23, 2024 15:45

Merge branch 'main' into trek/zero-2656-add-telemetry-tracking-for-ve…

c3821a8

…rcel-dev

vercel bot deployed to Preview October 23, 2024 20:17 View deployment

jeffsee55 reviewed Oct 23, 2024

View reviewed changes

packages/cli/test/unit/commands/dev/index.test.ts Outdated Show resolved Hide resolved

trek and others added 3 commits October 23, 2024 16:09

Update packages/cli/test/unit/commands/dev/index.test.ts

f3701f0

Co-authored-by: Jeff See <jeffsee.55@gmail.com>

[cli] add telemetry tracking to vercel inspect

f819339

Merge branch 'trek/zero-2656-add-telemetry-tracking-for-vercel-dev' o…

dc1b6f2

…f github.com:vercel/vercel into trek/zero-2656-add-telemetry-tracking-for-vercel-dev

vercel bot deployed to Preview October 23, 2024 21:10 View deployment

fix typo

a21b8ee

vercel bot deployed to Preview October 23, 2024 21:12 View deployment

EndangeredMassa previously approved these changes Oct 24, 2024

View reviewed changes

trek dismissed EndangeredMassa’s stale review via f9f4269 October 24, 2024 13:15

trek requested review from timneutkens, ijjk, ztanner and huozhi as code owners October 24, 2024 13:15

vercel bot deployed to Preview October 24, 2024 13:16 View deployment

trek added 2 commits October 24, 2024 08:27

Remove wrongly added file

8347148

skip flakey next e2e test

5af2f84

trek force-pushed the trek/zero-2656-add-telemetry-tracking-for-vercel-dev branch from f9f4269 to 5af2f84 Compare October 24, 2024 13:27

vercel bot had a problem deploying to Preview October 24, 2024 13:28 Failure

jeffsee55 and others added 6 commits October 24, 2024 09:11

[cli] Add telemetry for vercel bisect (#12362)

97501b3

[cli] Track --confirm flag value in vc deploy (#12361)

b620167

Noticed that the comment wasn't strictly true, so pass in the actual value of the `--confirm` flag to the telemetry client.

[cli] Add telemetry for vercel link (#12360)

5490783

This one required adding unit tests for the `--repo` flag, which was a bit tricky.

Add back wrongly removed file

717a602

Merge branch 'trek/zero-2656-add-telemetry-tracking-for-vercel-dev' o…

3a6337e

…f github.com:vercel/vercel into trek/zero-2656-add-telemetry-tracking-for-vercel-dev

vercel bot deployed to Preview October 24, 2024 14:13 View deployment

trek requested review from EndangeredMassa and jeffsee55 October 24, 2024 14:43

Pin next to get tests to pass

0de6791

vercel bot deployed to Preview October 24, 2024 14:46 View deployment

Revert "Pin next to get tests to pass"

08e2091

This reverts commit 0de6791.

vercel bot deployed to Preview October 24, 2024 14:57 View deployment

Skip e2e next tests

04666ee

vercel bot deployed to Preview October 24, 2024 15:14 View deployment

trek commented Oct 24, 2024

View reviewed changes

EndangeredMassa approved these changes Oct 24, 2024

View reviewed changes

Merge branch 'main' into trek/zero-2656-add-telemetry-tracking-for-ve…

5261f9d

…rcel-dev

vercel bot deployed to Preview October 24, 2024 15:37 View deployment

trek added the pr: automerge Automatically merge the PR when checks pass label Oct 24, 2024

trek merged commit 905d469 into main Oct 24, 2024
138 checks passed

trek deleted the trek/zero-2656-add-telemetry-tracking-for-vercel-dev branch October 24, 2024 16:31

vercel-release-bot mentioned this pull request Oct 24, 2024

Version Packages #12345

Merged

TooTallNate added a commit that referenced this pull request Oct 25, 2024

[next] Re-enable e2e tests

232b316

e2e tests for the Next.js builder were disabled in #12349.

TooTallNate mentioned this pull request Oct 25, 2024

[next] Re-enable e2e tests #12393

Merged

kodiakhq bot pushed a commit that referenced this pull request Oct 29, 2024

[next] Re-enable e2e tests (#12393)

849ecc7

e2e tests for the Next.js builder were disabled in #12349.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[cli] add telemetry tracking for vercel dev invocations #12349

[cli] add telemetry tracking for vercel dev invocations #12349

Uh oh!

trek commented Oct 23, 2024 •

edited

Loading

Uh oh!

changeset-bot bot commented Oct 23, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

EndangeredMassa Oct 24, 2024

Uh oh!

trek Oct 24, 2024

Uh oh!

EndangeredMassa Oct 24, 2024

Uh oh!

trek Oct 24, 2024

Uh oh!

trek Oct 24, 2024

Uh oh!

trek Oct 24, 2024

Uh oh!

trek Oct 24, 2024

Uh oh!

EndangeredMassa Oct 24, 2024

Uh oh!

socket-security bot commented Oct 24, 2024

Uh oh!

Uh oh!

Uh oh!

[cli] add telemetry tracking for vercel dev invocations #12349

[cli] add telemetry tracking for vercel dev invocations #12349

Uh oh!

Conversation

trek commented Oct 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot bot commented Oct 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

socket-security bot commented Oct 24, 2024

Uh oh!

Uh oh!

Uh oh!

trek commented Oct 23, 2024 •

edited

Loading

changeset-bot bot commented Oct 23, 2024 •

edited

Loading