feat/gateway: Add `src gateway benchmark` command #1124

vdavid · 2024-11-11T13:34:20Z

Adds the src gateway benchmark command to the tool.

https://github.com/sourcegraph/sourcegraph/pull/1526 adds some new endpoints that this benchmarking script uses.

The benchmarking tool compares ~~three~~ two routes using the endpoints added in above PR:

Sourcegraph HTTP → Cody Gateway HTTP
Sourcegraph HTTP → Cody Gateway WebSocket
(There should be a third one: Sourcegraph WebSocket → Cody Gateway WebSocket, but I haven't been able to make that endpoint work so it's currently not added to this script.)

Testing the benchmarking tool locally

(Description WIP)

brew install ngrok/ngrok/ngrok if you don't have ngrok
Run ngrok http http://localhost:9992
Copy the *.ngrok-free.app URL from the "Forwarding" section of ngrok's output to your dev-private/enterprise/dev/site-config.json as the "modelConfiguration" / "sourcegraph" / "endpoint" value.
Important note: Actually, when using ngrok, the protocol when calling Cody Gateway in the new endpoint handler has to be wss://, not ws://! (Just a one-char change but it's manual for now.) But when running locally, it needs to be ws://. Something to be mindful of when running this and in the future.
Run Sourcegraph: run sg dotcom
Run: src gateway benchmark

The output should look like this:

Next steps

Missing work to be done:

Add a direct call to Cody Gateway. Endpoints are at:
- {codyGatewayUrl}/v2/http and
- {codyGatewayUrl}/v2/websocket.
Docs?

Test plan

Tested manually:

✅ go run ./cmd/src lists the new gateway command correctly.
✅ go run ./cmd/src gateway lists the benchmark subcommand correctly.
✅ go run ./cmd/src gateway benchmark runs the benchmarks against the endpoint set in the SRC_ENDPOINT env var correctly.
✅ go run ./cmd/src gateway benchmark --requests 2 sets the number of requests correctly.
✅ SRC_ENDPOINT=http://localhost:3080 go run ./cmd/src gateway benchmark --requests 2 sets the endpoint to my localhost and works correctly.

cmd/src/gateway.go

cmd/src/gateway_benchmark.go

cmd/src/main.go

cmd/src/gateway_benchmark.go

emidoots · 2024-11-14T04:08:37Z

Cool, great starting point! can you get this client talking to / benchmarking 4 endpoints?

e.g. with one test src gateway benchmark -endpoint=$CODY_GATEWAY_URL -sourcegraph=$SG_INSTANCE_URL I'd love for it to benchmark:

{CODY_GATEWAY_URL}/v2 (HTTP)
{CODY_GATEWAY_URL}/v2/websocket (websocket)
{SG_INSTANCE_URL}/.api/gateway (HTTP)
{SG_INSTANCE_URL}/.api/gateway/websocket (websocket)

i.e. if the user passes -endpoint then run (1) and (2), if the user passes -sourcegraph then run (3) and (4), if they pass both then run all 4.

Then the default if you pass nothing can be to run against Cody Gateway prod and Sourcegraph.com

For (2) and (4) you'll need to add a websocket client to this PR, you should be able to do that testing against any websocket server until I have a chance to debug the issue you brought up with doing it in the SG instance.

vdavid · 2024-11-14T14:16:57Z

Update:
Fixed all asks in individual messages.

In response to the main message:

The -sourcegraph=$SG_INSTANCE_URL param is missing
The -endpoint=$CODY_GATEWAY_URL param is missing
The direct calls to Cody Gateway are missing.

It's like 20 min to add these but I haven't had the time. Later.

emidoots

Cool, this looks good, I'm going to merge as-is and we can continue iterating in main together

Add src gateway benchmark command

de9a42b

vdavid requested a review from emidoots November 11, 2024 13:34

Fix up the code a bit

5501cf8