added provider flag to diff edit cli #5334
Conversation
|
Coverage ReportExtension CoverageBase branch: 47% PR branch: 48% ✅ Coverage increased or remained the same Webview CoverageBase branch: 17% PR branch: 17% ✅ Coverage increased or remained the same Overall Assessment✅ Test coverage has been maintained or improved Last updated: 2025-08-04T21:07:18.196401 |
There was a problem hiding this comment.
Pull Request Overview
This PR adds provider support to the diff-edits evaluation CLI, enabling the use of either OpenRouter or OpenAI as the API provider. The change allows users to specify --provider openai or --provider openrouter when running evaluations.
- Added provider configuration option with OpenAI support alongside existing OpenRouter functionality
- Updated CLI interface to accept provider parameter across evaluation commands
- Enhanced dashboard error handling and display for better debugging experience
Reviewed Changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| evals/diff-edits/types.ts | Added optional provider field to TestInput interface |
| evals/diff-edits/TestRunner.ts | Added provider parameter to constructor and CLI option handling |
| evals/diff-edits/ClineWrapper.ts | Implemented provider-specific API handler initialization for OpenAI and OpenRouter |
| evals/cli/src/index.ts | Added provider CLI option to main command interface |
| evals/cli/src/commands/runDiffEval.ts | Updated command handler to pass provider option to underlying evaluation |
| evals/diff-edits/dashboard/app.py | Enhanced error reporting with detailed descriptions and guidance |
Comments suppressed due to low confidence (1)
evals/diff-edits/ClineWrapper.ts:211
- [nitpick] The variable name
openAiOptionsis inconsistent with the naming pattern used foropenRouterOptions. Consider usingopenAiNativeOptionsto match the handler class nameOpenAiNativeHandler.
const openAiOptions = {
Related Issue
Issue: #XXXX
Description
Test Procedure
Type of Change
Pre-flight Checklist
npm test) and code is formatted and linted (npm run format && npm run lint)npm run changeset(required for user-facing changes)Screenshots
Additional Notes
Important
Add provider flag to CLI for specifying API provider in diff editing evaluations.
--provideroption torunDiffEvalcommand inindex.tsandrunDiffEval.ts.openrouter.ClineWrapper.tsto useOpenRouterHandlerorOpenAiNativeHandlerbased onprovider.runSingleEvaluation()to handle provider-specific logic.NodeTestRunnerinTestRunner.tsto initialize with a provider.app.pyto handle new error codes and descriptions related to provider usage.providertoTestInputinterface intypes.ts.This description was created by
for 9c599ae. You can customize this summary. It will automatically update as commits are pushed.