Evaluator

Pick an eval set, pick the runtimes and models you want to compare, and press Run. Every row builds the same cases from the same prompts. When a case ends, hidden tests are run on what the agent actually built, and the page shows how many passed, how long it took, and what it used.

This is how you answer “is the expensive model worth it here” with numbers from your own machine, instead of a vendor’s benchmark.

Off by default. Turn on the evals feature flag, then run Local Workflows: Open Evaluator from the command palette. With the flag off the command is hidden.

  1. 1. Setting up a run
  2. 2. The sets that ship
  3. 3. How a case is scored
  4. 4. How the run behaves
  5. 5. Reading the report
  6. 6. Where a run is kept
  7. Next

1. Setting up a run

Control What it does
Eval set which set runs. The sets are listed in 2
The case list every case of the set, all ticked. Untick the ones you do not want
Runs per case 1 to 20. One run of a hard case is mostly luck, so repeat it
A runtime row a runtime, a model (auto, or one from the runtime’s own list), and an effort level
Add runtime one more row
Add every runtime puts back a row for each runtime that has none
Run starts it. The line beside the button says how many sessions that is
Stop ends the run. Cases that already finished keep their results
  1. The first time the page opens it starts with one row on auto for every runtime the chat offers — Copilot, Claude, then the ACP agents. When ai.providers is set in settings.json, that list decides it. Remove the rows for runtimes you do not have.
  2. Your list is kept across opens, even an empty one. The page does not put the default rows back on its own — that is what Add every runtime is for.
  3. The model list is the same one the chat picker uses, read from ~/.local-workflows/models/<runtime>.json. The refresh button beside the picker fetches it again. See the model list.
  4. The effort level only shows when the model is not auto and the runtime reports effort levels for it.
  5. The same runtime twice with different models is fine. The exact same runtime, model and effort twice is refused, and the page says to use Runs per case instead.

2. The sets that ship

In the picker in this order:

Set Cases Limit per case
Smoke add — 2 tests 3 minutes
Node basics csv-parser (7 tests), wordcount-cli (8), todo-api (11) 10 minutes
Node real work fix-checkout (9 tests), todo-paging (17) 15 minutes
Node hard builds expr-calculator — 12 tests 15 minutes
  1. Run Smoke first. It is one tiny case. It proves each runtime starts, signs in, writes a file and reports its usage — before you pay for a long set.
  2. Node basics is three small programs written from nothing.
  3. Node real work starts from code that already exists. In fix-checkout the handed-over code passes 6 of the 9 tests and three bugs are described in a BUG_REPORT.md; the other 3 tests are the bugs. In todo-paging the handed-over code passes 11 of 17 and the other 6 are the feature.
  4. Node hard builds is one long build with operator order, functions, variables and exact error positions. Run it 3 times or more.

A case is never added to a set. Adding one would make that set’s old runs no longer comparable with its new ones. A new kind of case goes in a new set.

3. How a case is scored

  1. A case is a prompt (resources/evals/<set>/cases/<case>/prompt.md) and a folder of tests. A case may also ship starter/ — existing code, and its own visible tests, copied in before the agent starts.
  2. The agent never sees the tests. They are copied into __eval__/ in the agent’s folder only after its turn ends, and then node --test runs them. Nothing is installed.
  3. Accuracy is tests passed out of the case’s declared test count, not out of what the test runner managed to count. Code that does not load must not shrink the denominator.
  4. A failed or timed-out turn is still tested — whatever it built so far is scored. A session that never starts is an error and scores 0.
  5. The report’s Accuracy column is tests passed over the tests of every case that finished. Time is the agent’s own turn, without the tests.

4. How the run behaves

  1. All rows run at the same time. Inside one row the cases and the repeats run one after another. Starting everything at once would hit rate limits and make the times meaningless.
  2. Every attempt gets a fresh session with Allow all permissions and nobody watching. A question from the agent is answered “carry on with an assumption”.
  3. Each session is held to its own case folder. The providers are built with no extra folders, so the directory limit keeps each attempt inside its own working folder.
  4. One run at a time. A second would compete with the first for the same runtimes and make both runs’ times mean little.
  5. Closing the tab does not stop the run. The run belongs to the window, not the tab; open the page again and you see where it is.

A shell command that builds a path inside its own text can get past the directory limit, because the limit checks the paths in a tool call. Do not point the Evaluator at a machine you care about.

5. Reading the report

Above the table:

Metric What it means
Most accurate the row with the highest share of tests passed
Fastest per case the row with the lowest average turn time
Fewest tokens the row with the smallest input plus output

The table is a row per runtime-and-model, with accuracy, time and usage. Open any earlier run from the list under the setup. Run again fills the setup from a saved run, so a comparison can be repeated.

Usage is added up from the runtime’s own usage notices — the same figures the chat’s usage card reads. Each runtime’s unit is kept as it is and never converted: USD, credits and premium requests are not the same thing. Money shows only with the sessionCost feature flag on, the same rule the usage card follows.

6. Where a run is kept

Everything is in your profile, not the workspace, so a run leaves nothing in your repository.

Path What is in it
~/.local-workflows/evals/<id>.json the whole run: the set, the cases, the rows, the repeats, and per attempt its status, times, test results, failures, usage, last reply and error. Written after every change
~/.local-workflows/evals/<id>/<n>-<runtime>-<model>/<case>/run-<k>/ the folder that attempt’s agent worked in
…/run-<k>.log that attempt’s log, beside the folder rather than inside it

A run left running by a window that closed part way through shows as stopped.

Next

Back to top

Local Workflows is a VS Code extension. Everything it does is declared in a YAML file you own.


- 13-Oct-2026 02:53 PM +0000