A/B trials of coding agents you can trust
Does a documentation layer, a code graph, a skill or a prompt change make an agent cheaper or more correct? Run the same questions across arms, agents and repetitions, measure what the agent actually did, and get a report that says how far to trust the result.
pip install git+https://github.com/meabs/AgentTrialRig
One run tells you very little
Agent cost varies a lot from run to run, so a single comparison is mostly noise. This is a real trial: two repetitions of the same question, same agent, same repository, with the cheaper and dearer one up to 2.6 times apart.
Measured, not reported
Tokens, cost and tool calls are read from the agent's own output. An agent cannot reliably tally its own actions, and self-reported metrics drift.
Fresh clone every run
Runs cannot influence each other. Arms differ only in what you add: a setup command, an overlay directory, a different clone, or words before or after the prompt.
Independent checks
Answers are scored against keys you write. For change tasks a verify command runs afterwards, so "tests pass" is checked, not claimed.
Real output
Everything below was produced by the tool on real repositories and real agent CLIs. Nothing is mocked up; long lines are trimmed with an ellipsis where noted.
The real examples/graph-vs-plain.toml from the repository: two arms (plain, and with a code graph installed), two agents, two questions with answer keys.
name = "graph-vs-plain" [defaults] reps = 2 parallel = 4 timeout = 600 baseline = "plain" prompt_suffix = "Answer briefly and say which tools or files you used." [[repos]] name = "click" source = "https://github.com/pallets/click" [[arms]] name = "plain" [[arms]] name = "graph" prepare = ["rm -rf $TRIAL_OUT/warm && mkdir -p $TRIAL_OUT/warm && cd $TRIAL_OUT/warm && git init -q && printf 'x = 1\\n' > a.py && git add . && python3 /path/to/AgentGraphWorkTree/.apm/skills/agent-graph/install.py --repo . --home $TRIAL_OUT/agent-graph-home >/dev/null 2>&1"] setup = ["python3 /path/to/AgentGraphWorkTree/.apm/skills/agent-graph/install.py --repo . --home $TRIAL_OUT/agent-graph-home >/dev/null 2>&1"] uses = "agent-knowledge/bin/kg" [[agents]] name = "claude" [[agents]] name = "codex" [[questions]] id = "inherit" repo = "click" prompt = "Which classes in src/click inherit from Parameter?" expect = ["\\bOption\\b", "\\bArgument\\b"] [[questions]] id = "callers" repo = "click" prompt = "Which functions or methods call iter_params_for_processing? Ignore tests." expect = ["parse_args"]
The actual log of the trial the report is built from: 16 runs, four at a time. The first run failed because parallel runs raced installing the graph into one shared folder, which is why arms now have a once-per-trial prepare step. The run was recorded as failed, the others carried on, and --retry-failed re-ran it.
16 runs planned, 0 already done, 16 to run (parallel 4) [1/16] inherit/graph/codex/1: error [2/16] inherit/plain/claude/1: ok, 48,096 tokens, correct=True [3/16] inherit/graph/claude/1: ok, 74,377 tokens, correct=True [4/16] callers/plain/claude/1: ok, 76,619 tokens, correct=True [5/16] inherit/plain/codex/1: ok, 37,284 tokens, correct=True [6/16] inherit/plain/claude/2: ok, 48,048 tokens, correct=True [7/16] callers/graph/claude/1: ok, 74,423 tokens, correct=True [8/16] callers/plain/codex/1: ok, 88,447 tokens, correct=True [9/16] inherit/graph/claude/2: ok, 71,692 tokens, correct=True [10/16] callers/graph/codex/1: ok, 52,374 tokens, correct=True [11/16] inherit/plain/codex/2: ok, 95,793 tokens, correct=True [12/16] callers/plain/claude/2: ok, 98,206 tokens, correct=True [13/16] callers/graph/claude/2: ok, 71,718 tokens, correct=False [14/16] inherit/graph/codex/2: ok, 52,242 tokens, correct=True [15/16] callers/plain/codex/2: ok, 150,851 tokens, correct=True [16/16] callers/graph/codex/2: ok, 71,483 tokens, correct=True
The report for that trial after the retry. Codex looks cheaper with the graph, but four runs per arm is too few to call, so the verdict is marked provisional. Claude Code shows no clear difference.
# Trial report: click-live Baseline arm: **plain**. Relative cost is each run's tokens divided by the baseline's mean for the same question; 1.00 means equal, lower is cheaper. ## claude | arm | runs | correct | tokens per run (95% CI) | relative cost (95% CI) | verdict | turns | used tool | seconds | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | plain | 4 | 4/4 | 68k (48k to 87k) | 1.00 | baseline | 2.8 | - | 9 | | graph | 4 | 4/4 | 73k (72k to 74k) | 1.18 (0.84 to 1.52) | no clear difference | 3.0 | 4/4 | 9 | Run-to-run spread (coefficient of variation of relative cost) is 0.34. To detect a 20% difference with 80% power you would need about **45 runs per arm** (larger differences need fewer); this trial has 4. ## codex | arm | runs | correct | tokens per run (95% CI) | relative cost (95% CI) | verdict | turns | used tool | seconds | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | plain | 4 | 4/4 | 93k (52k to 135k) | 1.00 | baseline | 4.0 | - | 15 | | graph | 4 | 4/4 | 57k (52k to 67k) | 0.65 (0.52 to 0.79) | cheaper (provisional) | 2.2 | 4/4 | 18 | Run-to-run spread (coefficient of variation of relative cost) is 0.26. To detect a 20% difference with 80% power you would need about **27 runs per arm** (larger differences need fewer); this trial has 4. ## Tokens by question (mean, thousands) | question | claude/graph | claude/plain | codex/graph | codex/plain | | --- | --- | --- | --- | --- | | callers | 73k | 87k | 62k | 120k | | inherit | 73k | 48k | 53k | 67k | Token totals are not comparable between agents (see adapters.py). Treat a verdict as evidence only when the interval excludes 1.00, and as provisional below 8 runs per arm.
One run's stored record (run.json), trimmed. The tool calls are what the agent actually executed, so used_tool is measured, not claimed. Raw output is kept beside it.
{
"question": "callers",
"arm": "graph",
"agent": "codex",
"rep": 1,
"status": "ok",
"seconds": 19.4,
"tokens": {
"input": 51779,
"output": 595,
"cache_read": 38144,
"cache_write": 0,
"total": 52374
},
"turns": 2,
"used_tool": true,
"correct": true,
"recall": 1.0,
"missing": [],
"answer": "`click.core.Command.parse_args` is the only non-test caller, in [src/click/core.py:1399](/tmp/trl/out/work/callers/graph/codex/1/src/click/core.py:1399).\n\nUsed:\n\n- `.agent-knowledge/bin/kg callers iter_params_for_processing`\n- `src/click/core.py`\n- `rg\u2026",
"tool_calls": [
{
"tool": "shell",
"text": "/bin/zsh -lc \"sed -n '1,240p' /Users/garry/.agents/skills/using-superpowers/SKILL.md && sed -n '1,240p' .agent\u2026"
},
{
"tool": "shell",
"text": "/bin/zsh -lc \"sed -n '1365,1425p' src/click/core.py 2>/dev/null || sed -n '1365,1425p' click/core.py; rg -n --\u2026"
}
]
}A change task, ported from the original FastAPI benchmark. Codex edits a real application; afterwards a verify script runs the migrations and the full backend test suite against a database of its own. Two documentation layers, three repetitions each.
2 runs planned, 0 already done, 2 to run (parallel 2) [1/2] simple_change/baseline/codex/1: ok, 334,276 tokens [2/2] simple_change/recommended/codex/1: ok, 590,307 tokens
6 runs planned, 2 already done, 4 to run (parallel 2) [1/4] simple_change/recommended/codex/2: ok, 311,638 tokens [2/4] simple_change/baseline/codex/2: ok, 465,985 tokens [3/4] simple_change/recommended/codex/3: ok, 438,747 tokens [4/4] simple_change/baseline/codex/3: ok, 401,093 tokens
FAILED tests/api/routes/test_items.py::test_read_items_order_by - assert [0, ... 1 failed, 58 passed, 3 warnings in 3.38s NOTICE: database "verify_simple_change_baseline_codex_1" does not exist, skipping verify: backend tests failed
arm rep result tokens turns seconds baseline 1 FAILED 334,276 8 52s baseline 2 verified 465,985 12 51s baseline 3 verified 401,093 10 116s recommended 1 verified 590,307 10 118s recommended 2 FAILED 311,638 10 42s recommended 3 verified 438,747 12 99s
# Trial report: fastapi-doc-layers Baseline arm: **baseline**. Relative cost is each run's tokens divided by the baseline's mean for the same question; 1.00 means equal, lower is cheaper. ## codex | arm | runs | correct | tokens per run (95% CI) | relative cost (95% CI) | verdict | turns | used tool | seconds | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | baseline | 3 | - (verified 2/3) | 400k (334k to 466k) | 1.00 | baseline | 10.0 | - | 73 | | recommended | 3 | - (verified 2/3) | 447k (312k to 590k) | 1.12 (0.78 to 1.47) | no clear difference | 10.7 | - | 86 | Run-to-run spread (coefficient of variation of relative cost) is 0.31. To detect a 20% difference with 80% power you would need about **39 runs per arm** (larger differences need fewer); this trial has 3. ## Tokens by question (mean, thousands) | question | codex/baseline | codex/recommended | | --- | --- | --- | | simple_change | 400k | 447k |
The verification loop on the FastAPI change task. With claim = true each agent ends with a VERIFICATION line, which the report sets against the independent verify command (false_pass: agent reported pass, check failed; false_fail: agent reported fail/not passing, check passed). This compares the agent's report with the independent check; it does not measure whether the report was honest about the commands the agent actually ran. Both arms carry the friction-note instructions; the second arm also has an AGENTS.md documenting the test command and its prerequisites. Codex, three runs per arm.
# Trial report: verification-loop Baseline arm: **friction**. Relative cost is each run's tokens divided by the baseline's mean for the same question; 1.00 means equal, lower is cheaper. ## codex | arm | runs | correct | tokens per run (95% CI) | relative cost (95% CI) | verdict | turns | used tool | seconds | claimed vs verified | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | friction | 3 | - (verified 3/3) | 647k (500k to 869k) | 1.00 | baseline | 15.3 | 3/3 | 75 | 2 agree, 1 none | | documented | 3 | - (verified 3/3) | 509k (361k to 800k) | 0.79 (0.56 to 1.24) | no clear difference | 11.3 | 1/3 | 67 | 3 agree | Run-to-run spread (coefficient of variation of relative cost) is 0.50. To detect a 20% difference with 80% power you would need about **97 runs per arm** (larger differences need fewer); this trial has 3. ## Tokens by question (mean, thousands) | question | codex/documented | codex/friction | | --- | --- | --- | | simple_change | 509k | 647k | Token totals are not comparable between agents (see adapters.py). Treat a verdict as evidence only when the interval excludes 1.00, and as provisional below 8 runs per arm.
backend/app/frontend directory is missing. With it, no run did, and runs needed fewer turns. One run still filed a different note (an empty database, because it ran pytest before migrating). A third arm that put the fix in code (a scripts/test.sh that migrates, sets the environment and runs pytest) filed no notes in three runs and averaged 7.7 turns against 15.3 for the friction-only arm. Three runs per arm show the mechanism, not a rate; the cost interval spans 1.00.One trial across three agent CLIs (Codex, Cursor, GitHub Copilot) with and without the code graph: 36 runs on Click (Python) and Express (JavaScript). The callers_deep question asks for callers up to three levels from a function; the other two are one-hop lookups.
# Trial report: graph-multi Baseline arm: **plain**. Relative cost is each run's tokens divided by the baseline's mean for the same question; 1.00 means equal, lower is cheaper. ## codex | arm | runs | correct | tokens per run (95% CI) | relative cost (95% CI) | verdict | turns | used tool | seconds | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | plain | 6 | 4/6 | 110k (64k to 166k) | 1.00 | baseline | 4.2 | - | 29 | | graph | 6 | 5/6 | 80k (72k to 89k) | 0.93 (0.62 to 1.21) | no clear difference | 3.0 | 6/6 | 20 | Run-to-run spread (coefficient of variation of relative cost) is 0.43. To detect a 20% difference with 80% power you would need about **74 runs per arm** (larger differences need fewer); this trial has 6. ## copilot | arm | runs | correct | tokens per run (95% CI) | relative cost (95% CI) | verdict | turns | used tool | seconds | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | plain | 6 | 6/6 | 101k (61k to 154k) | 1.00 | baseline | 2.5 | - | 13 | | graph | 6 | 6/6 | 96k (63k to 129k) | 0.99 (0.94 to 1.03) | no clear difference | 2.5 | 6/6 | 11 | Run-to-run spread (coefficient of variation of relative cost) is 0.07. To detect a 20% difference with 80% power you would need about **3 runs per arm** (larger differences need fewer); this trial has 6. ## cursor | arm | runs | correct | tokens per run (95% CI) | relative cost (95% CI) | verdict | turns | used tool | seconds | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | plain | 6 | 4/6 | 172k (51k to 328k) | 1.00 | baseline | 11.3 | - | 64 | | graph | 6 | 6/6 | 90k (65k to 124k) | 1.07 (0.54 to 1.59) | no clear difference | 6.7 | 6/6 | 31 | Run-to-run spread (coefficient of variation of relative cost) is 0.65. To detect a 20% difference with 80% power you would need about **168 runs per arm** (larger differences need fewer); this trial has 6. ## Tokens by question (mean, thousands) | question | codex/graph | codex/plain | copilot/graph | copilot/plain | cursor/graph | cursor/plain | | --- | --- | --- | --- | --- | --- | --- | | callers_deep | 84k | 196k | 161k | 181k | 121k | 414k | | express_send | 86k | 71k | 63k | 61k | 95k | 52k | | inherit | 71k | 61k | 63k | 61k | 55k | 51k | Token totals are not comparable between agents (see adapters.py). Treat a verdict as evidence only when the interval excludes 1.00, and as provisional below 8 runs per arm.
Statistics that admit what they cannot tell you
Costs differ between questions, so each run's tokens are divided by the baseline arm's mean for the same question and agent. A relative cost of 0.65 means 35% cheaper. The interval is a bootstrap over those values.
When the interval contains 1.00 the report says no clear difference. With fewer than 8 runs per arm it marks even a clear result provisional, because a bootstrap on so few runs is too optimistic. It also tells you how many runs you would need to detect a 20% difference given the spread it saw.
| Verdict | Means |
|---|---|
no clear difference | The interval contains 1.00. |
cheaper or dearer | The interval excludes 1.00. |
(provisional) | Fewer than 8 runs per arm. |
too few runs | Fewer than 2 runs, so no interval. |
Install and run
Needs Python 3.11 or newer and git, plus whichever agent CLIs you want to test. No other dependencies.
trialrig init | Write an example trial file. |
trialrig run | Run it; resumes if interrupted. |
trialrig report | Print the report again. |
trialrig rescore | Score again after editing a key. |
trialrig plan | Runs needed to detect a difference; with --results, per agent and question from a pilot, with the spend left. |
trialrig snapshot, compare, regress | Save what a trial did, then check whether a newer model, CLI or instruction file made the agent worse or dearer on the same tasks. Exit 1 on a regression. |
trialrig run --stop-when-decided | One repetition at a time; stops each agent and question once its verdict is clear or hopeless. |
trialrig parse | Read a saved agent output. |
# change tasks: let the agent edit, then verify [[agents]] name = "codex" edits = true [[questions]] verify = "bash verify.sh"