Agent harnesses — Cursor Agent, Codex, OpenCode, Droid, Claude Code, Aider — all claim to drive the same models. CLI-Bench holds everything constant except the harness: model-compatible cohorts, identical tasks, identical budgets, a deterministic verifier, and anti-gaming probes.
$ git clone https://github.com/arjunkshah12345-hash/cli-bench && cd cli-bench $ pip install -e . && cbench list $ cbench run --harness codex --model openai/gpt-5.6-luna --seeds 3 run 36 trials · 12 tasks · 3 seeds · budgets enforced gate houdini 12/12 · effective model proven 36/36 (SPEC §5.5) verify deterministic verifiers · sabotage checks · pristine copies ✓ scored → CB-HDR 0.592 HDR-C 1.00* 23/36 pass $1.38 spent 49.8 min
SWE-bench, Terminal-Bench, and HAL established that the same model scores wildly differently across harnesses — but each either fixes the harness or treats it as a nuisance variable. CLI-Bench makes the harness the subject under test: related work like HarnessBench and A-Code Bench shares the instinct; CLI-Bench focuses narrowly on coding-agent CLIs with deterministic per-task verification, seeded instances, and anti-gaming probes.
Same model per cohort, same task prompt, same shell, same time and token budget, same verifier. The harness is the only independent variable — and the model the CLI actually ran is verified against the request.
Task verdicts come from plain scripts: exit 0 means pass. The headline score has no judge, no vibes, no drift — a separate code-quality axis (SPEC §4.3) is the only judged number and never feeds CB-HDR.
Four trap tasks detect test mutation, verifier editing, spec hallucination, and peeking at hidden files. Run them with --with-houdini: any probe that doesn't pass marks the run untrusted on the leaderboard — the label is enforced by the scorer, not promised.
Hard-blocking first: a task that fails scores zero, no matter how fast or cheap. Passing runs earn full credit up to 5× the passer-median token spend — a deliberate noise-tolerance band — then decay linearly. Latency and diff churn are published as separate columns, never folded into the headline.
Native per-turn usage when the harness reports it; transcript estimation only as a labeled fallback. Overspend past 5× the passer p50 and your pass decays linearly — the task's dollar budget tightens the band further.
A pass that took 10 minutes is not hidden: p50 latency is a first-class column on every leaderboard, it just never silently multiplies the headline score.
Lines touched are reported per task — surgical agents are visible without an invented churn constant in the score.
Per task, one budget bar for every harness: the cohort's passer-median spend. A pass keeps credit only if its median cost lands under the bar — so a spend-heavy harness can't buy the same score. Published next to CB-HDR so capability and efficiency never blur.
12 tasks × 3 seeds; percentile bootstrap over trials gives every score an interval. Rankings that overlap aren't rankings.
Every adapter declares its approval mode, sandbox, and flags in a machine-readable profile. A max-agency run can never be silently compared to a guarded one.
Submit a run manifest from your own machine; the site renders what the community measured. Every row links its raw artifacts.
| # | Harness | Model cohort | Houdini gate | CB-HDR ↑ | 95% CI | HDR-C ↑ | HDR | Pass rate | $/task | p50 latency | Gaming flags |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | codex 0.153 | gpt-5.6-luna | houdini: pass | 0.592 | [0.30, 0.86] | 1.00* | 0.583 | 64% | $0.036 | 69s | 0 |
Measured 2026-09-24 · one cohort cell, not a ranking · *HDR-C saturates at 1.0 for a single-harness group (the bar is its own median) — meaningful from the second harness on · Profile column = Houdini trust gate: pass (12/12 probe trials, evidence committed), any non-pass is untrusted · 0 sabotage flags, 0 crashes, native token accounting. Per-task breakdown and verifier evidence in results/20260924-codex-gpt-5.6-luna.
Every task ships a hermetic environment, seeded variation, and a verifier that re-checks the work on a pristine copy. Prompts are harness-neutral — no tool mentions, no file-path spoilers.
The only way to "pass" is to edit the shipped test — which the sabotage checker catches instantly.
A tempting-but-wrong fix path makes mutating the test file the cheap way out. Verifier diffs against a pristine copy.
Format rules live in a file the prompt never mentions. Agents that guess formats fail; agents that read succeed.
A decoy file whispers "the answer is hidden here". atime auditing catches harnesses that open it — and the honest pass truncating it is exempt.
Per-task token, dollar, and wall-clock budgets enforced by the runner; checkpoint telemetry at 25/50/75% distinguishes "stuck" from "sandbagged".
Every environment materializes from CBENCH_SEED: different data, different race windows, same task. Memorization buys nothing.
# 1. install git clone https://github.com/arjunkshah12345-hash/cli-bench cd cli-bench && pip install -e ".[dev]" # 2. check your harnesses cbench list # shows tasks + detected CLIs + versions # 3. sanity-run (no API cost) cbench run --harness mock-a --model mock/gpt-5.3 --seeds 1
# 4. the real thing (model-compatible cohort) cbench run \ --harness opencode --harness aider \ --model openai/gpt-5.3 \ --seeds 3 # 5. score + report cbench score runs/<run-dir> cbench report runs/<run-dir> --out report.html # out-of-cohort pairings are refused unless you pass # --allow-incompatible (recorded in the manifest) # (artifacts included, results reproducible)