GPT-6.1 Sol and Claude Opus 5.5 passed every run; Claude Haiku 5.5 passed 6 of 9 and GPT-6 Luna none.

Version 1 used tasks from public task sets. For version 2 I wrote my own tasks in one real open-source web app: an audit for data people shouldn't see, a feature that shipped broken, and a data export that has to get fast without changing its output. Each task has hidden checks the model never sees. Security and Broken feature score a run by the share of their graded parts it passes, and a run passes when every part does. Slow export scores a run half by how much of my own fix's time savings it matches and half by its query savings, capped at 1, and a run passes at 0.8 or more within the task's time and query limits. A run that breaks any of the app's own tests scores 0. Tasks, prompts and graders are private so later runs stay fair.

Before counted runs began, each task was checked the same way: my own fix scores 1, the untouched code scores 0, and the tested wrong fixes scored below 1. Three outside reviewers (GPT-6 Astra, Grok 4.7 and Gemini 4 Argon) checked every task for fairness, and I made the tasks harder twice after trial rounds where too many runs passed. The trial rounds aren't counted.

This is version 2, frozen 15 October 2026: 3 areas, 3 runs each, the same settings and the same containers. Each version tests something different, so scores don't carry over from version 1.

Claude Opus 5.5 and Claude Haiku 5.5 ran in Claude Code 2.1.284, and GPT-6.1 Sol and GPT-6 Luna in Codex CLI 0.159.0, all at medium effort, 20 minutes per run (2 runs hit the limit and were graded on the code at that point). Each run ran in its own container with no internet except the model's API. Its change was then copied into a fresh container and graded there, after the app's own tests were put back. The two tools ran side by side; the export task, which measures time, was graded afterwards one run at a time with nothing else running.

Costs are estimates at each vendor's standard list prices, from the tokens each run logged. Claude Haiku 5.5 is a range because Anthropic charges more for prompts over 100k tokens and prompt sizes weren't logged; GPT costs are a range because Codex doesn't log cache writes, which OpenAI bills at 1.25 times the input price. 2 runs stopped at the time limit before logging tokens, so they have no cost estimate and aren't in the totals. The runs ran on subscriptions, so these are not what was spent.

Limits: 3 areas and 3 runs each, one app, one tool and one effort setting per model. Two models passed every run, so these scores can't tell their ability apart on this suite, only which was faster and cheaper here.

Every run's area, score, time and cost is in results.json, under deep_v2_runs and deep_v2_suite.