Claude Opus 5.5 had the best average on the speed-up and schedule tasks, GPT-6.1 Sol on the forecast task. Both averaged 0.98 or more on every correctness task.

In version 2 the two strongest models passed every run, so its scores couldn't tell them apart. For version 3 I wrote six tasks in six languages: two in real open-source code as released, four in applications I wrote for the suite. Three are open-ended: they measure how much a run improved the code, on a log2 scale, and aren't capped at my own fix's score; the other three score the share of hidden checks a run gets right. Each task keeps its own score, so there is no overall number or rank. Tasks, prompts and graders are private so later runs stay fair.

The hidden checks and hidden data never enter the model's container, and every model got the same seeds. A run whose code doesn't build or breaks the project's own tests is invalid and scores 0. Fable 5.1 and GPT-6 Astra reviewed the plan before the counted runs began; trial runs aren't counted.

This is version 3, frozen 15 October 2026: 6 tasks, 2 runs each, the same settings and the same containers for every model. Each version tests something different, so scores don't carry over from earlier versions.

Claude Opus 5.5 and Claude Haiku 5.5 ran in Claude Code 2.1.284, and GPT-6.1 Sol and GPT-6 Luna in Codex CLI 0.159.0, all at medium effort, 20 minutes per run (10 for the job service). The code was saved at the halfway point without stopping the model, and that copy was graded too. 11 runs hit the limit and were graded on the code at that point. Each run ran in its own container with no internet except the model's API, and was graded in a fresh container. Tasks that measure time were graded one run at a time with nothing else running. Runs were paced to stay under a fixed share of each subscription's usage.

Costs are estimates at each vendor's standard list prices, from the tokens each run used. Claude Code logs no usage for a run stopped at the time limit, so for those 11 Claude runs I counted the tokens from the run's own requests, the same way Claude Code counts them on runs that finished. Claude Haiku 5.5 is a range because Anthropic charges more for prompts over 100k tokens and prompt sizes weren't logged; GPT costs are a range because Codex doesn't log cache writes, which OpenAI bills at 1.25 times the input price. The runs ran on subscriptions, so these are not what was spent.

Limits: 6 tasks and 2 runs each, one tool and one effort setting per model. Two runs can't show which gaps would hold up over more runs, so none is called significant here.

Every run's task, score, time and cost is in results.json, under deep_v3_runs and deep_v3_suite.