Claude Haiku 5.5 vs GPT-6 Luna

Claude Haiku 5.5 ran in Claude Code 2.1.284 and GPT-6 Luna in Codex CLI 0.159.0, in the bench's Docker container with the account's apps, plugins and connectors off, at medium and at xhigh effort. Those are each tool's own settings: Claude Code's --effort and Codex's reasoning effort. Left alone, Claude Code 2.1.284 runs Haiku 5.5 at high and Codex runs GPT-6 Luna at medium, so I ran both at the same two levels. Speed, TPS and skills results use the runs that passed their check; the failed ones are counted below and kept in the data. Real app work scores every run. The plan was written down before the runs.

The speed test is the explain task the GPT-6.1 Sol page uses: the tool reads a small Python file and explains how it adds up monthly totals and what happens with a bad date, without changing anything. The clock runs from starting the tool until it exits, one run at a time, with nothing else running, except the run that replaced a failed GPT-6 Luna medium run and the Claude Code check below, which ran while other tests ran. A run passes when its answer contains both "month" and "ValueError".

TPS comes from its own test: the same task, asking for a walkthrough of the file of at least 600 words, six runs per model and effort, one at a time. TPS is the answer reply's output tokens minus its hidden thinking, each vendor's own count, over the seconds from its first text to its last, as the text reaches my machine. A run counts only when its text arrived in at least 20 separate bursts, where text more than 5 ms after the previous text starts a new one; text a service holds back and sends at once arrives in one or two, and its rate would mean nothing. In the trial runs before the counted ones, Claude Sonnet 5.5's answers arrived in 1 and 2 bursts, which is why the GPT-6.1 Sol page has no Claude TPS, while Haiku 5.5's arrived in about a hundred. Anthropic and OpenAI count tokens differently, so a Haiku token and a Luna token are not the same amount of text.

Why TPS has its own test: on the explain task, GPT-6 Luna's answers ran 82 to 132 visible tokens. On 13 of its 23 passed speed runs, on both Codex releases, they came in 6 to 11 bunches, too few to time, and OpenAI's per-turn timing event carried none of its engine fields. On the other 10, the text came word by word (61 to 90 bursts, 57.7 to 61.5 TPS) and the event carried them. At xhigh, the bunched runs reached their first words sooner, 1.1 to 1.7 s after the answer's request against 2.8 to 4.1 s, which a delay on this machine would not cause. OpenAI's timing events suggest it delivered GPT-6 Luna's text two ways. Leaving the bunched runs out would have timed only the runs that came word by word, so the TPS test asks for a long answer, which gives every run enough bursts to count.

A newer Codex release did the same. On Codex CLI 0.161.0, the newest, at medium, 5 of 6 passed and 3 streamed; at xhigh, 6 of 6 passed and 2 streamed. On 0.159.0, the counted runs gave at medium, 6 of 7 passed and 3 streamed; at xhigh, 6 of 6 passed and 2 streamed. The rest came in bunches on both releases. The newer release's runs are a check and stay out of the results.

Every failed run was GPT-6 Luna at medium on the explain task: 1 of 7 speed runs, 4 of 18 skills and ASD-STE100 runs (2 with the placebo and 2 with ponytail) and 1 of 6 runs on Codex CLI 0.161.0. Each answered in one reply without trying a tool; the ones whose text was kept said none of its tools could read the file, though Codex had given it a shell. In a probe of 36 more runs at medium, 3 did the same. On every request, the failed ones included, OpenAI's reply listed the same 11 tools, Codex's shell (functions.exec) among them, and the failed runs' prompts were 12,290 to 12,294 tokens against 12,290 to 12,296 on runs that passed. No GPT-6 Luna xhigh run and no Haiku 5.5 run did this. Results use passed runs. The speed table lists its failed run, and every failed run stays in results.json.

Claude Code 2.1.284 predates the first release that lists Haiku 5.5, 2.1.293: 2.1.284 accepts the model, but prices its runs at Opus 5.5's rates and runs it at high effort when none is set. So I reran Haiku 5.5's speed test on 2.1.293: 6.6 to 9.0 s at medium and 11.4 to 14.9 s at xhigh, against 6.4 to 7.9 s and 8.9 to 16.1 s on 2.1.284, with 503 and 1,955 thinking tokens in the answer in a typical run against 500 and 2,087. The runs overlap and every counted run set its effort, so the 2.1.284 runs stand, though I had planned to rerun everything if the default differed, and it did: with no effort set, 2.1.293 ran Haiku 5.5 at medium, the same default Codex uses for GPT-6 Luna. Only speed was checked, so skills and real-work token counts on 2.1.293 could differ. These check runs ran while other tests were running.

The skills test is the Skills page's: four small tasks (explain, feature, bug fix and build), each run three times with no skill, with a placebo instruction file and with caveman, ponytail and karpathy-skills as instruction files, for both models at both efforts. The ASD-STE100 test adds Karpathy's one-line instruction to the explain task, three runs each. Costs use each vendor's list price per token for prompts up to 100k tokens. No skills run's prompt passed 100k; eight real-work runs did, and their costs show as a range there.

Changes after the runs started, each logged with its reason: one failed GPT-6 Luna medium speed run was replaced by a seventh, and every result uses passed runs; GPT-6 Luna's costs became a range because Codex logs no cache writes; Haiku 5.5 runs whose prompts passed 100k tokens show a cost range because only the last request's size was kept; Haiku 5.5's cache writes use the one-hour rate of $0.20, the one Claude Code's requests pay; the X video shows the six passed GPT-6 Luna medium runs; TPS moved to its own long-answer test, and a TPS median needs at least three runs that count; and that test logs only Codex's WebSocket messages, because with its full trace log on, Codex 0.159.0 began to hang after its turn. The plan and every change to it, as logged, are in haiku-luna-plan.txt.

The run records are in results.json, under h2h_timing_runs, h2h_tps_runs, h2h_runs, h2h_real_runs, h2h_cli_check, h2h_tools_probe and h2h_timing_smoke.