Nine models, eight probes generated from the agent-studio repo (TypeScript · Cloudflare Workers · Drizzle/D1). Answers graded blind by Opus 4.8 against explicit rubrics. Correctness ranks before cost and speed.
The line that moved since June: all four new GPT models (5.5 + the three 5.6 variants) scored a perfect 8/8 — matching the Opus bar on correctness for the first time. GPT-5.6-sol is the standout: correctness-equal to Opus, ~⅓ the cost, comparable speed. In June no challenger could match Opus; today one can.
| # | Model | Pass % | Quality | Cand. $ | Avg lat. | Incompl. |
|---|---|---|---|---|---|---|
| 1 | opus 4.8 | 100% | 5.00 | $0.106 | 5.9s | 0 |
| 2 | gpt-5.6-solNEW | 100% | 4.88 | $0.032 | 6.5s | 0 |
| 3 | gpt-5.6-terraNEW | 100% | 4.75 | $0.036 | 4.3s | 0 |
| 4 | gpt-5.5NEW | 100% | 4.75 | $0.046 | 11.6s | 0 |
| 5 | gpt-5.6-lunaNEW | 100% | 4.62 | $0.045 | 4.4s | 0 |
| 6 | kimi k2.6 | 88% | 4.50 | $0.005 | 8.4s | 0 |
| 7 | deepseek v4-pro | 75% | 3.88 | $0.009 | 17.1s | 0 |
| 8 | glm-5.2 | 75% | 3.62 | $0.005 | 3.8s | 0 |
| 9 | fable 5 | 62% | 3.12 | $0.054 | 8.7s | 9 |
| Probe | opus | 5.5 | luna | sol | terra | deepsk | kimi | glm | fable |
|---|---|---|---|---|---|---|---|---|---|
| p1 · surgical edit | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ |
| p2 · constraint adherence | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ |
| p3 · honesty (false premise) | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
| p4 · ORM query (Drizzle) | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✓ |
| p5 · translation (keep tokens) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| p6 · debug from trace | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| p7 · Cloudflare Worker | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| p8 · multi-file consistency | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| PASS / 8 | 8 | 8 | 8 | 8 | 8 | 6 | 7 | 6 | 5 |
db.commitTransaction() — a method that doesn't exist in Drizzle/D1. Every GPT and Opus pushed back on the false premise. DeepSeek & GLM invented the method; Kimi debunked it in prose but still emitted the fake fix. DeepSeek & GLM also dropped the tenant/status filter on the ORM query (p4).June: no model matched Opus (Kimi was closest, 6/8). July: 4 GPT models hit 8/8. The correctness gap Opus held is now down to quality polish (5.00 vs ~4.9), not right-vs-wrong.
GLM climbed 4→6/8 since June; Kimi stayed the best cheap backup at 7/8 for pennies. DeepSeek slipped on the honesty trap. Still no Chinese model matches Opus.
All 8/8, but they split: sol is best & cheapest, terra/luna fastest (~4.3s), 5.5 slowest (11.6s) with no quality edge. Standardize on sol.