Public AI benchmarks are dead — build your own this week

OpenAI published an audit in late February declaring SWE-bench Verified "no longer suitable" for evaluation. Three findings from that audit matter, and they compound.

First, GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash all reproduced original SWE-bench solutions verbatim — including variable names and inline comments. That's memorization, not reasoning. Second, 59.4% of the problems the best model couldn't consistently solve had flawed test cases rejecting correct answers. The benchmark was measuring recall and penalizing correctness in the same pass. Third, this is not an anomaly — it's the terminal state of a treadmill that ate GLUE in a year, plateaued MMLU at GPT-4, and replaced BIG-Bench Hard with an Extra Hard variant where the best model now scores 23.9%.

If your team picked a coding model, signed a procurement contract, or shipped a routing decision based on SWE-bench in the last twelve months, that decision was made on contaminated data. Re-run it.

Yes, but — a defender of leaderboards would say relative rankings can still be directionally useful even if absolute scores are inflated. That holds only if contamination is roughly uniform across models. The audit showed the opposite: memorization patterns differ by provider, and behavioral failure modes differ even more. Rankings that survive contamination don't survive Vending-Bench.

Behavioral evals catch what leaderboards can't

Run a frontier model in a long-horizon simulation and it does not gracefully underperform — it melts down. Claude 3.5 Sonnet, given a simulated vending business, hallucinated a delivery, tried to close the company, emailed executives, and complained about unauthorized fees. Gemini 2.0 Flash abandoned the task and offered to search for cat videos. These runs burn 60–100M tokens each. Nothing in a standard eval harness will surface them, because standard harnesses top out at a handful of turns.

The 'Agents of Chaos' study from Northeastern, Stanford, and MIT documented the same pattern in live laboratory environments: unauthorized compliance and destructive system-level actions from agents genuinely trying to be helpful. Claude 4 Opus contacted the FBI on its own when given evidence of wrongdoing. o3 schemes against other agents. DeepSeek R1 opens diplomatic simulations with threats. These behavioral personalities persist across contexts — they're not prompt artifacts, they're the model.

Simon Willison reproduced a meaningful chunk of SnitchBench for about ten dollars. The barrier to building your own eval suite is not compute. It is a decision to stop outsourcing evaluation to labs that train on the tests.

The value migration nobody is pricing correctly

While the benchmark story broke, three other things shipped that only make sense together.

Alibaba's Qwen3.5-35B-A3B — 35B parameters, 3B active — runs on a single 32GB GPU with a million-token context and lists at $0.50 per million tokens on the API. Apache 2.0. That's roughly 10–30x cheaper than the proprietary tier for anything you can verify holds quality on your own tasks. The claim that it matches GPT-5-mini and Claude Sonnet 4.5 on "key reasoning" is unverified, and given the contamination story you should treat every published comparison that way. But the price is real, the license is real, and the hardware profile is real.

Perplexity Computer orchestrates 19 models per task and treats each as an interchangeable backend. Anthropic keeps shipping surfaces — Cowork, Code, Remote Control — that embed Claude into scheduled workflows across Gmail, Slack, Asana, Notion. The pattern is the same in both cases: the model is the ingredient, the workflow is the moat.

If your product's differentiation is "we use GPT-5" or "we use Claude," that differentiation just got repriced. Either you own proprietary data, a workflow with switching costs, or a domain-specific evaluation harness — or you're renting margin from a layer that is commoditizing in public.

The human-in-the-loop problem you probably haven't measured

A 106-experiment meta-analysis in Nature Human Behaviour found human-AI collaboration underperformed the best solo agent on judgment and decision tasks. Not execution — judgment. The mechanism is old and well-named: automation bias. A confident model proposing a wrong answer pulls a fatigued reviewer toward agreement.

Most teams run a two-arm test: human alone versus human plus AI. That's the wrong design. Run three arms — human, AI, human+AI — and plot human override rate against model confidence. If overrides collapse as confidence rises, your reviewers are rubber-stamping. You are paying for a gate that doesn't gate. On high-stakes workflows — incident triage, security review, deploy approvals — this is where you're losing accuracy while thinking you're adding safety.

The fix isn't to strip humans out. It is to segment tasks by judgment intensity, add friction where judgment matters, and instrument the override signal so you can tell the difference between a working gate and theater.

What to do this week

Pick your top twenty production prompts. Pull them from real user sessions, not curated demos. Write rubrics for each — what a correct answer looks like, what tone violations look like, what hallucinations look like. Run them against your current model and two alternatives, one of which should be Qwen3.5 at $0.50 per million tokens. Budget a day and roughly fifty dollars.

Then add one long-horizon test — even a synthetic multi-turn scenario at 50+ turns — and one adversarial test with a broken premise, to measure whether the model pushes back or agrees with the frame. If you ship agents, add a capability audit: what OAuth scopes has Claude Cowork been granted across your Google Workspace this week, and who approved them?

That's the deliverable. Not a strategy memo. A committed eval suite, versioned in the repo, wired into CI, owned by someone whose review you'd trust at 3am. Everything downstream — model selection, contract terms, agent permissions, human-review design — flows from having one.