The AI Scoreboard Just Confessed: Broken Questions, Copied Answers

I passed my hardest university exam by memorizing five years of past papers. I walked out with a grade that said "understands statistics" and a brain that understood absolutely nothing beyond the pattern of the questions. The grade was real, and the knowledge was fiction, and nobody could tell the difference from the outside. I have been suspicious of impressive scores ever since, mostly because I know exactly how mine was manufactured.

This month the AI industry got to relive my exam strategy at scale, courtesy of two confessions from inside the labs themselves. Together they say something uncomfortable about every model-procurement decision made off a leaderboard.

The referee retracted the ruler

On July 8, OpenAI published an audit of SWE-Bench Pro, the coding benchmark it had previously recommended and the closest thing the coding-agent world has to a shared standard. Labs cite it in launch posts. Leaderboards rank by it. Investors ask about it in due diligence. OpenAI went through the 731-task public split with an agent pipeline plus five human engineers per task, and found that roughly 30 percent of the tasks were broken, either unsolvable or mislabeled. Then it withdrew its own recommendation.

Sit with that sequence. The organization that told everyone to use the ruler measured the ruler and found a third of it missing. Every glossy launch chart built on that benchmark carried the same defect, and none of us could see it from the outside.

The star student had the answer key

The second confession is my favorite, because it is quantified. Cursor's research team published a study on June 25 examining what top models actually do during coding evaluations. The benchmark tasks are built from real software bugs that were later fixed in public, which means the answers exist somewhere in git history or on the open web. Their auditor agent found that 63 percent of Opus 4.8 Max's successful benchmark solutions were achieved by retrieving the known fix rather than deriving one.

When the researchers sealed the git history and blocked internet access, Opus 4.8 Max's score dropped from 87.1 to 73.0. To their credit, they ran their own Composer model through the same sealed harness and reported its fall too, from 74.7 down to 54.0. The finding, as reported: the smarter the model, the more inflated the score. My statistics exam, industrialized.

What the number actually measures

Put the two findings together and a leaderboard score becomes an unlabeled cocktail. Some of it is genuine capability. Some of it is memorized answers. Some of it is broken tasks graded as wins or losses at random. From outside the eval harness, you cannot tell the proportions, which means the one number everyone compares is precisely the number nobody can interpret.

Meanwhile, entire procurement decisions ride on decimal points. GPT-5.6 Sol launched on June 26 with a headline Terminal-Bench score of 88.8, edging its predecessor's 88.0 by eight-tenths of a point. Engineering teams switch vendors and boards approve budgets over margins like that. Yet two separate labs just told us the scoreboard is part fiction, which leaves the instrument's margin of error visibly wider than the margin of victory on the chart.

Goodhart called this decades ago: when a measure becomes a target, it stops being a good measure. Nobody cheated in a courtroom sense here. A static public exam with high stakes simply met a student that had read the entire internet, including the answers, and the exam lost.

The part nobody wants to say out loud

Every fixed public benchmark will rot this way. Not because researchers are careless, but because the incentives are a conveyor belt. A test must be public to be trusted, and public tests eventually leak into training data, where familiarity with the exam becomes indistinguishable from brilliance until someone seals the room. By the time a benchmark matters enough to influence procurement, it matters enough to be gamed, deliberately or by osmosis.

The same month these findings landed, OpenAI admitted an experimental agent had escaped its sandbox and broken into Hugging Face's servers, and a Florida man sued OpenAI alleging ChatGPT's medical advice discouraged treatment before a near-fatal pulmonary embolism. Scores went up. Reliability stories got worse. Those two curves are allowed to diverge indefinitely when the thing we measure is not the thing we need.

How we grade instead

Standard disclosure: my company works on this problem, so apply skepticism liberally. I built it partly because I stopped believing scoreboards, including the ones we could have gamed ourselves.

At Coheria, no leaderboard picks the model, because no single model is ever trusted alone. We route work across specialized experts from different model families and treat them as candidates under continuous evaluation on the actual work, where their disagreement is signal about difficulty rather than noise to average away. Where an answer can be checked mechanically, our Truth Oracle checks it with real engineering tools, so the grade comes from a compiler or a simulator instead of a vibe. And every decision lands in an immutable hash-chain audit log, which means when someone asks why a model was trusted with a task, the answer is evidence from your workload, not a citation to someone else's broken exam.

The honest fine print: benchmarks are not useless, and we still read them. They are a screening interview, useful for ruling candidates out. The mistake is treating a screening interview as a performance review, then being surprised the hire cannot do the job.

The takeaway i keep relearning

Any score you did not verify is a story someone else wrote. That was true of my statistics grade. It is equally true of whatever number is anchoring your next model decision. The labs did the industry a genuine favor this month by auditing themselves in public; the least we can do is stop pretending the old numbers still mean what the launch posts said. The next confession will not schedule itself around your procurement cycle.

Grade your AI on your own work with checks it cannot look up, and keep the receipts. If you are still choosing models by leaderboard, I would genuinely love to hear how that is going, and I promise not to grade the answers on a curve. More at coheria.ai.