All articles

Gemini 4 Carbon Shows Why Benchmark Points Don't Buy Finished Code

7 min read1,397 words
Concentration RiskEvaluation and Benchmarks
Physical separation and categorization: many similar notes are grouped together on the corkboard while one differently colored note is isolated on a separate surface.. Workplace productivity and visual task/project management in an office setting.

AI Beat News reported on November 13 that Google calls it its strongest model. Within days of launch, the record split in two. Sysnote reported on November 10 that the model's own safety report says it is not materially more capable than Gemini 4 Argon. On November 11, News BPData described significant improvements arriving alongside a model that is slow and clumsy and that often gives incorrect answers, and said the development team is actively working on fixes.

Nokia Power User reported on November 12 that the model rolled out on November 10 with Gemini app access limited to Google AI Ultra subscribers, while developers reached it through the Gemini API and Google AI Studio. AI Beat News reported on November 13 that pricing opened at $2 per million input tokens and $10 per million output, rising to $4 and $20 from November 30. Hold those facts together for a moment: a strongest-model claim, a safety report that declines to back it, early reports of frequent wrong answers, and a price that doubles within three weeks of launch, all in the same news cycle. The launch is denominated in capability points, but the thing a development organization buys is denominated in something else entirely. Carbon did not create that gap; it has just made the gap unusually hard to ignore.

The unit of value is the completed run

A coding engagement is not a quiz. It is a run: read the repository, plan the change, write the code, execute the tests, interpret the failures, repair the damage, and finish with something a team can actually merge. The buyer's unit of value is that completed, correct run, because nothing short of it ships. The run has state and history, so step twelve inherits every assumption made at step three, whether that assumption was sound or invented. A model that performs impressively on forty-nine steps and fabricates a configuration flag on the fiftieth has delivered zero units of value.

Benchmarks measure a different unit. They reward bounded successes: a scoped task gets a grade, then the next task starts fresh with no debt carried over from the last one. That design is legitimate for comparing models. A leaderboard can be accurate about every task it contains and still say nothing about whether a model finishes a week of work. The failure sits in the translation, when a launch treats movement in the bounded unit as evidence of value in the end-to-end unit and prices the product accordingly.

The two units do not convert at par.

Why point gains do not compound into delivery

Chain arithmetic explains the gap. If a model succeeds at each step nineteen times out of twenty, and a realistic run needs forty steps, the run completes cleanly about once in eight attempts, and every failed attempt still consumes tokens and attention. That math is unforgiving. Raising per-step accuracy by a point moves completion only slowly, and one class of error wrecks the curve entirely: the confident fabrication that passes itself downstream before anything checks it.

Carbon's early record is the model-level version of this problem, not a workflow fault. Slow and clumsy is a throughput cost, and throughput costs can be budgeted. Often giving incorrect answers is a validity cost, and validity costs cannot be budgeted, because a hallucination inside a run does not reduce the run's score, it changes the run's outcome. The work is not eighty percent done; it is wrong, and someone now has to find where.

An unexpected stop behaves the same way. A run abandoned at step thirty is not a partial deliverable, it is a liability, because the half-finished state must be understood before it can be resumed or safely discarded. This is why capability gains do not necessarily compound into delivery reliability: the failure modes that matter are the ones that poison everything around them, and benchmark points are indifferent to poison.

The benchmark field is questioning its own ruler

None of this is an outsider's complaint. AI researchers and the companies that make the models have themselves questioned the reliability of traditional benchmark methodologies. On DeepSWE, a popular coding benchmark, AOL reported on November 12 that models from Anthropic and OpenAI, as well as DeepSeek, score within a single point of one another, a spread that tells a buyer almost nothing about which model will finish real work.

Even the inputs are suspect. Zaobao India reported on November 11 that Apple researchers designed controllable puzzle environments, such as the Tower of Hanoi and River Crossing, specifically to avoid standard math benchmarks that are susceptible to data contamination. When careful teams must build toy worlds to find out what a model can actually do, the headline numbers in a launch deck deserve the skepticism they are getting.

The newest benchmarks concede the point by construction. Mercor released OWE-Bench on November 12, a benchmark of 26 long-horizon engineering and research tasks. Mercor says every task is graded on unseen cases and scored on a continuous scale. Under that methodology, an agent that improves a program halfway to its performance ceiling earns 50 percent rather than zero, which is more honest measurement, and it is also an admission that pass-or-fail scoring was hiding the distance between progress and completion. In delivery terms, though, a program improved halfway to its ceiling is still a program you cannot ship.

Horizon effects cut the same way. AI Weekly reported on November 11 that on the Infrathon benchmark, 135 of 150 agent runs were still improving during the second half of a 24-hour budget. In the same November 11 report, AI Weekly relayed the authors' conclusion that model rankings change with the evaluation horizon, so shorter benchmarks can mis-rank agents and hide optimization gains. If a ranking depends on when the evaluators stopped the clock, it was never measuring the thing you planned to buy.

A hallucination is a priced defect

Token pricing makes the cost concrete. Every token of an invalidated run is billed, and the retry is billed again. The diagnosis is paid in engineer hours that cost more than both. AI Beat News reported on November 13 that Carbon's listed per-token input and output rates double from November 30. A model that often gives incorrect answers does not simply deliver less value per run; it multiplies spend per delivered unit, because the denominator is completed work and the numerator includes every failed attempt.

The expensive part is rarely the tokens. Plausible wrong code must be read by someone senior enough to recognize that it is wrong, and that review happens on every run, because nobody knows which kind of run they have until the review is finished. The buyer pays once for the attempt and again for the cleanup, and only one of those payments appears on the vendor's invoice. Recovery work of this kind never appears in a launch announcement, and it never appears in a benchmark score either. It appears in your sprint.

None of this says Carbon lacks value. The reporting notes significant improvements, and the team is working on fixes. It says the metric in the launch and the metric in your cost structure are different metrics, and only one of them doubles on November 30.

What to measure before the price doubles

Evaluate the next frontier model the way you would evaluate a contractor. Measure end-to-end completion rate on your own repositories and your own tickets. Measure defect rate in the work that reaches review and recovery hours per completed task. Run the same task to completion several times and count how often it finishes without a human rescue. Benchmark deltas can shortlist candidates, but they cannot sign off on them, and this launch is a clean demonstration of why.

We use an editorial label for this situation. Generation 1 names the model-centered bargain: you buy capability and supply the missing delivery work yourself. The label does not certify any vendor's maturity. On its current record, Carbon is a Generation 1 purchase, and the delivery work it leaves behind is yours, at twice the token price after November 30.

If you have run Gemini 4 Carbon end to end on production work and your completion data tells a better story than the early coverage does, we want to see it. Bring the same measurements we described above. They are the only numbers that settle the argument.

Part of this page did not load, so its menus, forms, dialogs and animations may not work. The text of the page is shown in full. Reload the page to try again.