A Statistical Analysis of LLM Benchmaxxing

Over the past two years or so, a small ritual has followed almost every new model from a Chinese lab. The release post goes up, the benchmark table looks fantastic, and within hours the replies converge on a single word: benchmaxxed.22Why (and how) labs benchmax is a whole other story, and one I deliberately stay out of. My best guess is a mix of two motives: researchers wanting to show leadership that their models keep getting better, and those same researchers hoping that, if the evaluation setup around a benchmark is general enough, gains on the benchmark will carry over to everyday tasks.

I decided to put the accusation to a statistical test. The key observation is simple: a model can't be tuned toward a benchmark that didn't exist yet. Benchmarks released after a model therefore work as a natural control group. If a model does noticeably better on older benchmarks than on newer ones, even after accounting for how capable it is and how hard each benchmark is, that gap deserves a closer look.

To measure the gap, I build on the Epoch Capabilities Index (ECI), a statistical model that estimates each LLM's capability from its scores on many benchmarks at once. I add a single parameter to it: a benchmaxxing coefficient, which captures whether a model overperforms on benchmarks released before it, relative to benchmarks released after it.

Capability minus difficulty

The idea behind ECI fits in a sentence: how well a model scores on a benchmark depends on how capable the model is and how hard the benchmark is. In symbols:

The S-shaped function σ turns the gap between capability and difficulty into a score between 0 and 1. A model far above a benchmark's difficulty aces it, a model far below it struggles, and the slope α sets how abruptly one turns into the other. Fit this to every available model–benchmark score, and out come a capability for every model and a difficulty for every benchmark, all on one common scale.

One extra term

I add exactly one term: a benchmaxxing coefficient, β, which switches on only for benchmarks that came out before the model did.

Because β sits right next to capability, it is measured in the same units. Think of it as bonus capability points that a model seems to have, but only on benchmarks that predate it.

β ≈ 0

The model does about as well on older benchmarks as on newer ones, once capability and difficulty are accounted for. Nothing to see here.

β > 0

The model overperforms on benchmarks released before it. This is the fingerprint benchmaxxing would leave behind.

β < 0

The reverse: the model does relatively better on benchmarks it could not have seen.

One more detail. Epoch's dataset is too sparse to estimate β separately for every model, so I fit it at the level of the organization: one coefficient per lab, written β_ℓ(m), shared by all of that lab's models. Each coefficient is therefore a lab's average tendency, not a verdict on any single release.

So, who's benchmaxxing?

Here are the fitted coefficients, one per lab, with 95% confidence intervals, sorted by β.

How far would the scores fall?

A coefficient is abstract, so here is what it means for the index itself. Take the most suspicious reading: all of a lab's β is benchmaxxing, and none of it is anything else. Then a model's fair capability is the one it gets when β is part of the fit, and the diamonds in the chart above show how far that moves the lab's ECI scores, on average over its models. Since a part of β may well be something other than benchmaxxing, read the diamonds as the most it could be worth.33The most at the estimated β, that is; the intervals leave room for more and for less.

For Google's 34 models, the average fall is 1.4 ECI points. It is not the same for every model. A model scored only on benchmarks that predate it loses the whole of β, or a bit more near the top of the scale;44All scores stay on Epoch's scale, which fixes Claude 3.5 Sonnet at 130 and GPT-5 at 150, so a fall is measured against those two models. They move too: OpenAI's and Anthropic's β are slightly negative, so allowing for β lifts GPT-5 by about half a point and Claude 3.5 Sonnet by about a fifth of a point against their benchmarks. Since the scale holds them at 150 and 130, the strongest models lose a little extra, which is why the newest Gemini models fall by 1.9 rather than 1.55. a model that was also scored on benchmarks released after it loses less, because those scores pin down its capability. Gemini 3.7 Flash and Gemini 3.8 Flash, the two newest Gemini models in the data, are scored almost only on older benchmarks and lose 1.9 points each. Among the 270 models in my fit, that moves them from 14th and 15th place to 20th and 24th. At the other end, Gemini 2.0 Flash (December 2024), scored on two benchmarks on each side of its release, loses 0.2. Alibaba's and xAI's models would fall by about a point; Anthropic's and OpenAI's barely move.

The launch tables, without β

The same arithmetic applies to the numbers labs print in their release posts. Take a score Google reported for a Gemini model on a benchmark that was public before the model came out. If all of Google's β were benchmaxxing, then without it the model would sit β capability points lower on that benchmark's S-curve. I take Google's number, move it down the curve by that much, using the benchmark's own slope α from the fit, and read off the score. Here are the two most recent Gemini launches, Gemini 3.7 Flash and Gemini 3.8 Flash, with every benchmark from Google's tables that is also part of the ECI, in the same version and with the same metric.55That rule leaves out most of each table. Terminal-Bench 2.1 (the ECI uses 2.0, and 28 of its 89 tasks were fixed in 2.1), HLE-Verified (a revised set of questions), OSWorld 2.0 (Google reports a partial score; Epoch counts only fully completed tasks) and GDPval-AA (an Elo rating; the ECI uses a win rate) are near misses. The rest, including SWE-Bench Pro, Terminal-Bench 3.0 and 4.0, CharXiv Reasoning and LVBench, are not in the ECI at all.

Four to five points per benchmark is more than the 1.9 points each model loses on the ECI, and that is not a contradiction: benchmark scores and the ECI are on different scales. These benchmarks are steep, and the scores sit in the middle of their S-curves, where a small shift in capability moves the score the most. Epoch's records agree with Google's numbers on FrontierCode and DeepSWE to within 0.2 points. On Humanity's Last Exam, Epoch's own run of Gemini 3.8 Flash scored 41.7, close to my 41.2, but since Google does not say how it ran the benchmark, I would not read much into that.

Twins

The coefficient comes out of a statistical model, and a model brings assumptions with it. Here is a check that needs almost none. Take two models from different labs with the same ECI, released six to nine months apart. The same capability means the same expected score on every benchmark, so which twin does better on a given benchmark should have nothing to do with when that benchmark came out. If one twin was tuned toward the benchmarks that existed when it was built, it should beat the other on those, and not on the benchmarks that appeared after its release.

Here is one such pair. Gemini 3.6 Flash (ECI 154.4, released July 2026) and GPT-5.2 (ECI 153.5, released December 2025) came out seven months apart, and Epoch scored both on the same 14 benchmarks. Ten of those were public when Gemini 3.6 Flash was released; four appeared later.

On the ten benchmarks that predate it, Gemini 3.6 Flash scores 2.4 points higher than its twin on average; on the four that came after it, 1.2 points lower. That is the pattern benchmaxxing predicts. But look at the spread: single benchmarks range from 21 points in GPT-5.2's favor to 29 in Gemini's, and the largest gap is on SimpleQA Verified, a benchmark built by researchers at Google DeepMind and Google Research.66Haas et al., SimpleQA Verified (2025). A lab doing well on a benchmark its own researchers built is a story of its own, and the coefficient cannot tell it apart from benchmaxxing. Dropping SimpleQA Verified, DeepSWE and Chess Puzzles together, the three benchmarks that matter most to Google's coefficient, shrinks it to about 1 ECI point (Part 2, Section 2.6). One pair is an anecdote.

So I did it for every pair. Among the models with a published ECI, 74 pairs consist of a Google model and a model from another lab within 2 ECI points of it, released 6 to 9 months apart. Another 247 such pairs contain no Google model, and they are the control. For each pair I take the focal model's advantage over its twin on benchmarks that predate the focal model, minus its advantage on benchmarks that came out after it. That needs benchmarks on both sides, which leaves 45 of the Google pairs.

Two things stand out. First, the later twin does better on older benchmarks even when Google is not in the pair: at the same ECI, newer models lean toward older benchmarks, whoever made them. That is why the comparison needs a control. Second, Google's models go the way the coefficient says in both orientations, by 2.0 and 1.6 points beyond the control. But the standard errors are two to three times that size. The twins point the same way as β; on their own, they cannot tell it from noise.

For anyone who leans heavily on these models in day-to-day work, the results may not come as a big surprise.

Notably, this wasn't the only analysis in which Google came out on top. I repeated the exercise under a range of alternative specifications (Part 1), and then played referee to my own result, with fifteen objections a careful statistician could raise and the analysis that answers each (Part 2). Google's pattern survives most of them. Under the tests that assume the least, which resample the models, shuffle release dates or resample the benchmarks, it has a p-value between 0.01 and 0.02. (Which is also why I asked Gemini to define the term at the top. Who better to ask?) I did not single out Google before looking, though, and once that search over labs is counted, the evidence sits at the edge of the usual 5% bar.

Two of the objections change what the pattern means, though neither says it is not benchmaxxing. It is not a jump at the release date. Google's models do best on benchmarks released about a year before them, and less well on both older and newer ones. A smooth curve over benchmark age, fitted for each lab, explains the data better than the step, and once the curve is in, the step is gone. That is not the curve swallowing a real step: in simulated data with a step of Google's size planted at the release date, the curve finds it at full size. The curve is less sure-footed with a gentler kind of benchmaxxing, a lift on whatever benchmarks were current in the two or three years before a model: planted the same way, about half of it, or more, disappears into the curve. What is left is a profile over benchmark age. It may be tuning to the benchmarks of recent years, which is a kind of benchmaxxing, or a relative weakness on the newest kinds of benchmark, which is not.

There is evidence for some of the second. The newest benchmarks are mostly agentic: of the 19 released in 2026, 14 have the model work over many steps with tools, in a terminal, a computer, a game, a Python sandbox or a proof checker. Google's models score lower on these than other labs' models of the same ECI, by about 3.5 ECI points, more than any other lab. That accounts for most of Google's decline on newer benchmarks and about a third of its lead over the other labs. It does not account for all of it. On the benchmarks that are agentic in the strict sense, OpenAI, Alibaba and DeepSeek fall short by as much as Google, and with those benchmarks left out, Google's pattern remains. How much the gap explains also depends on how a few benchmarks are classified, chiefly the two FrontierMath sets of 2026, which Epoch runs with a Python tool. And I went looking for it only after seeing the result. A weakness on agentic and tool-using work is part of the story, not all of it; new Gemini releases will help tell the rest.

When I started this project, I fully expected the Chinese labs to light up, and had more or less drafted my closing line: Reddit knew it all along. The data had other ideas. The Chinese labs' coefficients are all smaller than Google's, and every one of their intervals includes zero. The one lab that stands out is on the other side of the Pacific, and whether it is benchmaxxing is a question the data cannot yet settle.

The long versions

This post is the short version. Two companion posts hold everything else. Both were written by Claude Opus 5.5, an AI model made by Anthropic, from the analyses behind this post.