Every model we fitted, from the simplest up

When the treatment is a date, everything that changes with time is a rival.

1.1 The question, and the ladder

The main post asks whether language models do better on benchmarks that existed before they did, after accounting for how capable each model is and how hard each benchmark is. Benchmarks released after a model are a natural control, because nobody can tune a model toward a test that does not exist yet. Epoch AI's Capabilities Index (ECI) puts 270 models and 60 benchmarks on one scale, so the comparison needs nothing but public scores. Most contamination checks need the model itself, to rerun it on rephrased or freshly written items (Dekoninck, Müller and Vechev 2024; Zhang et al. 2024); this one does not.

This post is the long version. It describes every model we fitted, from the simplest to the most flexible, and the two ends of that ladder disagree in an instructive way. The most flexible model gives every model its own curve over benchmark age. It finds effects as large as 148 ECI points, on a scale where Claude 3.5 Sonnet sits at 130 and GPT-5 at 150, and it predicts held-out scores worse than Epoch's model in every fold. One of the simplest gives each lab a single shift on the benchmarks that were public when its models came out. It finds one lab that stands out, Google, at $+1.55\pm 0.53$ ECI points, and no other lab more than 1.3 standard errors from zero.

Each rung below adds one thing to Epoch's model. For each we report what it bought in prediction, as the change in held-out squared error under 10-fold cross-validation over cells (24.80 for Epoch's model, so negative is better), and what it said about Google.

In short, every rung that can estimate a step per lab finds Google's, and only Google's. The rungs that free the shape of the effect show why the step is a summary rather than a mechanism: the bend the data prefer is not at the release date. Part 2 puts the Google result through fifteen objections and ends with a sharper reading, which we carry into the last section here.

1.2 The data, and Epoch's model

A cell is a model and a benchmark for which Epoch reports a score: 2,843 cells, 17.5% of the 270 by 60 matrix. A model has a median of 9 benchmarks (4 to 30), and a benchmark a median of 33.5 models (6 to 182). Each model has a release date $t_i$ and each benchmark a public release date $s_j$. The gap ${\Delta }_{ij}=t_i-s_j$, in years, is positive when the benchmark was public at the model's release. We call those cells post and the others pre; 29.5% of the cells are pre.

Scores. Epoch rescales each raw score so that chance is 0 and the attainable maximum is 1, and clips it to $[0.001,0.999]$ for the fit. We quote residuals on this scale as points of score, from 0 to 100, and effects in ECI points; on Google's cells, one ECI point is worth about two and a half points of score. There are no item counts, so a score of 0.6 on a 40-task agentic benchmark and on a 14,000-question multiple-choice test enter the fit identically. When a model has several raw results on a benchmark (different settings, scaffolds or sources), the cell keeps the largest, which biases it upward a little. Part 2 checks that neither choice drives the result.

Data. All fits use Epoch's public benchmark snapshot of 29 September 2026 (CC-BY 4.0), the same input as the published index. Epoch has updated the data twice since, on 1 and 2 October, with 53 new cells: 52 for four new models (GPT-6 Sol, GPT-6 Luna, GPT-6.1 Sol and Grok 4.7) and one for Claude Opus 5.5. No existing score, date or lab changed, and no Google model was added. Refitted on the 1 October snapshot, the headline comparison of Section 1.6 is unchanged ($z$ of 3.05 and 3.17, against 3.04 and 3.18; Part 2). The code at github.com/aemrebarut/eci-benchmaxxing reruns every fit in this post on whichever snapshot you give it, including the current one.

The model. Epoch's index is a two-parameter logistic item-response model (Birnbaum 1968; Lord 1980), fitted to aggregate scores rather than to item responses (Ho, Denain, Atanasov, Albanie and Shah 2025). Model $i$ has a capability $C_i$, benchmark $j$ a difficulty $D_j$ and a slope $a_j$, and

with $\sigma$ the logistic function and the parameters chosen by least squares with a small ridge. Reported capabilities are an affine map of $C$ that puts Claude 3.5 Sonnet at 130 and GPT-5 at 150, so one raw unit is about 19 ECI points. Every effect below goes through the same map. The unit moves a little when an extension moves the two anchor models, so comparisons across models are best made in standard errors. Our refit with Epoch's code reproduces the published index (correlation 0.99997, median difference 0.006 ECI points). Epoch's default solver stops slightly short of the optimum, harmlessly for the index, but on one of the extensions below it reported success 5.1 ECI points from the minimum, so every fit here uses an exact solve with a check of convergence.

What the model leaves out matters more than what it contains. There is no time: a model's capability is one number for all its benchmarks, whatever their age. There is no variance model: every cell has the same weight. And the cells of one model are independent given its capability. The first absence is the subject of this post; the other two are Part 2's.

1.3 Rung 0: what the residuals say

Before adding anything, we looked at the residuals of Epoch's own fit, $y_{ij}-\sigma ({\widehat{a}}_j({\widehat{C}}_i-{\widehat{D}}_j))$, against the gap. They have a standard deviation of 7.5 points of score, and pooled over all models their smoothed curve is flat to within 2 points. Split by lab, the mean residual on post cells minus pre cells is $+2.9\pm 0.9$ points for Google, $+1.1$ for Alibaba, $-1.1$ for OpenAI and $-1.6$ for Anthropic (with cells treated as independent, which flatters the standard error). Split further by release year (Figure 1.2), Google's 2026 models show a bump of about 7 points at a gap of 0.7 years, and its 2025 models a smaller one.

This is where the Google signal first appeared, and it is also where a signal can be manufactured. These residuals come from a fit in which Google's own models helped to set every capability and every difficulty, and any process the model leaves out that differs between old and new benchmarks can leave a pattern in them. The next step is to put the effect inside the fit. First, though, we need to know which effects the data can see at all.

1.4 What these data can see

Every model on the ladder keeps a free capability for each model and a free difficulty for each benchmark, and both follow time closely: in Epoch's fit the model's release date correlates 0.90 with its capability, and the benchmark's release date 0.85 with its difficulty. So any effect of the form $u(t_i)+v(s_j)$, one function of the model's date plus one of the benchmark's, is absorbed by $C_i$ and $D_j$. The gap is a function of both dates, and a function of the gap is absorbed exactly when it splits that way. For a continuous $\phi$,

(Set $s_j=0$ and then $t_i=0$ to write $u$ and $v$ in terms of $\phi$, and the identity becomes Cauchy's functional equation.) This is the identification problem of age-period-cohort analysis (Mason, Mason, Winsborough and Poole 1973; Holford 1983), in which age is period minus cohort and only the nonlinear part of each effect can be estimated. Here the model's date plays period, the benchmark's date cohort, and the gap age.

It follows that these data can only see bends in the gap: steps, kinks and curvature, never a straight-line trend. A common gain that grows steadily with a benchmark's age is indistinguishable from newer models being a little more capable and newer benchmarks a little harder.

Two consequences shape the ladder. First, the simplest effect the data can see is a step at the release date, which is also the simplest model of benchmaxxing: a gain on every benchmark that was public when the model came out. Second, everything else that bends in time competes with it: benchmarks whose scores drift with the model's date, ceilings, the best-of-several-results selection, and models that are relatively good at the benchmarks of their own era. The ladder adds these one at a time.

1.5 Rung 1: a step at the release date

The step model adds a constant to a model's capability on every benchmark that was public at its release,

with $g$ the group a model belongs to. A level for the pre cells would be absorbed by $C_i$, so only the difference ${\beta }_g$, post minus pre, is identified, and only through models with cells on both sides of their release. This is the coefficient of the main post. We report near-unpenalized fits (a ridge of ${10}^{-3}$ on the steps), use the stronger ridge that cross-validation picks where the question is prediction, and give as the standard error the larger of two estimators, one clustered by model and one model-based. Whether those standard errors can be trusted is Part 2's subject.

One step for everyone. With a single $\beta$ shared by all models, 196 of which have cells on both sides of their release, the step is $+0.20\pm 0.40$ ECI points, and adding it makes held-out predictions very slightly worse. There is no common jump at the release date.

One step per lab. Google appears in Epoch's data under three lab names: Google DeepMind; "Google DeepMind,Google", for two versions of Gemini 2.0 Flash; and Google, for Gemini 3.1 Flash-Lite. We merge them into one group of 34 models, 28 of them with cells on both sides. With one step per lab, 19 labs are identified. Google's step is $+1.55\pm 0.53$ ECI points, about 2.9 standard errors, and this is the headline number of the series. No other lab is more than 1.3 standard errors from zero. The next largest step is Alibaba's, $+1.02\pm 0.82$; OpenAI's is $-0.43\pm 0.51$ and Anthropic's $-0.11\pm 0.50$. The main post shows all of them in its forest plot. Cross-validation prefers the steps to Epoch's model, but barely (by 0.013 on 24.80, in 6 of 10 folds), and with the ridge it picks, Google's step shrinks to $+0.44\pm 0.25$. The steps explain a small part of the variation in scores, which is what a modest effect looks like.

The signal is Gemini's; Gemma has too few models with cells on both sides to say anything. Cross-validation cannot tell the groupings apart. Leaving out one benchmark at a time, the merged step stays between $+1.14$ (without DeepSWE) and $+2.09$ (without ProofBench), so no single benchmark carries it. Part 2 removes several at once.

One step per lab and year. Finer still, with one step for each lab in each release year (47 identified groups, using Epoch's lab names), cross-validation prefers no steps at all: held-out error falls steadily as the ridge grows, until every step is within 0.1 ECI points of zero. The near-unpenalized steps look like noise. Their precision-weighted mean is $+0.25\pm 0.33$, a test that all 47 are equal gives $p=0.99$, and the largest are Anthropic 2024 ($z=-3.4$), Google 2026 ($+2.0$) and Google 2025 ($+2.0$), where two or three values beyond 2 are expected by chance among 47. A year of one lab's models spreads the same data too thin.

1.6 Rung 2: benchmarks that drift

Figure 1.3 shows the largest thing Epoch's model leaves out. At the same ECI, models released before 2024 score higher on Winogrande and HellaSwag than models released later, and lower on GSM8K.

The step model's residuals say the same thing benchmark by benchmark. Regressed on the model's release date within each benchmark, 9 of 51 slopes are beyond two standard errors. Old commonsense and knowledge benchmarks drift down (Winogrande, HellaSwag, LAMBADA, TriviaQA, ARC AI2), and GSM8K and the agentic benchmarks drift up. Within a benchmark, pre and post cells differ in model date, so a drift is partly a pre-post difference, and the two must be fitted together. The drift model gives each benchmark its own linear trend in the model's date,

where ${\overline{t}}_j$ is the mean model date among benchmark $j$'s cells. A drift common to all benchmarks is absorbed by $C$ and $D$, so only differences between benchmarks are identified, and a ridge on $\kappa$ fixes the common part. With the drift in, ${\beta }_g$ is identified by the step at each benchmark's release date against that benchmark's own trend.

This is the largest improvement on the ladder. With the ridge on the drifts chosen by cross-validation, held-out error falls by 2.36 on 24.80, about 10%, in 9 of 10 folds, and the steps add a further 0.02. The number of benchmarks whose residuals still trend beyond two standard errors falls from 9 to 2. The drifts are large (Figure 1.4): TriviaQA $-17.7$ ECI points per year of model date, HellaSwag $-9.8$, Winogrande $-8.7$; DeepSWE $+14.8$, OpenBookQA $+9.4$, GSM8K $+9.0$, APEX-Agents $+8.7$. Fifteen of the 60 are beyond two standard errors.

Every lab's step rises. With the drift in, Google's step goes from $+1.55$ to $+2.23\pm 0.57$, and most other labs rise with it (Figure 1.5): OpenAI from $-0.43$ to $+0.38$, Anthropic from $-0.11$ to $+0.64$, DeepSeek from $+0.25$ to $+1.19$. The single common step goes from $+0.20$ to $+0.95\pm 0.43$. This does not mean that everyone benchmaxxes. The benchmarks that drift down are old, so nearly all their cells are post cells, and they were scored mostly on models from 2023 and 2024. Without a drift term, the fact that newer models do worse on them than their capability predicts is charged to the post cells of the labs whose models were scored on them, which pulls those labs' steps down. With the drift term it is charged to ${\kappa }_j$ instead, and those steps rise together; Meta's, whose models sit differently across the drifting benchmarks, barely moves. The common part of the steps is the part the design protects least. It is a step at gap zero pooled across labs, and any smooth time effect the drift does not capture can load on it.

So the comparison that says which lab stands out is not each step against zero, but each step against the others. We use Google's step minus the precision-weighted mean of the other labs' steps,

with the full covariance of the estimates. Without the drift, the other labs average almost exactly zero, and this contrast is $+1.55\pm 0.51$ ECI points. With the drift they average $+0.62$, and the contrast is $+1.61\pm 0.51$. The drift lifts the floor, and Google's distance from it stays where it was. This contrast is the number Part 2 defends.

What the drift is, the data in hand cannot say. It is a date effect at fixed capability, and two explanations fit it. One is a change of protocol, for example from scoring a base model by the likelihood of each answer option to letting a chat model generate its answer, which can move scores either way depending on the task. The other is the training mix of each era. Both concern how a score came to be rather than benchmaxxing, and Epoch's raw rows, which record the settings behind each score, could tell them apart.

1.7 Rung 3: ceilings instead of drift

The benchmarks that drift down share a second feature: their best scores are low. On Epoch's scale, the highest score on Winogrande is 0.78, on LAMBADA 0.80, on TriviaQA 0.88 and on HellaSwag 0.94. A benchmark with mislabelled or ambiguous items has a ceiling below 1, and a logistic curve forced to reach 1 bends to compensate, which could look like drift. So we gave each benchmark a ceiling, multiplying its curve by $1-c_j$ with $0\le c_j\le 0.9$ and a ridge on $c_j$, and compared four structures by the same cross-validation.

The ceilings explain almost none of the drift. Winogrande's fitted ceiling is 1.00, for example, and its drift barely changes. The low maximum is a fact about which models were scored, not about the benchmark. No model released after 2024 has a score on Winogrande, HellaSwag, LAMBADA, TriviaQA, ARC AI2, GSM8K or MMLU, and the most capable model scored on any of them has an ECI of 132, against 168 at today's frontier. Nobody near the top was tested, so the curve has nowhere to bend. Where strong models were scored, the ceilings are real and sensible: SimpleBench 0.82, MMLU 0.87 (in line with its known label errors), ARC-AGI-2 0.93 and GPQA diamond 0.93. They improve the fit a little and change nothing about Google.

1.8 Rung 4: one free curve for all models

So far the effect has been a step. Section 1.4 says the data can see any bend, so the next rung frees the shape, while keeping it common to all models: every model gets the same curve over the gap, with deviations by lab and by model shrunk toward it. The curve is zero before a kink at $x_0$, rises with slope one just after it and levels off at $\tau$, $h(\Delta )=\tau (1-e^{-max(\Delta -x_0,0)/\tau })$; its coefficient $m$ is in ECI points per year of gap. The model also writes the rival explanations in as terms of their own, rather than leaving them to the residuals: a slope per model on the benchmark's date (a model relatively good at the benchmarks of its era), a steepness per model, and a selection term for cells that are the best of several raw results. It has no benchmark drift, so the ladder is not strictly nested here: this rung is more flexible in the shape of the effect and in its rivals, not in everything.

Because $\tau$ exists only when there is an effect, the test of $m=0$ has a nonstandard null distribution (Davies 1987; Hansen 1996), which we calibrated by simulation. We drew 1,000 replicates from the fitted model without the effect and 1,000 wild-bootstrap replicates with one random sign per model (Wu 1986; Mammen 1993), and refitted each from scratch.

There is a bend. At $\tau =4$ years, $m=+2.57$ ECI points per year of gap (bootstrap SE 0.87), or $+2.27$ points at a gap of one year, with $p=0.009$ against the simulated null and $p=0.020$ against the wild bootstrap. It survives seven variants of the model, with $m$ between $+2.0$ and $+3.6$, including one with all the rival terms dropped. One further variant is a warning rather than a check: a small ridge on the capabilities pulls the strongest models toward the anchors, the gap term compensates, and the effect inflates to $+4.7$. The bend is also common. The labs' own slopes, shrunk toward the shared one, run from $+2.3$ to $+3.0$, and no lab, Google included, is two standard errors from the common slope.

But the bend is in the wrong place. If the effect came from models having seen benchmarks, the bend would sit at the release date. Moving the kink and refitting, the evidence for a bend is strongest a year before the benchmark existed (Figure 1.6). We know what this profile looks like under each hypothesis, because every simulated data set was refitted at every kink location. With no effect the profile can peak anywhere, but the observed advantage of a kink at $-1$ over a kink at $0$ occurs in only 0.1% of replicates. With the fitted effect and its kink at the release date, the profile peaks at zero in 68% of replicates and the observed advantage never occurs (0 of 500), although the test detects that effect in 83% of them. So the common bend is real, and it is not tied to the release date. Early access to benchmarks before their public date would have to come a year early, for every lab. Something that changes smoothly with calendar time and differs between old and new benchmarks, like the drift of rung 2, is the likelier reading.

The kink's location is a placebo: it asks whether the effect starts where the hypothesis says it should. Part 2 turns the same tool on Google's step, with a similar result.

1.9 Rung 5: a free curve for every model

The most flexible model gives every model its own coefficient on a shared curve. It adds ${\psi }_{i}f({\Delta }_{ij})$ to $C_i$ inside the logistic, where $f$ is a restricted cubic spline in the gap with five knots (Harrell 2015), shared by all models and scaled so that $f(0)=0$ and $f(1)=1$, and ${\psi }_i$ is one coefficient per model. If models gain on benchmarks they could have seen, most ${\psi }_i$ should be positive, $f$ should rise after zero, and $f$ should be flat before zero, where the benchmark did not exist yet. The negative gaps are a built-in placebo.

The model fits better in sample (the residual sum of squares falls from 15.87 to 12.96, for 273 more parameters), and its effects would be news: a median of 13.6 and a maximum of 148 ECI points. It also predicts held-out scores worse than Epoch's model in every fold (held-out root mean squared error 0.107 against 0.093), and forcing $f$ to zero before the release date makes it worse still.

Section 1.4 explains why. The fitted $f$ is nearly a straight line, from $-5.3$ at a gap of $-3$ years to $2.8$ at 7 years, and it is as steep before the release date as after it: the placebo fails. A straight line in the gap does almost nothing in this model. Take $f(\Delta )=\Delta$ and split ${\psi }_i$ into its mean $\overline{\psi }$ and a deviation, ${\psi }_i(t_i-s_j)={\psi }_i{t}_i-\overline{\psi }{s}_j-({\psi }_i-\overline{\psi })s_j$. The first two pieces are absorbed by $C_i$ and $D_j$. What is left is a slope per model on the benchmark's date: a model that is relatively good at old benchmarks. That is specialization by era, not benchmaxxing, and the data have plenty of it. The models with the most negative coefficients are the early open models (Figure 1.7). Falcon-7B, LLaMA-7B and MPT-7B do better than their capability predicts on PIQA, Winogrande, HellaSwag and OpenBookQA, and worse on GSM8K, MMLU and BBH. Old commonsense benchmarks and old small models share an era, and a coefficient on the gap finds them. With 4 to 30 cells per model and a median spread of the gap within a model of 1.6 years, 270 such slopes are mostly noise, which is what cross-validation says.

Shrinking the coefficients harder should have brought back Epoch's model, and it did not: a larger ridge on $\psi$ made held-out error worse, and the largest effects survived. The reason is that $(\psi ,f)$ and $(c\psi ,f/c)$ give identical fits for any $c\ne 0$, so a ridge on $\psi$ alone is evaded by making $f$ large. The fix is to penalize both factors, which, minimized over rescalings, is a scale-invariant penalty on the size of their product: the variational form of the nuclear norm familiar from matrix factorization (Srebro, Rennie and Jaakkola 2005). With that penalty, one coefficient per lab has a unique optimum and beats Epoch's model in 7 of 10 folds, by about 1%. But its fitted curve looks nothing like benchmaxxing. Scaled so that $f(1)=1$, it is 7.4 at a gap of $-3$ years, $-0.1$ at 2 years and 6.4 at 7 years: large where the benchmark did not yet exist. One coefficient per model has four to six distinct local optima, and capabilities move by up to 74 ECI points between them. A coefficient per model on a covariate that barely varies within a model is, for the most part, a coefficient on something else.

1.10 What survives

Up the ladder, two kinds of result separate. The effects common to all models are the ones this design protects least. A single step at the release date is zero without drift and $+0.95$ with it, moving with whatever the drift term absorbs. The common bend of rung 4 is real, but it sits a year before the release date. And a free curve per model finds the era each model belongs to. None of these is evidence of benchmaxxing.

The comparison between labs is stable. Every rung that can estimate a step per lab finds Google's, and only Google's. Google's distance from the other labs, about 1.6 ECI points, does not move when the drift, the ceilings, the grouping of Google's models or any single benchmark changes. In the language of causal inference this is a difference in discontinuities: a regression discontinuity design (Imbens and Lemieux 2008) with labs as groups. It inherits that design's strength, because smooth time effects common to all labs cancel. It also inherits its weakness: anything specific to one lab that changes smoothly with a benchmark's date does not cancel, and the step model cannot tell such a curve from a jump. That is where Part 2 picks up, and its answer changes how the result should be read. In three questions:

Is there a Google pattern? Yes. Under the resampling tests that assume the least (over models, over benchmarks and over lab labels), it has a $p$-value between 0.01 and 0.02, and it is strongest on the scores Epoch measured itself. But Google was not named before we looked, and once the search over labs is counted the evidence sits at the edge of 5% (0.03 to 0.08, depending on the method).

Is it the jump at release that training on public test sets would leave? No, for three reasons. Among benchmarks released after a Google model, those released within about a year of it score higher, relative to the other labs, than those released later; tuning to those particular benchmarks cannot produce that, although early access to a benchmark before its release, or tuning toward the kind of task in fashion at the time, could (this rests on about ten Google models). A smooth curve in benchmark date, fitted for each lab, explains the data better than the step, and in simulations with a step of Google's size planted at the release date, that curve finds it at full size, so it is not hiding one. And on the same benchmark, Google's models released after it do not gain over Google's models released before it.

Is it benchmaxxing at all? Open. What is left is a profile over benchmark age: Google's models do best, relative to the other labs, on benchmarks released about a year before them, and worst on the newest ones. There are two readings. It could be tuning to the benchmarks of the moment, which is a kind of benchmaxxing. Or Google's models could be relatively weaker on the newest kinds of benchmark, whatever their own release date, which is not. The newest benchmarks do share a type: 14 of the 19 released in 2026 have the model work over many steps with tools. Google's models score lower on these than other labs' models of the same capability, and with the tool-using mathematics benchmarks counted in, by more than any other lab's; that takes most of Google's curve over benchmark age and about a third of the contrast. On the strictly agentic benchmarks, though, OpenAI, Alibaba and DeepSeek fall short by as much, and without them Google's contrast keeps a $z$ of 2.4. The split depends on how a few benchmarks are classified, and we classified them after seeing the profile (Part 2, the end of Section 2.10). The same-benchmark comparison leans toward the second reading, but it has little power, and a lift spread over the benchmarks of the two or three years before a model can lose about half its size to the curve, so the first reading is not ruled out.

Three things would tell the readings apart: coding the protocol behind each score, a model with lab effects by type of benchmark, and the prospective test registered in Part 2, which new Gemini releases will run for us. Until then, the code at github.com/aemrebarut/eci-benchmaxxing lets anyone refit every model here on Epoch's current data.

References