Fifteen objections and our replies

A result is not robust because it survives the checks its authors chose. It is robust if it survives the checks its critics would choose.

2.1 The claim, and how we attack it

Part 1 leaves one number to defend. In the step model (1.3), with all of Google's models merged into one group, Google's models gain more on the benchmarks that were public at their release than the other labs' models do. The contrast (1.5), Google's step minus the precision-weighted mean of the other 18 identified groups' steps, is:

The claim is narrow. It does not say that Google trained on test sets, nor that other labs did not. Section 1.6 showed that the design protects the common part of the steps poorly. It says only that Google's step stands out from the others, and that the gap is large enough to matter.

We imagined sending Part 1 to a statistics journal and wrote down what constructive referees would say. The objections fall into four families:

• the inference: is $z\approx 3$ honest?
• the choice of test: did we look at 19 labs and report the one that stood out?
• the design: is the step a step, or something else that changes with time?
• the data: are the scores what we think they are?

For each objection we state it as a referee would, then report the analysis that answers it and a verdict. A result holds if the contrast stays near three standard errors and is weaker if it survives with a smaller $z$. An objection reframes the result if the pattern survives but the analysis shows that the step model describes it wrongly. Two objections do this. Neither is evidence that Google does not benchmaxx: they change what the pattern is, not whether it is there. Table 2.1 summarizes the verdicts and the sections give the details.

The short answer. This post ends in a different place from Part 1, and we say so first. The step model asks three questions at once, and the objections answer them differently.

• Is there a Google pattern? Yes. No change of variance estimator, test, weighting or data removes it. Honest inference puts it at $p\approx 0.01$ to 0.02 rather than the 0.002 that the standard errors of Part 1 imply: three resampling tests, over models, over benchmarks and over labels, give $p$ between 0.008 and 0.021. Only the deliberate worst case, deleting together the five benchmarks that matter most, leaves a $z$ of about one. Google was not named before we looked, and once the search over labs is counted the evidence is at the edge of 5% (Section 2.7).
• Is it a jump at the release date? No. Placebo cutoffs placed where no leak is possible find the same contrast (Section 2.9). A smooth curve in benchmark date fits better than the step, and comparing Google's models released before and after the same benchmark finds nothing (Section 2.10). In simulated data the curve finds a planted step at full size, so it is not hiding one. What the step summarized is a profile: Google's models do best, relative to the other labs, on benchmarks released about a year before them, and worst on the newest benchmarks. These two objections are about the shape of the effect. They tell against the simplest story, test sets that leak into training and pay off from the day a model ships, but not against benchmaxxing in general.
• Is it benchmaxxing? Open. Two readings fit the profile: tuning to the benchmarks of the moment, which is a kind of benchmaxxing, and a relative weakness on the newest kinds of benchmark, which is not. Part of the profile is a weakness on agentic and tool-using benchmarks, which are most of the newest ones; how large a part depends on how a few benchmarks are classified (end of Section 2.10). New Gemini releases can separate the readings (Section 2.7).

Unless stated otherwise, all fits use the 2026-09-29 snapshot of Part 1 and the near-unpenalized ridge on the steps ($\lambda ={10}^{-3}$). The code is in analysis/robust.py and analysis/23_appendix.py, and each study writes out/app23_<study>.csv. Every resampling study uses fixed seeds.

2.2 Objection 1: the ridge flatters the Wald test

Referee. The standard errors come from a penalized fit. A ridge shrinks the parameters and their estimated variance, and Wald statistics computed from penalized fits are known to be optimistic. How much of $z=3$ is the penalty?

The fit has three ridges, and each needs its own answer:

• Steps. $\frac{1}{2}\lambda {\sum }_g{\beta }_g^2$ with $\lambda ={10}^{-3}$. This is negligible against a residual sum of squares of 15.7.
• Drift. With drift, $\frac{1}{2}\mu {\sum }_j{\kappa }_j^2$ with $\mu =0.1$. This is not negligible, because it is what fixes the common part of the drift (Section 1.6).
• Epoch's own. Epoch's code adds $(0.1/389)\sum x^2$ over all 389 of its parameters ($C_i$, $D_j$ and the free slopes $a_j$). This fixes the location of the scale and, it turns out, part of its unit.

In principle the referee is right. The model-based covariance of a penalized least-squares fit is $s^2(H+P{)}^{-1}$, where $H$ is the Gauss-Newton Hessian and $P$ the penalty. This treats the penalty as prior information. As a posterior covariance under the matching Gaussian prior it is coherent; as the sampling variance of the estimator it need not be.

The direct check is to remove every ridge and refit. Without penalties the fit has an exact symmetry, $C_i+t$ and $D_j+t$ for any $t$, which we remove with the gauge row ${\sum }_j{D}_j=0$. With drift, the model also absorbs a drift common to all benchmarks into $C$ and $D$, so we add ${\sum }_j{\kappa }_j=0$. Neither row changes the fitted values.

Removing the ridges lowers every Wald $z$. For the contrast, $z$ falls by 0.22 without drift and by 0.27 with drift. The referee has the direction right, and the size is about a quarter of a standard error. Section 2.3 shows that this loss belongs to the Wald approximation rather than to the evidence.

The column for $b$ holds a second and larger lesson. The ECI unit is $b=20/(C_{\mathrm{h}\mathrm{i}}-C_{\mathrm{l}\mathrm{o}})$, so the two anchor models sit twenty points apart. Epoch's ridge costs only 0.025 in residual sum of squares, yet removing it moves $b$ from 19.3 to 25.9. With drift, removing the ridge on $\kappa$ roughly halves $b$, from 23.9 to 12.6, because the convention for the common drift changes which capabilities the anchors are compared at.

A result "in ECI points" is therefore partly a result in a unit that the penalties choose. Ratios such as $z$ do not depend on the unit. We treat $z$ as the primary quantity and ECI points as a translation.

Verdict: holds. Without any ridge the contrast has $z$ 2.82 and 2.91, and Section 2.3 recovers the loss.

Moral. Removing a penalty is the cleanest way to see what it was doing. Here it was worth a quarter of a standard error in the Wald test, and it also set the unit of the answer.

2.3 Objection 2: Wald is the wrong test for a nonlinear model

Referee. The model is nonlinear in its parameters, and the contrast is a combination of steps multiplied by the unit $b$, which itself depends on two capabilities. Wald tests are not invariant to such reparameterizations, and their intervals can be poor in nonlinear least squares. Use likelihood ratio tests and profile intervals.

For a linear function $c^{\prime }\beta$ on the ECI scale and a value $v$, we refitted the model under the constraint $b(C)c^{\prime }\beta =v$, imposed as a stiff penalty row so that every other parameter re-optimizes. The test statistic is

where edf is the effective number of parameters of the unconstrained fit. Under the null, $F(v)$ is approximately $F_{1,n-\mathrm{e}\mathrm{d}\mathrm{f}}$, and the signed root $r(v)=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}(\widehat{v}-v)\sqrt{F(v)}$ is approximately standard normal. The test of no contrast is $r(0)$. The profile interval is the set of $v$ with $|r(v)|<1.96$ (Bates and Watts 1988, chapter 6; Venzon and Moolgavkar 1988).

The LR test is stronger than the Wald test in every row, and every profile interval lies further from zero than its Wald interval (Figure 2.1). The profile is steeper on the side of zero because pushing the contrast toward zero forces the other parameters to compensate. That compensation costs more fit than the quadratic approximation assumes. For Google's step alone the signed roots are 3.20 without drift and 4.36 with drift, against Wald values of 2.91 and 3.88. Without the ridge, the LR roots (3.37 and 3.35) are close to those with it: the penalty barely changes the likelihood surface near the null.

Both tests assume that the residuals are independent with a common variance. Objections 3 to 5 are about that assumption.

Verdict: holds. The likelihood ratio test gives signed roots of 3.34 to 3.42.

Moral. Here the likelihood ratio test is the more appropriate test, and it strengthens the claim. Without the ridge, the Wald test loses a quarter of a standard error and the LR test recovers more than that.

2.4 Objection 3: the cells of one model are not independent

Referee. A model's scores on different benchmarks share its harness, its prompt format, its reasoning setting and its luck on the day. The model-based standard errors treat 2,843 cells as independent. The effective sample for Google's step is closer to the number of Google models with cells on both sides of their release, which is 28. Use inference that respects the clustering and stays valid with few clusters.

Part 1 computed cluster-robust standard errors by model as well, and reported the larger of the two. The clustered ones are smaller than the model-based ones: 0.44 against 0.51 for the contrast. The referee should not be satisfied by that. With few effective clusters, sandwich estimators are biased downward, and tests based on them over-reject, sometimes badly (Cameron, Gelbach and Miller 2008; MacKinnon and Webb 2017). We ran two procedures that do not rely on the sandwich.

Delete-one-model jackknife. We refitted the model 268 times, each time without one model; the two anchor models always stay. From the refits we computed the CV3 variance

which is conservative in small samples (MacKinnon, Nielsen and Webb 2023).

Wild cluster bootstrap with the null imposed. We fitted the model under $H_0$ (contrast 0, or Google step 0). We then built bootstrap data by multiplying each model's restricted residuals by one Rademacher sign, $y_{ij}^{*}={\widehat{y}}_{ij}^0+v_i{\widehat{\epsilon }}_{ij}^0$ with $v_i=\pm 1$, refitted the unrestricted model, and recorded two $t$ statistics, the model-based one and the one clustered by model. The $p$-value compares the observed clustered $t$ with its bootstrap distribution. This is the restricted wild cluster bootstrap-$t$ of Cameron, Gelbach and Miller (2008), which had the most accurate size in their simulations and in those of MacKinnon and Webb (2017). It also gives the true size of each nominal 5% Wald test: the share of bootstrap $|t|$ beyond 1.96.

The two procedures disagree in an instructive way. The jackknife puts the standard error at 0.49 and 0.51, close to the model-based value, so $z$ stays near 3.2. The wild bootstrap is more demanding, with 999 bootstrap data sets for the contrast and 499 for the step. In data generated under the null, both Wald tests over-reject: the clustered one has a true size of 12 to 13% at a nominal 5%. The observed contrast has $p=0.008$ and 0.009, a normal $z$ of about 2.6. The clustered $z$ of 3.5 and 3.8 in Section 2.1 overstate the evidence. With 28 Google models identifying the step, this is the few-clusters problem the referee describes (Figure 2.2, a).

Verdict: weaker. With models as clusters, the contrast has $p\approx 0.01$, not the 0.002 of the Wald test or the 0.0004 of the clustered sandwich.

2.5 Objection 4: benchmarks are clusters too

Referee. Cells on one benchmark share its scorer, its version changes and its saturation, so errors may be correlated within benchmarks as well as within models. A test that clusters on one dimension can still over-reject.

The two-way cluster variance of Cameron, Gelbach and Miller (2011) is $V_{\text{model}}+V_{\text{benchmark}}-V_{\text{cell}}$. We estimated each term without the sandwich where we could:

• $V_{\text{model}}$ and $V_{\text{benchmark}}$ from the delete-one-model and delete-one-benchmark jackknives (2.2);
• $V_{\text{cell}}$ from the heteroskedasticity-robust sandwich.

The result is the reverse of what clustering by model showed. The sandwich clustered by model is smaller than the model-based standard error; the sandwich clustered by benchmark is larger, and the delete-one-benchmark jackknife is larger still.

The sandwich and the jackknife disagree, and with 60 clusters of very different sizes (6 to 182 cells) neither can be taken at face value. The sandwich is biased downward with few or unbalanced clusters, and the CV3 jackknife is conservative (MacKinnon, Nielsen and Webb 2023). The wild bootstrap with weights by benchmark and the null imposed arbitrates.

Wild bootstrap by benchmark. We repeated the restricted wild cluster bootstrap of Section 2.4 with one Rademacher sign per benchmark instead of one per model: 999 bootstrap data sets for the contrast and 499 for Google's step, in each structure. The $p$-value uses the benchmark-clustered $t$ (Figure 2.2, b).

The bootstrap lands between the sandwich and the jackknife. A $p$ of 0.02 corresponds to a normal $z$ of about 2.3. The benchmark-clustered sandwich is close to honest, with a true size of 5 to 8%, and the delete-one-benchmark jackknife is conservative, as expected. The model-based Wald test of Part 1 is not honest under this kind of dependence: it rejects a true null 17 to 21% of the time at a nominal 5%, and its 5% critical value is 2.6 to 3.0 rather than 1.96.

The source of the dependence is the lab-by-benchmark structure that Section 2.10 finds. The residuals of one lab's models on one benchmark move together, and Google's 28 models share their benchmarks.

The objection raises a question about what the claim is. Part 1 asks whether Google's scores on the 60 benchmarks in Epoch's index gain more at release than other labs' scores. For that question the benchmarks are fixed, and inference that treats models as the sampled units answers it. A broader claim, that Google gains more on public benchmarks in general, treats Epoch's 60 as a sample from a population of benchmarks. Its uncertainty includes how the Google effect varies across benchmarks, and the benchmark-clustered inference is the relevant one. The first question is about the index, the second about the lab.

Verdict: weaker for the claim about the lab, where the contrast has $p\approx 0.02$ rather than 0.002. The claim about Epoch's 60 benchmarks is unchanged.

2.6 Objection 5: one model or one benchmark carries it

Referee. Twenty-eight Google models identify the step. If one Gemini release with an unusual score profile drives it, the claim is about that model, not about a lab.

The jackknife refits of Sections 2.4 and 2.5 double as an influence analysis.

• Models. Without any single model, the contrast stays between $+1.26$ (without Gemini 3.1 Pro) and $+1.71$ (without Gemma 4 31B IT), with $z$ from 2.41 to 3.24, without drift. With drift it stays between $+1.37$ and $+1.78$, $z$ 2.52 to 3.42. The most influential models are Gemini models, as they should be, since they identify the step. Of the four models whose removal lowers the contrast most, three are Gemini models (3.1 Pro, 3.5 Flash and 3.7 Flash) and one is Claude 3.5 Sonnet (October 2024).
• Benchmarks. Without any single benchmark, the contrast stays between $+1.24$ (without DeepSWE) and $+1.97$ (without OTIS Mock AIME 2024-2025), $z$ 2.52 to 3.48. With drift it stays between $+1.38$ (without Chess Puzzles) and $+2.17$ (without ProofBench), $z$ 2.70 to 3.90.

Single deletions can hide joint influence. We therefore also dropped, together, the three and the five models, and the three and the five benchmarks, whose single deletions lowered the contrast most. This is a worst case by construction.

Verdict: weaker. No single model or benchmark carries the contrast, but five benchmarks together carry between half and two thirds of it. All six named benchmarks were released between October 2024 and May 2026, against a median of March 2025 for the 60 benchmarks in the data. Section 2.10 returns to this.

2.7 Objection 6: you looked at 19 labs and reported the one that stood out

Referee. Google was not named before the data were seen. With 19 groups, one $z$ near 3 is less surprising than it looks, and the paper's own omnibus test of equal steps does not reject.

The referee is right that the hypothesis is post hoc, so the test must account for the search. We give three answers.

The maximum over labs. If we had tested every lab against the others and reported the largest, the reference distribution is that of ${max}_g|z_g|$ under the null of no differences. We simulated it from the joint normal distribution of the 19 contrasts, with the correlations of the model-based covariance. The probability that some lab's $|z|$ reaches Google's is 0.043 without drift and 0.028 with it. A Bonferroni correction gives 0.046 and 0.028. Google survives the search, narrowly.

Permutation. Section 2.8 describes a permutation test that relabels models among labs. In each relabelled data set we recorded the largest $|z|$ among the 12 labs with at least five models. The share of relabelled data sets in which some lab reaches Google's $|z|$ is 0.062 without drift and 0.081 with drift. This is the one multiplicity-adjusted test in this post that does not reach 5%. It charges two penalties at once: the search over labs, and the spread of $z$, which is wider than normal (Section 2.8).

The omnibus test. The test that all 19 steps are equal has $p=0.48$ because it spreads its power over 18 directions, only one of which is Google. A test aimed at the most extreme group, such as the maximum above, has far more power against a single deviant group. The two results do not contradict each other.

A prospective test. The real remedy for a post hoc hypothesis is a test on data that did not exist when it was formed. We state one now:

• Specification. Freeze the model of Part 1: Google merged, drift, near-unpenalized steps. Add a separate step for the Google models released after 2026-10-01.
• Prediction. That step, minus the precision-weighted mean of the other labs' steps, is positive.
• Test. One-sided at 5%.

If information grows with the number of models, five new Google models give the test power of about 0.36 against a contrast of 1.55, ten give 0.57, and twenty give 0.82. The October update contains none. Section 2.10 shows that the same test also separates the two readings of the pattern that remain.

Verdict: weaker. Adjusted for the search over labs, the evidence is at the edge of 5%.

2.8 Objection 7: the labels follow the calendar

Referee. Google's models are not spread over time like everyone else's. Anything that changed with time, such as benchmark saturation, harness upgrades or Epoch's own evaluation practice, would load on whichever group is concentrated in the relevant period.

The test answers the referee directly. We permuted the organization labels among models released in the same half-year, keeping each half-year's mix of labels. We then refitted the model and recomputed Google's contrast, 999 times for each structure (the seeds are fixed). If the contrast reflects when Google released its models rather than who made them, relabelled data should reproduce it as often as not.

The relabelled $z$ has mean $+0.37$ without drift and $+0.46$ with drift, and standard deviation 1.19 and 1.29 (Figure 2.2, c). Two things follow. Timing pushes the contrast of any group concentrated in Google's half-years up a little, by about 0.4 standard errors. And the relabelled $z$ is more variable than a standard normal, consistent with the benchmark clustering of Section 2.5. Google's observed $z$ is beyond 98.9% of the relabelled values in absolute value without drift and beyond 98.4% with drift: $p=0.011$ and 0.016.

Verdict: weaker. The label matters beyond the timing, but the test that respects the timing gives $p\approx 0.01$ to 0.02, not 0.002.

2.9 Objection 8: why the release date?

Referee. Contamination cannot start on the release day. A model's training data end months before it is released, and a benchmark released a week earlier could not have leaked. A step exactly at $\Delta =0$ is the wrong shape for a leak, and an analysis that fits only that cutoff cannot show that anything happens there. Move the cutoff and see what happens.

We moved the cutoff in four ways.

Shifted cutoffs. We replaced ${\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{t}}_{ij}=\mathbf{1}({\Delta }_{ij}>0)$ with $\mathbf{1}({\Delta }_{ij}>h)$ for $h$ from $-1.5$ to $+1.5$ years in quarter-year steps, and refitted (Figure 2.3). For $h>0$ the post cells are benchmarks public at least $h$ years before the model was released. For $h<0$ the post cells also include benchmarks released up to $|h|$ years after the model.

On the full data, the pattern looks like what a leak through training data would predict:

• Just before release. For $h$ between $+0.25$ and $+0.75$ the contrast is larger than at the release date, with $z$ from 3.6 to 4.5. Moving benchmarks released in the last few months before a model into the pre group sharpens the contrast, as it should if those months were after the training cutoff.
• More than a year before. For $h\ge 1.25$ the contrast is gone ($z$ below 1). The steps are not a general advantage on old benchmarks: they separate the benchmarks of the last year or so before release from those that came later.
• After release. For $h<0$ the estimates stay positive, but the standard errors grow quickly and $z$ stays below 2.5. Under a step at the true cutoff, a cutoff placed too early mixes clean cells into the post group, which still leaves a positive but diluted estimate.

The last point means that shifted cutoffs on the full data are not a clean placebo test. Imbens and Lemieux (2008, section 7.1) recommend placebo cutoffs that use only the data on one side of the real cutoff, where no effect should exist. That test contradicts the story above.

One-sided cutoffs. We kept only the pre cells, where the benchmark was released after the model (845 cells), and placed a fake cutoff 0.25 to 1 year after the release. We then kept only the post cells (1,998 cells) and placed fake cutoffs 0.25 to 1.5 years before the release. A leak at release predicts nothing on the pre side, because none of those benchmarks existed when the models were trained.

The pre side fails the placebo. Consider only benchmarks released after a Google model. Those released within about half a year to a year of the model score higher, relative to the other labs, than those released later, with $z$ up to 3.3. A leak at release cannot produce this. The post side shows a gradient as well, up to 0.75 years before release, which is why the full-data contrast peaked at cutoffs of $+0.25$ to $+0.75$ years. (The table reports $z$ only, because each one-sided subset refits the unit $b$, here 10 to 23.)

Donuts. Dropping the cells within $d$ of the release date, which are the most ambiguous about leakage, changes little for $d$ up to three months: the contrast is $+1.94$ to $+2.30$ ($z$ 2.8 to 3.7) without drift and $+2.26$ to $+2.49$ ($z$ 3.0 to 3.9) with it. At $d=6$ months the standard error doubles and $z$ falls to 1.7 and 2.3.

Bandwidths. Keeping only cells within $w$ years of the release date gives $z$ near 3 for every $w\ge 1.5$ (2.9 to 3.4) and about 1.5 to 1.8 for $w=1$, where few models still have cells on both sides.

Verdict: reframes the result. The Google contrast does not begin at the release date, so a step there is the wrong description of it. That is a finding about the step model, not a finding that Google does not benchmaxx. The contrast is still there, but it is not switched on by a benchmark's release: the one-sided cutoffs find it among benchmarks released after the model, which that model could have been tuned on only through early access. Tuning toward the kind of task in fashion at the time could still produce it. Section 2.10 asks what shape the pattern has.

Moral. A placebo is only a placebo if no effect is expected there. Shifted cutoffs on the full data told a plausible story about training cutoffs. Cutoffs placed where no leak is possible showed that the story was not about leaks.

2.10 Objection 9: a smooth trend can look like a step

Referee. The step is defined by the sign of $t_i-s_j$. For a given model, the post cells are its older benchmarks. Suppose Google's models do relatively better on older benchmarks for a reason unrelated to leakage. Their training mix might favour knowledge and multiple-choice formats, or Google might have been late to the agentic tasks that dominate the newest benchmarks. The step would then find a Google effect. The drift term of (1.4) gives each benchmark a trend in model date, but it does not give each lab a trend in benchmark date. Add one, and show that the step survives.

This is the central objection to any regression discontinuity. A jump is credible only when smooth functions of the relevant dates are controlled (Imbens and Lemieux 2008), and the control should be low-order (Gelman and Imbens 2019). The running variable of the step is ${\Delta }_{ij}=t_i-s_j$, but the confounder in the referee's story is the benchmark's vintage $s_j$. Within one model, $\Delta$ and $s$ differ by a constant. So the step can be told apart from a lab's smooth curve in $s$ only through models of the same lab released at different times, each with its cutoff at a different benchmark vintage.

We added a lab-specific function of benchmark date for the seven labs with the most models (Google, OpenAI, Anthropic, Alibaba, Meta, DeepSeek and xAI):

We tried four forms of $f$, in years from 2024:

• linear;
• quadratic;
• linear with separate slopes before and after each model's release;
• the strongest version, a linear trend in benchmark date for every model with at least three distinct benchmark dates (with a small ridge ${\lambda }_x$ on the slopes).

A trend common to all labs is absorbed by $D_j$, so each $f_g$ is relative to the labs without one.

The contrast survives a linear trend per lab at $z\approx 2.5$. It does not survive a quadratic. With separate slopes on each side of the release, the standard error grows fivefold and the design has no power left. With a slope for every model, the estimate keeps about 60% of its size, but $z$ falls to 1.3 to 1.4.

The quadratic says that Google's relative performance falls with benchmark vintage, faster for the newest benchmarks: its linear and quadratic terms are $-0.53$ and $-0.13$ ECI points per year and per year squared. For most Google models the newest benchmarks are exactly the pre cells, so a curve that bends down at the new end can do the step's work. Three further analyses ask which of the two the data prefer.

Step or curve. With each lab's linear trend in the model, we compared four fits for Google: no extra term, a step at release, a quadratic in benchmark date, and both. The step and the quadratic each add one parameter.

With the same number of parameters, the quadratic fits better than the step. Once the quadratic is in, the step adds almost nothing: $F=0.09$ without drift and 2.1 with it, for steps of $+0.34\pm 0.64$ and $+1.09\pm 0.69$ ECI points. The quadratic given the step has $F=8.9$ and 5.9.

Same benchmark, different release dates. The most demanding design removes every lab-specific benchmark profile. It adds an effect for each (lab, benchmark) pair among the seven largest labs: 272 columns, with the small ridge ${\lambda }_x$. The steps are then identified only by comparing a lab's models released before a benchmark with the same lab's models released after it, on that same benchmark. This is a triple difference over lab, benchmark and timing. Google has models on both sides of the release of 26 benchmarks.

The contrast is zero. The design has less power, since the standard error nearly doubles. Without drift it cannot exclude a contrast of 1.55, which is 1.9 standard errors above the estimate; with drift, the headline value of 1.61 lies 3.0 standard errors above it. The lab-by-benchmark terms are not noise: they lower the residual sum of squares from 15.71 to 10.92 for 263 effective parameters, an $F$ of 3.6 on 263 and 2,176 degrees of freedom.

The picture. Figure 2.4 plots the residuals of Epoch's model, without any steps, against the gap. Google's residuals rise by about 1.5 ECI points across zero, which is within the noise of a quarter-year bin, and keep rising: from about $-4$ for benchmarks released more than half a year after the model to about $+2.7$ for benchmarks released 0.9 years before it. They then fall back below zero for benchmarks more than 1.25 years old. The other labs' residuals are close to zero after the release date and slightly negative before it.

Would the curve hide a real step? A design flexible enough to absorb an effect will report none, so the null of the quadratic is informative only if the quadratic can find a step that is there. We checked by simulation (analysis/26_inject.py). The step world is the Part 1 fit, with its steps. The curve world is the fit with lab quadratics and Google's step held at zero. Each simulated data set is the world's fitted values plus the residuals of the fit, with one random sign per model, so each model keeps its own pattern of residuals (a wild bootstrap). We refitted each data set with the step alone, with the quadratic by lab, and with the lab-by-benchmark effects: 400 simulations per cell, 200 for the lab-by-benchmark design.

The quadratic does not hide a real step. In the step world it recovers the step at nearly full size, with a standard deviation about 60% larger than the step design's, and so with power of about two thirds rather than 0.96. The real quadratic estimates, $+0.04$ and $+0.22$, would be unusual if the step were real: only 1.8% and 4.8% of the step world's simulations come out as low, and 3.5% and 1.0% of the lab-by-benchmark ones. The reverse holds as well. A world of curves alone, with no step at all, gives the step design a contrast of $+1.50$ and $+1.39$, about the size we observed, and a significant one in 91% and 84% of the simulations. The data look like the curve world, not the step world.

Would the curve hide tuning to the benchmarks of the moment? The first reading in the verdict below is a form of benchmaxxing that a step does not describe: a lift on the benchmarks released in the year or two before a model, and none on older or newer ones. We planted that as well (analysis/26b_inject_hump.py). In the hump worlds, Google's step is replaced by a lift that rises from zero at the model's release to a peak on benchmarks 0.9 years older than the model, where Figure 2.4 peaks, and falls back to zero on benchmarks 1.8 years older (the narrow hump) or 2.7 years older (the broad hump). Each hump is scaled so that its average over Google's post cells equals the step, which puts the peak at 4.6 and 4.0 ECI points (6.6 and 5.7 with drift). There are 200 simulations per cell, 100 for the lab-by-benchmark design.

Here the curve does absorb part of the effect. Against the step design's contrast in the same world, the quadratic keeps about 98% of a step, 71% and 53% of a narrow hump, and 52% and 30% of a broad one (Epoch's structure and drift). In the real data it keeps 3% and 14%. A narrow hump leaves a quadratic estimate as low as the real one in 3.5% and 10% of the simulations, a broad hump in 8% and 29%. The lab-by-benchmark design tells the same story, at 7% to 19%. So the curve is a fair test of a jump at release and of a lift confined to benchmarks less than about two years old. It cannot exclude a lift that reaches benchmarks up to three years old: a curve over benchmark date can absorb about half of such a lift, or more. A plausible reason is that 87% of Google's cells come from models released in the two years from September 2024, so a broad lift measured from each model's release looks much like a lift on the benchmarks of one period.

Verdict: reframes the result. The other checks of this post leave a Google pattern in place, weaker under benchmark clustering. But the pattern is a profile over benchmark age, not a step at the release date, and once each lab may have its own curve the step is not needed. Like Objection 8, this corrects the model rather than clearing Google. Two readings of the profile remain, one of them a kind of benchmaxxing, and the data in hand cannot fully separate them.

• Tuning to the benchmarks of the moment. A lab that optimizes for the benchmarks that are current while it trains, roughly those released in the year before release, would produce a hump like Figure 2.4. Older benchmarks are saturated or out of fashion, and newer ones do not exist yet. This is a form of benchmaxxing, though not the leak at release that the step encodes. The triple difference leans against it, within its limited power: Google's models released after a benchmark do not gain on that benchmark relative to Google's models released before it, which tuning to that benchmark would predict. The injection test leans against a narrow version of it, a lift on benchmarks less than two years old, but not against a broad one.
• A profile over benchmark vintage. Google's models are relatively weaker on the benchmarks introduced in 2025 and 2026, many of them agentic or long-horizon, whatever the model's own release date. For most Google models these are the pre cells, so the step model reads a capability profile as a release-date effect. The triple difference is what this reading predicts: once each lab has its own effect on each benchmark, nothing is left for the step. A benchmark-type-by-lab model tests the reading more directly; we fitted one after seeing the profile (below).

Agentic benchmarks. We classified the 60 benchmarks by how Epoch runs them, from Epoch's benchmark pages and the benchmarks' own papers (data/benchmark_types.csv). Nineteen are agentic: the model acts over many steps with tools, as in SWE-bench Verified, OSWorld, Terminal Bench, METR Time Horizons and DeepSWE. Thirty-two are not. Nine are borderline, mostly hard mathematics with a Python sandbox or a proof checker (the four FrontierMath sets, ProofBench, Surface Evolver Bench), plus WeirdML, GDPval and Furniture Assembly. They are also the newest benchmarks: 14 of the 19 released in 2026 are agentic or borderline. Whether a benchmark is agentic is a property of the benchmark, so a term common to all labs is absorbed by $D_j$. A term for Google's models on agentic benchmarks measures them against the other labs' models of the same capability. We added one to the step model, with the borderline benchmarks counted out (strict) and in (broad), and refitted the step model without the agentic benchmarks (analysis/31_agentic.py).

Google's models on agentic benchmarks, in ECI points relative to the other labs' models of the same capability, with the larger of the model-based and clustered standard errors, and the contrast (1.5) with its model-based $z$. Leaving out the agentic benchmarks leaves 2,439 cells (strict) and 2,005 (broad) of 2,843.

Google's models are weaker on agentic benchmarks in both codings, but the codings disagree on what that explains. In the strict coding the weakness is not Google's alone. With a term for each of the seven largest labs, OpenAI ($-3.4$), Alibaba ($-4.1$) and DeepSeek ($-2.8$) are about as weak as Google ($-3.1$), and the contrast does not move. In the broad coding Google's term is the most negative of the seven ($-4.1$, against $-0.6$ for OpenAI and $+0.4$ for Anthropic). It removes about a third of the contrast, and added to the curve above it takes most of Google's slope and curvature in benchmark date (linear term $-0.69$ to $-0.16$, quadratic $-0.21$ to $-0.08$, Epoch's structure). The difference between the codings comes mostly from the two FrontierMath sets of 2026: removing either from the broad coding brings the contrast back to $z=2.7$. Within the strict coding, Google's shortfall sits on the agentic benchmarks released after its models ($-6.2\pm 2.0$, on 14 cells, mostly Gemini 3.1 Pro and the 3.5 to 3.8 Flash models), against $-0.8$ on agentic benchmarks that predate them. That is the shape of the step itself, so on those cells the two readings predict the same thing.

So the vintage profile is in part a weakness on agentic and tool-using work, by an amount that depends on how a few benchmarks are classified, and the classification was made after the profile was seen. It does not account for all of the pattern. Without the strictly agentic benchmarks the contrast keeps a $z$ of 2.4 and 2.6, and only the broad coding, which leaves out 30% of the cells, takes it to 1.2 and 1.4.

The prospective test of Section 2.7 still discriminates. Each new Gemini model moves the release date to a new benchmark vintage. A vintage profile predicts that new Gemini models do relatively badly on the benchmarks of 2025 and 2026 whether or not those benchmarks predate them; a release-date effect predicts that they gain on them.

Moral. When the treatment is a date, every smooth function of time is a rival explanation. Here the smooth function fitted better than the jump. Google's step looks like a summary of a profile over benchmark age, and the step model had no way to tell a profile from a jump.

2.11 Objection 10: the scores are proportions

Referee. The responses are rescaled proportions in $[0,1]$, so their variance shrinks near the bounds. Least squares weights a cell at 0.98 as heavily as a cell at 0.5, and the model-based standard errors assume a common variance.

We refitted by iteratively reweighted least squares with quasi-binomial weights $1/(\varphi m(1-m))$, where $m$ is the fitted mean truncated to $[0.01,0.99]$. We used two forms of the dispersion $\varphi$: one common $\varphi$, and a separate ${\varphi }_j$ per benchmark, shrunk toward the common value with a weight of five cells, because some benchmarks have only six cells. We stopped after 12 iterations; by then Google's step moved by less than 0.05 ECI points per iteration.

Verdict: holds. The variance model moves the contrast by up to 0.3 ECI points in either direction, and $z$ stays at 3.0 or above.

2.12 Objection 11: a cell is a maximum, and developers report their best

Referee. Each cell keeps the best of its raw results. Developers report their own scores at launch, on the benchmarks that are public at launch, and they choose their settings. A post cell is thus more likely to be a developer's best-of-many than a pre cell, which can only come from later, independent runs. A lab that reports more of its own numbers will show a larger step without having trained on anything.

This objection predicts something specific, and we can test it in two ways.

Selection term. We added a covariate for the expected maximum of $n$ standard normals, where $n$ is the number of raw rows behind the cell, and in a second fit $\log n$. Its coefficient is positive: $+0.82$ ECI points per unit of expected maximum without drift and $+1.24$ with drift, so cells with more rows do score higher. But the contrast is $+1.61$ ($z$ 3.10) and $+1.74$ ($z$ 3.31): it rises slightly once the selection is controlled.

Sources. We classified each raw row by its source: Epoch's own evaluation, an independent third party (leaderboards, papers and evaluation organizations), or the developer. We then rebuilt the cells from subsets of the rows. Developer-reported scores are behind 26.3% of other labs' post cells but only 8.9% of Google's, so this objection would predict a smaller Google step, not a larger one.

The table gives $z$ only. Each subset refits the scale, and the ECI unit $b$ changes a great deal (Section 2.2), so ECI points from different subsets cannot be compared. On Epoch's own runs alone, where one harness scored every model and nobody chose which result to report, $z$ is 4.8 with half the Google models. This is the strongest single check in this post.

Verdict: holds.

2.13 Objection 12: some benchmarks were built to defeat particular models

Referee. Several benchmarks were built adversarially. Their authors kept only the questions that the frontier models of the day got wrong. Such a benchmark is artificially hard for the models used to filter it, so their scores understate their capability. A filter model is always older than the benchmark, so its cell is a pre cell, and an artificially low pre cell raises its lab's step. Re-fit without the filtered benchmarks, or flag the cells.

We went through the published construction of every benchmark in the data and sorted them into three groups.

• Filtered against named models (six benchmarks). Humanity's Last Exam kept questions that GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, o1, o1-mini and o1-preview failed (arXiv 2501.14249, appendix B.1). SimpleQA was written adversarially against GPT-4 models (arXiv 2411.04368), and SimpleQA Verified, the version in Epoch's data, also used GPT-4o, Gemini 2.0 Flash and Claude 3.7 Sonnet to select its questions (arXiv 2509.07968). Aider polyglot kept the 225 problems that at most three of seven models solved (aider.chat, 2024-12-21). APEX-Agents was filtered with GPT-5, Claude Opus 4.5 and Gemini 3 Pro (arXiv 2601.14242). FrontierMath Tier 4 paid mathematicians for problems that o4-mini could not solve, as reported by Scientific American; the data hold two versions of the tier.
• Filtered against unnamed models (six). LMCA, FrontierCode, FrontierSWE, DTBench and MirrorCode selected their hardest items using model scores or judge disagreement, without saying which models; BBH chose tasks on which models of 2022 fell short of humans.
• Filtered against models that are not in the data (nine). LAMBADA, ARC (AI2), OpenBookQA, HellaSwag, Winogrande, ANLI, PIQA, ARC-AGI-2 and Chess Puzzles filtered against n-gram and recurrent language models, retrieval solvers, BERT, RoBERTa, Kaggle programs or Stockfish.

Only the first group can bias a lab's step, and only through cells of the filter models themselves. The data contain 13 such cells: nine from OpenAI models, three from Anthropic and one from Google (Gemini 1.5 Pro on Humanity's Last Exam). All 13 are pre cells. If filtering lowered them, it raised the steps of OpenAI and Anthropic more than Google's, and so pushed the Google contrast down, not up.

We checked anyway. Each row removes or flags a different set of cells.

Removing the filter-model cells moves $z$ by 0.04. Removing whole benchmarks moves it more, by up to 0.6, but that is the cost of losing up to a quarter of the cells, not evidence of filtering: the last row drops benchmarks that no model in the data could have been filtered against. It also drops the anchor benchmark Winogrande, and the unit $b$ falls from 19.3 to 8.5, so only $z$ is comparable.

The flag finds no deficit. One common term for the 13 cells is $+2.44\pm 1.23$ ECI points without drift and $+1.74\pm 1.40$ with drift, so the filter models score, if anything, above what the model predicts. By lab, the term is $+6.8\pm 1.8$ for Anthropic and $+0.1\pm 1.6$ for OpenAI; Google's single cell sits at the floor of Humanity's Last Exam, where the term is not identified. The Humanity's Last Exam cells of the filter models are all near zero, as the construction intends, but the logistic curve is flat there, so least squares on the proportion scale gives them little weight. The flags are identified mostly by the cells of SimpleQA Verified and of Aider polyglot, whose filter was mild.

Verdict: holds. Adversarial filtering is a real feature of these benchmarks, but in this data set it touches 13 cells, 12 of them from other labs, and it would bias the Google contrast down.

2.14 Objection 13: three standard errors of what?

Referee. Statistical significance is not importance. How large is the effect in units a reader cares about?

The Google step adds 3.7 points on average (median 3.1, maximum 11.6) to the predicted score on Google's 258 post cells, measured on the rescaled score where 0 is chance and 100 is the ceiling. This uses the fit without drift, in which the step and the contrast are both about 1.55. On the ECI scale the contrast is about 1.5 points. The median half-width of the 95% intervals that Epoch publishes for individual models' ECI is 3.9 points.

The effect is therefore smaller than the stated uncertainty of any single model's index, so it will not move a model past a rival that is clearly ahead. It is systematic, though: it applies to every Google model on every public benchmark, in the same direction, and it is about what separates neighbouring models at the frontier.

Verdict: modest. Whatever its cause, the pattern is worth about one and a half ECI points, and a public summary should say so rather than call the scores "inflated".

2.15 Objection 14: the data have already changed

Referee. Epoch updated its data after your snapshot. Do the results still hold?

Epoch published a new benchmark snapshot on 2026-10-01, two days after ours. It adds 51 cells and four models: GPT-6 Sol, GPT-6 Luna, GPT-6.1 Sol and Grok 4.7, plus one new cell for Claude Opus 5.5. No existing cell changed its score, gap or organization, and no Google data were added. On the new snapshot the contrast is $+1.55$ ($z$ 3.05) without drift and $+1.60$ ($z$ 3.17) with drift, against 3.04 and 3.18 before.

Verdict: holds. The update is not a test of the Google claim, because it contains no Google models.

2.16 Objection 15: the garden of forking paths

Referee. Part 1 fitted a dozen models before it arrived at this one. Each choice moved the result: the grouping of Google, the drift, the ridge, the variance estimator. Show the whole distribution, not the favourite.

We collected every estimate of the contrast in this post that answers the question of Part 1 with a different analytic choice: 84 specifications in four families, 42 analytic choices each fitted with and without drift (Figure 2.5). Shifted and one-sided placebo cutoffs are left out, because they change the question rather than the way of answering it (Simonsohn, Simmons and Nelson 2020).

The curve has two parts.

• Outside the lab-specific family. $z$ reaches 1.96 in 65 of 70 specifications, with a median of 3.05. The four bootstrap $p$-values, shown as normal $z$, lie between 2.3 and 2.7. The exceptions are the delete-one-benchmark jackknife (1.90 and 1.73), a bandwidth of one year and a donut of six months, the last two where few Google models still have cells on both sides of their release.
• Inside it. Of the 14 specifications that give each lab its own trend or profile over benchmarks, only the linear trend reaches 1.96. The median is 0.18, and the triple difference is negative with drift.

The forking paths of Part 1 did not manufacture the contrast. The choices of variance estimator, ridge, weights and data move $z$ within a band around 3. The one fork that matters is whether each lab may have its own profile over benchmark vintage, and that is Objection 9, which reframes the result. The overall share of specifications that reach 1.96, 80%, is not itself evidence: it depends on how many variants of each family we chose to write down.

Verdict: weaker, for the reason given in Section 2.10.

Moral. A specification curve is worth drawing for its shape, not its share. Here the failures form one cluster, and that cluster is a design question.

2.17 A check without a fitted model: same-ECI twins

Every analysis so far runs through the step model or a relative of it. A cruder comparison needs no fitted model. Two models with the same capability have the same expected score on every benchmark, so the sign of their score difference should not depend on when a benchmark was released. A model tuned toward the benchmarks that were public at its release should beat a twin of the same capability on those benchmarks, and not on the ones released after it.

Design. We took every pair of models from different labs whose published ECIs differ by at most 2 points and whose release dates are 6 to 9 months apart (analysis/27_twins.py). There are 74 pairs with one Google model (27 Google models, 62 twins) and 247 pairs with no Google model, which are the control. The benchmarks that both models of a pair were scored on are split by release date into three groups: released before both models, between the two releases, and after both. The difference is $100(y_A-y_B)$ in Epoch's rescaled score, where 0 is chance and 100 the ceiling. Here $A$ is the Google model in Google pairs and the later model in control pairs. For a focal model we take its mean difference on the benchmarks released before it minus its mean difference on those released after it, and call this its advantage. The focal model is the Google model in Google pairs, and either model in control pairs, to match the two orientations. Pairs without benchmarks on both sides of the focal model's release drop out. Standard errors come from a bootstrap over focal models (2,000 replicates), which resamples together the pairs that share a focal model.

The control pairs show a pattern common to all labs. The later model gains on the benchmarks released before it ($+1.76\pm 1.24$), and the earlier model loses on them ($-1.23\pm 1.03$). The contrast (1.5) removes a common pattern like this by construction, and the step model identifies it poorly (Section 1.6). Beyond the control, Google's models gain in both orientations, by $+2.00\pm 4.38$ and $+1.56\pm 2.39$ points. The step model's contrast is worth about 3.7 points of score (Section 2.14). The twins are consistent with that value and with zero.

The comparison is weak for a plain reason. A pair shares a median of four benchmarks, and a single benchmark's difference has a standard deviation of 11.6 points. The pair with the most benchmarks in every group is Gemini 3.6 Flash (ECI 154.4, July 2026) and GPT-5.2 (153.5, December 2025). On the ten benchmarks that predate Gemini 3.6 Flash, it is ahead by 2.4 points on average; on the four released after it, it is behind by 1.2. Its single differences range from $-21.4$ (Furniture Assembly) to $+29.1$ (SimpleQA Verified, a benchmark built at Google; Haas et al. 2025, arXiv 2509.07968).

The twins also fit the vintage reading of Section 2.10. When the Google model is the later of the pair, most of its advantage comes from the benchmarks released after both models, where it is behind by 2.6 points. That is a deficit on the newest benchmarks more than a gain on the older ones.

Verdict: consistent, not conclusive. Without a fitted model, Google's twins point the same way as the contrast, with standard errors two to three times its size.

Every Google pair follows. Months are the Google model's release date minus its twin's. Each group cell gives the mean difference, Google minus twin, in score points, with the number of benchmarks in parentheses. Advantage is the mean difference on benchmarks released before the Google model minus that on benchmarks released after it. Four pairs share no benchmarks. The control pairs are in out/app27_twins_pairs.csv.

2.18 What survives

The objections answer three questions, and it helps to keep them apart. Thirteen ask whether there is a Google pattern and how large it is. Two, Objections 8 and 9, ask what shape it has. None of them can say by itself whether it is benchmaxxing. Table 2.1 lists the verdicts. The same-ECI twins of Section 2.17, which need no fitted model, point the same way as the contrast but cannot tell it from noise on their own.

Is there a Google pattern? Yes, with modest confidence.

• A pattern on Epoch's benchmarks. Relative to the other labs, Google's models score higher on benchmarks released in the year or so before them than on older or newer ones. The three resampling tests, which make the fewest assumptions, agree: the contrast that measures it has $p$ between 0.008 and 0.021, by the wild bootstrap over models (0.008 and 0.009), the permutation within half-years (0.011 and 0.016) and the wild bootstrap over benchmarks (0.018 and 0.021). It survives the removal of the ridges, quasi-binomial weights, a selection term, the removal of developer reports and of adversarially filtered benchmarks, and the data update. It is strongest on Epoch's own runs ($z$ 4.8).
• A modest size. About 1.5 ECI points, or 3.7 points of score on Google's post cells, less than the stated uncertainty of a single model's index.
• A search penalty. Google was not named before the data were seen. Adjusted for the search over labs, the evidence is at the edge of 5%: 0.03 to 0.05 under normality, 0.06 to 0.08 by permutation.
• Less confidence beyond Epoch's benchmarks. Treated as a sample of benchmarks, the 60 give $p\approx 0.02$, and five benchmarks released between October 2024 and May 2026 carry between half and two thirds of the contrast. A claim about public benchmarks in general would rest on a handful of them.

Is it a jump at the release date? No.

• Not a step at the release date. The contrast appears at placebo cutoffs where no leak is possible, a curve in benchmark date fits better than the step (and finds a planted step at full size), and the comparison of Google's own models on either side of the same benchmark's release finds nothing.
• Not leakage at release. The step was the model's encoding of training on public test sets. Without the step, the data say nothing for such leakage at Google relative to the other labs, and the curve would rarely have erased a leak of the size the step estimated (in 2% to 5% of simulations, Section 2.10).

These answers are about the model, not about the lab. They tell against the simplest story, test sets that leak into training and pay off from the day a model ships, and they leave the pattern itself in place.

Is it benchmaxxing? Open.

• Tuning to the benchmarks of the moment would produce the profile, and it is a kind of benchmaxxing. The triple difference leans against it within its limited power, and the injection test leans against a narrow version of it but not against a broad one (Section 2.10).
• A weakness on the newest kinds of benchmark would produce it too, and it is not benchmaxxing. Part of the profile is such a weakness: Google's models do relatively worse on agentic and tool-using benchmarks, which are most of the newest ones. Counting the tool-using mathematics benchmarks, that takes about a third of the contrast; counting only the strictly agentic ones, it takes none, and the pattern remains without them. We looked for it after seeing the profile (Section 2.10).

In one sentence. On Epoch's data, Google's models look best on the benchmarks of their day, and the evidence for that pattern is good but not overwhelming ($p$ of 0.01 to 0.02, at the edge of 5% once the search over labs is counted). It is not the jump at release that training on test sets would leave. Part of it is a weakness on agentic and tool-using work, and the data in hand cannot yet say whether the rest reflects tuning to current benchmarks.

What would settle it. The prospective test of Section 2.7 needs new Google models. A vintage profile predicts that they do relatively badly on the benchmarks of 2025 and 2026 whether or not those benchmarks predate them; tuning predicts that they gain on the benchmarks that were current while they were trained. On the data in hand, a classification of the benchmarks by format and domain, written down before looking at the fits, would test the profile reading more cleanly than our post hoc one.

Bibliographic note

The referee reports are ours. Wald, likelihood ratio and profile methods for nonlinear least squares are covered by Bates and Watts (1988). Wild cluster bootstrap inference with few or unbalanced clusters is covered by Cameron, Gelbach and Miller (2008) and by MacKinnon and Webb (2017, 2018); MacKinnon, Nielsen and Webb (2023) review cluster-robust inference and the CV3 jackknife. Placebo cutoffs, donuts and bandwidths follow Imbens and Lemieux (2008). Gelman and Imbens (2019) argue against high-order polynomials in discontinuity designs. Gelman and Loken (2014) and Simonsohn, Simmons and Nelson (2020) discuss forking paths and specification curves.