Posts 6
- A Statistical Analysis of LLM Benchmaxxingbenchmarksstatistics
Do language models do suspiciously well on tests that existed before they did? Benchmarks released after a model work as a natural control group: I add one parameter to Epoch's Capabilities Index and estimate, lab by lab, how much better models score on the benchmarks they could have trained toward.
- Token Laundering
Say you want to build your own LLM. One of the oldest practices in the book is to generate high quality synthetic data using a state-of-the-art model, and then use that…
- My AI Stack (June 2025)
This is my regular monthly post to track my AI tool ecosystem. Here is the May post.
- Statistics in the Age of AI: Reflections from the GW Panel
I was invited as a panelist to GW Statistics Department’s 90th birthday conference on “Statistics in the Age of AI”. Here are my thoughts from the panel discussion and…
- My AI Stack (May 2025)
I’ve decided to start a monthly series tracking my AI tool ecosystem. This serves two purposes: documenting my own workflow changes and providing a practical reference…
- How Accurate Are Lords of Limited's Early Format Guesses? An AI Analysis
Spoiler: Surprisingly good, but maybe don’t trust them on Green.