How Models Are Built: Data, Training and Scaling Laws

A large language model is built in three stages. Pre-training teaches a neural network to predict the next token across trillions of words of text, code and images, and it is where most of the compute goes. Mid-training and post-training turn that raw predictor into a usable assistant: supervised fine-tuning on examples, preference training (RLHF and its successors), and, since late 2024, large-scale reinforcement learning on problems with checkable answers, which produced the "reasoning" models that think before they answer. Inference-time scaling, letting a model spend more tokens on a hard question, is the third lever. Behind all three sit scaling laws: empirical curves, first published by OpenAI researchers in January 2020 and corrected by DeepMind's "Chinchilla" paper in March 2022, showing that loss falls predictably as compute, parameters and data grow together. Those curves are the reason the industry spends tens of billions of dollars a year on training clusters: they turned model building from research into something closer to capital budgeting. They also expose the constraints. Compute doubles the cost of each step of improvement; the stock of high-quality public human text is finite (Epoch AI's central estimate is ~300 trillion tokens, which frontier training could fully use some time between 2026 and 2032, June 2024); synthetic data and distillation help but carry their own risks; and the economics of a model depend as much on what it costs to serve as on what it cost to train. This section explains the stages of model building, the scaling laws and their limits, the data supply chain (web crawls, licensed corpora, human labellers and synthetic data), distillation and its disputes, what the big training runs have cost, and how all of this shapes the business of the labs. A worked example computes a stylized compute and data budget from published scaling-law fits.