Pre-training

The Objective

The core of pre-training is almost comically simple to state. Given a sequence of tokens, predict the next one. That's it. The entire foundation of GPT, Claude, Llama, and every other modern large language model rests on this single objective: causal language modeling, which is a formal way of saying "read left to right, guess what comes next."

More precisely: given a sequence of tokens [t₁, t₂, ..., t_n], the model produces a probability distribution over its entire vocabulary for token t_n+1. It assigns a probability to every possible next token — "the" might get 0.12, "a" might get 0.07, "defenestration" might get 0.000003 — and the training signal comes from comparing that distribution against what actually came next in the training data.

The loss function is cross-entropy loss over the vocabulary. If the model assigns probability p to the correct next token, the loss for that prediction is −log(p). When the model is confident and correct (assigns high probability to what actually came next), the loss is small. When it's wrong or uncertain, the loss is large. Averaged across billions of predictions, this single number — the cross-entropy loss — is the objective the model minimizes during training.

What makes this objective powerful is that it's self-supervised. The training data doesn't need human labels. Any text corpus is automatically a source of training examples: every sequence of tokens contains a prediction target at every position. A single book with 100,000 tokens yields roughly 100,000 training examples. A dataset of trillions of tokens yields trillions of training examples, each one free.

The Data

Pre-training data for modern LLMs is drawn from a mix of sources, each contributing different kinds of knowledge and language. The goal is breadth: the model should encounter as many domains, registers, languages, and reasoning styles as possible.

Major sources

The total pre-training corpus for frontier models is on the order of trillions of tokens. Llama 2 (Meta, 2023) was trained on 2 trillion tokens.² Llama 3 (2024) scaled to approximately 15 trillion tokens.³ These are not small numbers. For reference, the entire English Wikipedia is roughly 4 billion tokens. A trillion-token dataset is like reading Wikipedia 250 times over, except the rest is web pages, books, code, and everything else.

Data quality

Raw web data is garbage. Anyone who has scraped the internet knows this. Common Crawl contains exact duplicates, near-duplicates, SEO spam, auto-generated text, pornography, personal information, malware payloads, and vast amounts of text that is technically English but carries no useful signal.

The filtering pipeline is a significant engineering effort in its own right:

• Deduplication: Exact and near-duplicate removal at the document and paragraph level. MinHash and locality-sensitive hashing are standard tools. Without deduplication, the model memorizes repeated content rather than learning general patterns. The RefinedWeb dataset (Penedo et al., 2023) demonstrated that aggressive deduplication of Common Crawl alone could match or exceed the quality of curated datasets.⁴
• Language filtering: Identifying and separating languages, removing pages with mixed or unidentifiable language. Typically done with fastText-based classifiers.
• Quality filtering: Heuristic and model-based filters to remove low-quality text. Common heuristics include document length, fraction of alphabetic characters, ratio of special characters, and perplexity scores from a smaller reference model. GPT-3's training data used a classifier trained on a curated "high-quality" reference set to score web documents.⁵
• Toxic content removal: Classifiers trained to identify hate speech, explicit content, and personally identifiable information. Imperfect — no filter catches everything, and over-filtering can remove legitimate content about sensitive topics.
• Data mixing: The final training set is a weighted mixture, not a flat concatenation. Wikipedia and books are typically upsampled (repeated) relative to web data because their quality-per-token is higher. The exact mixing ratios are a key design decision that affects model behavior, and most labs treat them as proprietary.

The Scale

To appreciate what pre-training involves, consider the compute required. Training a frontier model is not something you do on a workstation, a university cluster, or even a modest cloud setup. It is an industrial operation.

GPT-3 (175 billion parameters) required approximately 3,640 petaflop-days of compute to train — equivalent to running a petaflop-class system for ten years, or a thousand such systems for 3.6 days.⁵ In practice, OpenAI used a cluster of thousands of Nvidia V100 GPUs. The estimated cost at 2020 cloud GPU prices was in the range of $4.6 million for a single training run.⁶

The numbers have only grown. Meta's Llama 3 405B model used a cluster of 16,384 Nvidia H100 GPUs and trained for approximately 54 days.³ GPT-4's training costs have been estimated at over $100 million, though OpenAI has not confirmed specifics. The energy consumption of a single frontier training run is measured in gigawatt-hours — comparable to the annual electricity consumption of a small town.

The progression from GPT-1 to GPT-3 spans three orders of magnitude in parameters (117 million to 175 billion) and about two orders of magnitude in training data. Each jump brought qualitative changes in capability that were not predicted by extrapolating from the previous model. GPT-1 could complete sentences. GPT-2 could generate coherent paragraphs. GPT-3 could perform tasks it was never explicitly trained for.

What Emerges at Scale

The most consequential finding of the pre-training era is that large language models develop capabilities that nobody designed into them. These capabilities were not predicted from the architecture, not specified in the training objective, and not present in smaller models trained the same way. They appeared as the models got bigger.

In-context learning

GPT-3's most striking capability was in-context learning: the ability to perform a task by being shown examples in the prompt, without any parameter updates. Give GPT-3 a few examples of English-to-French translation, and it translates. Give it examples of sentiment classification, and it classifies. Give it examples of a made-up task — say, reversing words — and it often picks up the pattern. This is few-shot learning, and smaller models trained with the exact same objective simply couldn't do it.⁵

In-context learning was not engineered. Nothing in the training objective says "learn to learn from examples in the prompt." The model was trained only to predict the next token. But predicting the next token in a diverse corpus means recognizing that certain contexts establish patterns — a list of input-output pairs followed by a new input strongly predicts an output that follows the established pattern. The model learned to detect and follow patterns in context because doing so reduced its prediction loss.

Chain-of-thought reasoning

At sufficient scale, models can be prompted to "think step by step" and produce intermediate reasoning before an answer. This chain-of-thought behavior, first systematically studied by Wei et al. (2022),⁷ dramatically improves performance on arithmetic, logic, and commonsense reasoning tasks. Crucially, it's an emergent capability: the same prompting technique applied to smaller models produces no improvement. There appears to be a threshold — roughly in the range of 60-100 billion parameters — below which chain-of-thought prompting doesn't help and above which it does.

Zero-shot task performance

Perhaps most remarkably, large models can perform tasks with no examples at all — just a natural language instruction. "Translate this sentence to French." "Summarize this paragraph." "Write a Python function that..." This zero-shot capability emerges from the model having internalized enough task structure from its training data that it can generalize to novel instructions.

Scaling Laws

If larger models produce better results, the natural question is: how much better? And: what's the most efficient way to get there?

In January 2020, Jared Kaplan and colleagues at OpenAI published a paper that gave the field its first precise answers.⁸ They showed that the performance of language models — measured as cross-entropy loss on held-out test data — follows power-law relationships with three variables:

• N — the number of model parameters
• D — the size of the training dataset (in tokens)
• C — the compute budget (in FLOPs, floating-point operations)

The relationships are strikingly smooth. Over many orders of magnitude, the test loss decreases as a power law:

L(N) ∝ N^−0.076     L(D) ∝ D^−0.095     L(C) ∝ C^−0.050

The exponents are small — meaning you need to increase each variable by a lot to get a meaningful improvement. Roughly: a 10x increase in model size yields about a 17% reduction in loss. A 10x increase in data yields about a 20% reduction. These are diminishing returns, but they're predictable, and they don't plateau within the ranges that have been tested.

The Kaplan scaling laws had an immediate practical implication: for a fixed compute budget, it's more efficient to train a larger model for fewer steps than a smaller model for more steps. This is counterintuitive — it suggests you should build the biggest model you can afford and stop training before it fully converges — but the data supported it clearly.

The Chinchilla correction

Two years later, Hoffmann et al. at DeepMind published a correction that reshaped the field.⁹ Their paper, known as the "Chinchilla paper" after their model, argued that Kaplan et al. had underweighted the importance of training data relative to model size. The Kaplan laws suggested making models as large as possible for a given compute budget. Chinchilla showed that the optimal balance was roughly equal scaling of parameters and training tokens.

The specific finding: for compute-optimal training, a model should be trained on approximately 20 tokens per parameter. By this rule, a 70 billion parameter model should see about 1.4 trillion tokens. Chinchilla itself was a 70 billion parameter model trained on 1.4 trillion tokens, and it outperformed the 280 billion parameter Gopher model that had been trained on only 300 billion tokens — a model four times larger but fed five times less data.

This had immediate consequences. The field had been racing to build the largest possible models (the "bigger is better" era). Chinchilla showed that many existing models were undertrained — they had more parameters than their training data could support. The efficient path was not just "make it bigger" but "make it bigger and feed it proportionally more data."

Later work by the Llama team at Meta took this even further. Llama 1 (2023) trained a 65 billion parameter model on 1.4 trillion tokens — right at the Chinchilla-optimal ratio. But Llama 2 pushed to 2 trillion tokens for the same 70B model size, and Llama 3 went to 15 trillion tokens for a 70B model, far beyond what Chinchilla prescribed.²³ Why? Because Chinchilla optimizes for a fixed compute budget. If your goal is to produce the best possible model at a given size — perhaps because you need a model small enough to deploy on limited hardware — it pays to overtrain it on more data than the compute-optimal ratio suggests. The loss keeps decreasing, just less efficiently per FLOP.

The Bitter Lesson

In 2019, Rich Sutton — one of the founders of modern reinforcement learning — published a short essay called "The Bitter Lesson."¹⁰ His argument: across the 70-year history of AI research, general methods that leverage computation have consistently won over methods that try to encode human knowledge. Chess programs built on brute-force search beat chess programs built on grandmaster heuristics. Speech recognition systems based on statistical models beat systems built on linguistic rules. Machine translation systems trained on parallel text beat systems designed by linguists.

The "bitter" part is that researchers keep making the same mistake. They build clever, domain-specific systems that work well at small scale, and then general-purpose methods — with enough compute — blow past them. The lesson keeps being learned and then forgotten.

Pre-training is perhaps the purest example of the bitter lesson in action. The next-token prediction objective is not clever. It encodes no linguistic knowledge, no rules of reasoning, no understanding of the world. It's a general statistical objective applied with enormous compute to enormous data. And it produces systems that can translate, summarize, reason, write code, and answer questions — none of which were specified in the objective.

Sutton's conclusion: the two methods that scale with computation are search and learning. Everything else — hand-crafted features, domain-specific architectures, expert-designed rules — provides diminishing returns relative to simply using more compute for search and learning. The history of AI bears this out, and the scaling laws formalize it: make the model bigger, give it more data, spend more compute, and performance improves along a smooth, predictable curve.

This doesn't mean engineering doesn't matter. The Transformer architecture (Chapter 9) was a crucial innovation — not because it encoded linguistic knowledge, but because it was more parallelizable and therefore could leverage computation more efficiently than RNNs. The bitter lesson isn't that architecture doesn't matter. It's that the architectures that win are the ones that make the best use of compute.

The Compute Reality

Pre-training a frontier model requires infrastructure at an industrial scale. The key resource is GPU-hours — specifically, hours on Nvidia's data-center GPUs (A100, H100, and their successors). A rough accounting:

• Hardware: Thousands to tens of thousands of GPUs, interconnected with high-bandwidth networking (Nvidia's NVLink and InfiniBand). A single H100 GPU costs roughly $25,000-$40,000. A 10,000-GPU cluster represents $250-400 million in hardware alone.
• Training time: Weeks to months of continuous operation. During training, the cluster runs 24/7. Hardware failures are expected and must be handled automatically — checkpoint, restart, continue.
• Energy: A single H100 GPU draws approximately 700 watts at peak. A 10,000-GPU cluster draws 7 megawatts — plus cooling, networking, and overhead. A training run lasting 60 days consumes on the order of 10 gigawatt-hours of electricity.
• Engineering: Distributed training across thousands of GPUs requires sophisticated parallelism strategies: data parallelism (same model, different data on each GPU), tensor parallelism (splitting individual matrix operations across GPUs), pipeline parallelism (splitting model layers across groups of GPUs). Getting these right is a significant engineering challenge.

The cost and infrastructure requirements create a natural concentration of pre-training capability. As of 2025, the organizations that can train frontier models from scratch include OpenAI, Anthropic, Google DeepMind, Meta, Mistral, and a small number of Chinese labs (ByteDance, Alibaba, Baidu). The gap between what's needed for frontier pre-training and what an academic lab or startup can afford continues to grow.

Pre-training vs. Fine-tuning

Pre-training creates a general-purpose foundation. The model that comes out of pre-training is a powerful text predictor, but it's not immediately useful as an assistant, a code generator, or anything task-specific. It has absorbed the patterns and knowledge of its training data, but it has no notion of "follow the user's instructions" or "be helpful and honest."

That's where fine-tuning comes in — covered in detail in Chapter 13. The distinction matters:

An analogy: pre-training is like educating a person broadly — years of reading, observing, absorbing how the world works, building a general understanding. Fine-tuning is like job training — teaching them the specific behaviors, norms, and skills their role requires. The education doesn't specify the job, but without it, the job training has nothing to build on. A fine-tuned model that was never pre-trained would be useless. A pre-trained model that was never fine-tuned can be powerful but is unrefined — it might complete your sentence with the rest of a Wikipedia article, or continue your question with another question, rather than answering it.

The asymmetry in compute cost is important. Pre-training is the expensive part — millions of dollars, weeks of training, trillions of tokens. Fine-tuning is comparatively cheap — thousands of dollars, hours to days, thousands of examples. This asymmetry is what enables the current ecosystem: a few large organizations pre-train foundation models, and the rest of the field fine-tunes them for specific applications.

Why Next-Token Prediction Works

There's a lingering puzzle: why does predicting the next token produce something that looks like understanding? The model is never told what a fact is, what an argument is, or what it means for something to be true. It's just trying to minimize prediction error on text. How does that produce a system that can answer questions, write code, and reason about novel problems?

One way to think about it: predicting the next token in a sufficiently diverse corpus is equivalent to compressing that corpus, and compression requires understanding. If the text says "The capital of France is," predicting "Paris" requires factual knowledge. If the text is a proof, predicting the next step requires logical reasoning. If the text is a conversation, predicting the next utterance requires modeling intent, context, and social convention. Every domain of human knowledge and every mode of reasoning, insofar as they appear in text, become prediction targets.

This doesn't mean the model "understands" in any deep philosophical sense — that question is open and probably not well-posed. What it means is that the statistical objective of next-token prediction, applied at scale, creates internal representations that capture an enormous amount of structure about language, knowledge, and reasoning. Whether those representations constitute understanding or merely simulate it is a question for philosophers. For engineering purposes, the capabilities are real and useful.

Ilya Sutskever, co-founder of OpenAI, put it concisely: "If you predict the next token well enough, you must have a world model inside." Whether that "must" is philosophically justified is debatable. But empirically, models that predict the next token better also perform better on tasks that seem to require world knowledge, spatial reasoning, and causal inference. The correlation is strong enough to drive hundreds of billions of dollars in investment.

Pre-training takes a pile of text and a simple objective — predict what comes next — and, given enough parameters and enough compute, produces something that looks like knowledge. The scaling laws tell us this process is predictable and hasn't plateaued. The emergent capabilities tell us it produces surprises along the way. The bitter lesson tells us this has happened before, across the entire history of AI.

But the model that comes out of pre-training is a statistical object: billions of floating-point numbers organized into weight matrices. What do those numbers actually represent? What is the model's "knowledge," and in what sense does it "know" anything?