Inference

One Token at a Time

When a language model generates text, it doesn't produce the whole response at once. It generates one token at a time, and each new token depends on every token that came before it. This is called autoregressive generation: the model's own output becomes part of its input for the next step.

Here's what happens concretely when you send a prompt to a model:

1. The prompt is tokenized into a sequence of tokens.
2. The entire sequence passes through the transformer (all layers, all attention heads).
3. The final layer outputs a vector of logits -- one number per token in the vocabulary. A typical vocabulary has 32,000 to 128,000 tokens.
4. The logits are converted to a probability distribution via softmax.
5. A token is selected from this distribution (more on how below).
6. That token is appended to the sequence, and steps 2-5 repeat.

This loop continues until the model produces a special end-of-sequence token or hits a maximum length. Every response you've ever gotten from a language model was generated this way -- one token after another, each conditioned on all previous tokens.

This architecture means that generating a sequence of length N requires N forward passes through the transformer. Each forward pass is relatively cheap compared to a training step (no backpropagation, no gradient computation), but they add up -- a 1,000-token response requires 1,000 sequential forward passes. This sequential nature is the fundamental bottleneck of autoregressive generation, and most of the engineering tricks in this chapter exist to make it faster.

From Logits to Tokens: The Sampling Decision

After each forward pass, the model outputs a vector of logits -- raw, unnormalized scores. A positive logit means the model thinks the token is likely; a negative logit means unlikely. These are converted to probabilities by the softmax function:

This gives a proper probability distribution: all values between 0 and 1, summing to 1. For the prompt "The capital of France is," the distribution might put 0.85 on "Paris," 0.03 on "the," 0.02 on "a," and spread the remaining 0.10 across thousands of other tokens.

The question is: how do you pick the next token from this distribution? This is where decoding strategies come in, and they have a major impact on output quality.

Greedy decoding

The simplest strategy: always pick the token with the highest probability. This is greedy decoding. It's deterministic -- the same prompt always produces the same output. It's fast. And for factual questions, it often gives the best answer.

The problem with greedy decoding is that it produces dull, repetitive text for open-ended generation. It gravitates toward the most common continuations, avoids creative language, and frequently gets stuck in loops ("The the the the..."). This happens because the locally optimal choice at each step doesn't always lead to the globally best sequence.

Sampling

The alternative: sample randomly from the probability distribution. This introduces randomness -- different runs produce different outputs. A token with probability 0.3 will be selected 30% of the time, a token with probability 0.05 will be selected 5% of the time. This produces more diverse, creative text, but it also introduces the risk of sampling low-probability tokens that derail the generation ("The capital of France is Tuesday").

The practical strategies are all about controlling this trade-off between diversity and coherence.

Temperature

Temperature is a scalar that divides the logits before softmax:

The effect is intuitive once you see it:

• T = 1.0: The distribution is unchanged. Standard sampling.
• T < 1.0 (e.g., 0.3): Dividing by a number less than 1 increases the logits' magnitudes, making the distribution sharper. High-probability tokens become even more likely; low-probability tokens become negligible. As T approaches 0, sampling converges to greedy decoding.
• T > 1.0 (e.g., 2.0): Dividing by a number greater than 1 decreases the logits' magnitudes, making the distribution flatter. All tokens become more equally likely. As T approaches infinity, the distribution becomes uniform random over the entire vocabulary.

Temperature doesn't change which token is most likely -- it changes how much more likely it is than the alternatives. This makes it a useful control knob: lower temperature for factual tasks where you want the model's best guess, higher temperature for creative tasks where you want variety.

Top-k Sampling

Top-k sampling takes a different approach to controlling randomness: instead of adjusting probabilities, it restricts the candidate set. After computing the full probability distribution, only the k most likely tokens are kept. All other tokens are set to probability zero. The remaining probabilities are renormalized (scaled up so they sum to 1), and sampling proceeds from this reduced set.

With k = 50, the model can only choose among its top 50 candidates. This prevents the catastrophic low-probability samples that can derail generation ("Tuesday" as the capital of France) while preserving some diversity within the high-probability candidates.

The limitation of top-k is that k is fixed regardless of context. Sometimes the model is very confident and only 5 tokens are reasonable. Sometimes it's uncertain and 500 tokens are plausible. A fixed k is too restrictive in the second case and too permissive in the first.

Top-p (Nucleus) Sampling

Top-p sampling, introduced by Holtzman et al. (2019), solves this problem elegantly.¹ Instead of fixing the number of candidates, it fixes the cumulative probability mass. You sort tokens by probability from highest to lowest, then include tokens until their cumulative probability reaches p.

With p = 0.9:

• If the top token has probability 0.95, only that one token is considered (its mass already exceeds 0.9).
• If the top 50 tokens each have probability ~0.02, all 50 are included (it takes about 45 tokens to reach cumulative probability 0.9).

This adapts automatically to the model's confidence. When the model is sure, the candidate set is small. When it's uncertain, the candidate set expands. This produces noticeably better text than top-k for the same level of diversity.

In practice, most API providers use top-p sampling as the default (OpenAI, Anthropic, and others all expose a top_p parameter). Temperature and top-p are often combined: temperature adjusts the shape of the distribution, then top-p trims the tails.

The KV Cache

We said that generating N tokens requires N forward passes. But there's a crucial optimization that avoids redundant computation: the KV cache (key-value cache).

Recall from the attention mechanism (covered in earlier chapters): for each token, the model computes three vectors -- a query (Q), a key (K), and a value (V). Attention works by comparing the current token's query against the keys of all previous tokens, then taking a weighted sum of the corresponding values.

The critical observation: when generating token N+1, the keys and values for tokens 1 through N are exactly the same as they were when generating token N. The model is only adding one new token -- the previous tokens haven't changed. So there's no need to recompute their keys and values.

The KV cache stores the key and value vectors for all previously processed tokens. At each generation step, only the new token's keys and values are computed and appended to the cache. The attention computation uses the cached keys and values for previous tokens and the freshly computed ones for the new token.

Without the KV cache, generating a sequence of length N would require computing attention over the full sequence at every step -- O(N) work per step, O(N^2) total. With the KV cache, each step only computes the new token's contribution -- O(1) for the new computation (plus O(N) for the attention lookup against cached values), giving O(N) total new computation. This is the difference between generation being painfully slow and being merely slow.

The trade-off: the KV cache consumes memory. For each layer and each attention head, you store a key vector and a value vector for every token in the context. For a model with 32 layers, 32 heads, and 128-dimensional head vectors, the KV cache for a 4,096-token context is about 1 GB in fp16. For a 128K-token context, that's about 32 GB -- sometimes more than the model weights themselves. This is why long-context models need so much memory at inference time, and it's the main practical constraint on context window length.

Context Windows

The context window is the maximum number of tokens the model can process in a single forward pass. It's determined by two factors:

• Positional encoding capacity. The model needs a way to know where each token is in the sequence. The original transformer used fixed sinusoidal encodings, which have a natural length limit. Modern models use RoPE (Rotary Position Embedding), which can be extended beyond its training length with some degradation.²
• Memory and compute. Attention is O(N^2) in sequence length (each token attends to all others). The KV cache grows linearly with context length. Both put practical limits on how long a context can be.

Context windows have grown rapidly:

The jump from 2K to 1M tokens in five years is remarkable, but it came with significant engineering. Techniques like FlashAttention (Dao et al., 2022) make attention computation more memory-efficient by restructuring it to exploit GPU memory hierarchy.³ Multi-query attention and grouped-query attention reduce the KV cache size by sharing key and value heads across multiple query heads.⁴ Without these optimizations, million-token context windows would be impractical.

The Economics: Inference vs. Training

Training a frontier model costs tens to hundreds of millions of dollars. A single inference call costs a fraction of a cent. But consider the scale: a popular model might serve billions of inference requests per month. Over the model's lifetime, total inference cost often exceeds training cost by a large margin.

This asymmetry drives a massive engineering effort around inference efficiency:

• Quantization for inference. As Chapter 14 discussed for training, quantization is even more common at inference time. Many deployed models run in int8 or int4 precision. The quality loss is minimal; the speedup and memory savings are substantial.
• Batching. Process multiple requests simultaneously. The model weights are loaded once and shared across all requests in the batch. This amortizes the memory bandwidth cost of loading weights -- the main bottleneck in inference for large models.
• Continuous batching. Requests arrive and finish at different times. Rather than waiting for a full batch to complete, continuous batching inserts new requests into available slots as old ones finish, keeping the GPU fully utilized.

Speculative Decoding

A particularly clever trick for reducing latency is speculative decoding (Leviathan et al., 2022; Chen et al., 2023).⁵ The idea exploits the fact that autoregressive generation is bottlenecked by sequential steps, not by the computation within each step. A smaller, faster model can often predict what the larger model will say.

The process:

1. A small draft model generates k tokens quickly (e.g., k = 5).
2. The large target model processes all k tokens in a single forward pass (this is parallel, not sequential, because the tokens are already determined).
3. The target model's predictions are compared against the draft tokens. Any token where the target agrees is accepted. At the first disagreement, the target model's prediction is used and the remaining draft tokens are discarded.

If the draft model's predictions are frequently correct (which they are for straightforward text), you effectively generate multiple tokens per forward pass of the large model. In practice, speculative decoding achieves 2-3x speedup without any change in output quality -- the final output is statistically identical to what the target model would have produced on its own.

Putting It Together

Inference is where the rubber meets the road. The architecture, training, and fine-tuning from previous chapters all converge into this single operation: a forward pass through the transformer, repeated token by token, producing the text you read.

The key takeaways:

• Generation is sequential and autoregressive. Each token depends on all previous tokens. This is the fundamental computational structure.
• Temperature, top-k, and top-p control the trade-off between coherence and diversity. Lower temperature and smaller top-p for factual tasks; higher values for creative ones.
• The KV cache is the single most important inference optimization. It trades memory for computation, turning O(N^2) generation into O(N).
• Context window length is limited by memory and positional encoding, not by the model's architecture. Engineering advances have pushed this from 1K to 1M tokens in five years.
• Inference cost dominates training cost at scale. This drives an entire field of optimization: quantization, batching, speculative decoding.