The previous chapter established that current neural networks can't learn continuously without forgetting. But even if they could, there's a prior question: what would make them want to? A system capable of continual learning is necessary but not sufficient for genuine growth. It also needs a reason to seek out new knowledge — something that drives it to explore, to be dissatisfied with its current understanding, to find the gaps in its model of the world and fill them.
In biological organisms, this drive has a name: curiosity. And in the RL framework introduced in Chapter 21, curiosity has a formal counterpart: intrinsic motivation — reward signals that come from inside the agent rather than from the environment.
Standard reinforcement learning assumes the reward comes from the environment. The agent plays a game and gets points. The robot assembles a widget and gets a success signal. The language model generates text and gets a human preference score. This is extrinsic reward — defined externally, handed to the agent by something outside itself.
Extrinsic reward works well when you can define it precisely. But many interesting problems have sparse reward: the signal only appears rarely (e.g., winning a game after thousands of moves) or not at all until the task is complete. In these settings, the agent has to explore extensively before it ever encounters a reward signal, which makes learning extremely slow.
Intrinsic motivation supplements or replaces extrinsic reward with signals generated by the agent itself. The agent rewards itself for doing things that are, in some formal sense, "interesting" — regardless of whether the environment provides a reward. The most studied form of intrinsic motivation is curiosity: the agent seeks out states where it is maximally surprised, maximally uncertain, or maximally improving its understanding.
The most influential formal theory of curiosity comes from Jurgen Schmidhuber, who has been developing it since 1991.1 The core idea is elegant: curiosity is the drive to seek experiences that maximally improve the agent's internal model of the world.
Schmidhuber frames this as compression progress. The agent builds a predictive model of its experience — a compressor that tries to find patterns and regularities in the data stream. Curiosity is the reward for finding new patterns that improve the compression. In other words: the agent is rewarded not for prediction accuracy, but for improvement in prediction accuracy.
This distinction — between prediction error and prediction improvement — is crucial. Raw prediction error would drive the agent to stare at random noise forever, since random noise is maximally unpredictable. But compression progress is zero for random noise: the model can't improve its compression of truly random data. Curiosity, defined as compression progress, automatically distinguishes between learnable novelty (interesting) and irreducible randomness (boring).
Schmidhuber's 2009 paper formalized this as a unified theory of curiosity, creativity, and humor: all three involve the subjective experience of unexpected compression progress.2 A joke is funny because the punchline lets you suddenly compress the setup in a new way. A creative work is satisfying because it reveals a pattern you hadn't seen. These are interpretive claims, but the formal framework is precise.
The most widely cited implementation of curiosity-driven learning is Pathak et al.'s Intrinsic Curiosity Module (ICM), published in 2017.3 ICM provides a practical answer to the question: how do you compute a curiosity reward that an RL agent can actually use?
The architecture has three components:
The curiosity reward is the prediction error of the forward model in feature space, not in pixel space. This is the key design choice. Predicting pixels would be dominated by irrelevant details — the exact position of every leaf on a tree, the precise texture of a wall. But the feature encoder, trained via the inverse model, learns to represent only the aspects of the environment that the agent can actually affect through its actions. Prediction error in this space measures genuine surprise about the consequences of the agent's behavior, not just perceptual noise.
Pathak et al. demonstrated ICM on video game environments where the extrinsic reward was extremely sparse or entirely absent. In Super Mario Bros., an ICM-driven agent explored levels effectively with no extrinsic reward at all — it was motivated purely by the desire to see things its forward model couldn't predict. When combined with sparse extrinsic reward, curiosity-driven exploration found rewards much faster than random exploration.
There's a well-known failure mode for curiosity-driven agents, and it illuminates a deep issue. Imagine an agent in a room with a television showing static — random noise. If curiosity is defined as prediction error, the TV is the most interesting thing in the room. The agent will stare at the TV forever, because random noise is maximally unpredictable. But staring at static teaches nothing.
This is the noisy TV problem. It shows that raw prediction error is not a good curiosity signal. You need to distinguish between:
Schmidhuber's compression progress handles this correctly: staring at noise produces zero compression progress, so the curiosity signal goes to zero. ICM partially handles it through the feature space trick — if the noise isn't affected by the agent's actions, the inverse model won't include it in the learned features. But the problem isn't fully solved. In complex environments with many sources of stochastic variation, distinguishing learnable from irreducible uncertainty remains difficult.
More recent approaches, like Random Network Distillation (RND) by Burda et al. (2019), use a different signal: the prediction error of a learned network trying to match a fixed random network's outputs.4 Novel states produce high error (the learned network hasn't seen similar inputs), while familiar states produce low error. This is simpler than ICM and avoids some of its failure modes, but it's purely a novelty detector — it doesn't capture learning progress.
Curiosity-driven exploration within a single environment eventually saturates: the agent has seen everything, its model is as good as it's going to get, and the curiosity signal goes to zero. For a system that's supposed to learn indefinitely, this is a problem. The environment itself needs to keep generating new challenges.
This is the premise of open-ended learning: systems where both the agent and the environment co-evolve, creating an endless supply of novel challenges. The most prominent example is POET (Paired Open-Ended Trailblazer), developed by Wang et al. at Uber AI in 2019.5
POET works as follows:
The result is a co-evolutionary process where environments get progressively harder and agents get progressively more capable, with neither side having a fixed endpoint. POET discovered agents that could traverse terrain far more complex than any manually designed curriculum, because the environments evolved to be at the right difficulty level for the current agents — always in the sweet spot where learning progress is possible.
This brings us to an observation that cuts across all the formal theory. Alfonso articulated it from a different angle:
"With LLMs, there is a finite amount of uncertainty — whereas for humans, it's infinite. The limited amount of power that we have makes uncertainty so much more daunting. Compute power isn't what makes a human have that drive — so maybe that's not the constraint."
This maps precisely onto a real problem in intrinsic motivation research. Consider an agent with unlimited compute and unlimited memory. It could, in principle, build a perfect model of any finite environment — predicting every future state with certainty. Once it does, curiosity (prediction error or compression progress) drops to zero permanently. The agent has no reason to act, no reason to explore, no reason to do anything at all. It has achieved total certainty, and total certainty is total stasis.
Now consider an agent with severely limited compute and memory. It can't model everything. It has to choose what to attend to, what to remember, what to discard. Every choice to model one part of the environment means not modeling another part. Uncertainty is permanent and inescapable — not because the environment is infinitely complex (though it may be), but because the agent's resources are finite. This permanent uncertainty is what keeps the curiosity signal alive.
This is the constraint-as-feature principle: the limitation of finite resources isn't a problem to be solved — it's the mechanism that generates the drive to learn. An agent with bounded compute that operates in a sufficiently complex environment will always have more to learn than it can process. This permanent gap between what the agent knows and what it could know is what keeps the curiosity signal nonzero.
The biological parallel is direct. The human brain, despite its ~86 billion neurons and ~100 trillion synapses, is a severely bounded computational device operating in a world of effectively infinite complexity. The limbic system — the seat of motivation and drive (Chapter 2) — creates the subjective experience of curiosity precisely because the organism can never resolve all uncertainty. You are curious because you can't know everything. If you could, you wouldn't be.
This has an uncomfortable implication for AI research: if the goal is to build a system with genuine curiosity and drive, giving it more compute might be counterproductive. The drive to learn may require the inability to learn everything. This is speculative — it depends on how you formalize "drive" and "curiosity" — but it's consistent with both the formal theory (compression progress requires that compression isn't perfect) and the biological evidence (organisms with simpler brains often show more exploratory behavior per neuron than larger-brained ones).
The formal curiosity models have clear biological counterparts. The dopaminergic system in the brain, discussed in Chapter 2, doesn't just encode reward — it encodes reward prediction error. Schultz, Dayan, and Montague (1997) showed that dopamine neurons fire not when reward is received, but when reward is unexpected.6 When a reward is predicted perfectly, dopamine firing drops to baseline. When a predicted reward doesn't arrive, dopamine dips below baseline.
This is exactly the signal that curiosity-driven RL systems use: the gap between prediction and reality. The brain's reward system is, at its core, a prediction error computer. And the subjective experience of curiosity — the drive to investigate something novel, the satisfaction of understanding something new — appears to be linked to this prediction error signal.
But there's a crucial difference between biological and artificial curiosity systems. In biology, the motivation system is embodied — it's connected to survival, homeostasis, social needs, and the entire web of biological drives. Curiosity in a rat isn't just about prediction error; it's modulated by hunger, fear, energy levels, social context. The limbic system doesn't compute curiosity in isolation; it integrates curiosity with every other drive the organism has.
Artificial curiosity systems, by contrast, optimize a single mathematical signal in isolation. The agent maximizes prediction error (or compression progress) with no competing drives, no energy budget, no fear of predators, no social bonds. This is both a strength (clean mathematical formulation) and a fundamental limitation (it misses the rich context that makes biological curiosity adaptive rather than merely compulsive).
Curiosity-driven agents work impressively in simple environments. In 2D video games, mazes, and grid worlds, intrinsic motivation enables exploration and learning that random exploration cannot match. But scaling to complex, realistic environments has proven difficult:
The gap between artificial and biological curiosity remains large. A curious child and a curiosity-driven RL agent are separated by the same chasm that separates all current AI from biological intelligence: the absence of embodied context, competing drives, developmental history, and the kind of integrated motivation that arises from being a living organism with finite resources in an uncertain world.
Curiosity and intrinsic motivation address the question of why a system would learn. Continual learning (Chapter 22) addresses how it would remember. But both of these operate within the existing paradigm of neural networks on classical computers. The next chapter asks a different question entirely: could a fundamentally different computational substrate — quantum computing — change what's possible?
Next: Chapter 24 — Quantum Computing and AI. What quantum gives you, what it doesn't, and an honest assessment of where things actually stand.1 Schmidhuber, J. (1991). "A possibility for implementing curiosity and boredom in model-building neural controllers." Proc. International Conference on Simulation of Adaptive Behavior, pp. 222–227. This is one of the earliest formal proposals for curiosity-driven learning in neural networks.
2 Schmidhuber, J. (2009). "Simple Algorithmic Theory of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes." Journal of the Society of Instrument and Control Engineers 48(1):21–32.
3 Pathak, D., Agrawal, P., Efros, A.A., and Darrell, T. (2017). "Curiosity-driven Exploration by Self-Supervised Prediction." International Conference on Machine Learning (ICML).
4 Burda, Y., Edwards, H., Storkey, A., and Klimov, O. (2019). "Exploration by Random Network Distillation." International Conference on Learning Representations (ICLR).
5 Wang, R., Lehman, J., Clune, J., and Stanley, K.O. (2019). "Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions." arXiv:1901.01753.
6 Schultz, W., Dayan, P., and Montague, P.R. (1997). "A Neural Substrate of Prediction and Reward." Science 275:1593–1599.