WorldModels:Tutorial:01-WhatIsWorldModel
lastmod 2026-11-18created 2026-11-172 edits
What Is a World Model is Chapter 1 of the World Model tutorial. Starting from the intuition of a "simulator in your head", it covers what a model means in reinforcement learning (a transition model and a reward model), the difference between model-free and model-based methods, Dyna, the history of the field, ways to classify world models, and the four problems that make world models hard. This chapter also sets up the "model-based learning loop" that every later chapter repeats, and the pendulum environment used for the hands-on parts.
- Original: 01. What Is a World Model?
- Code: this chapter has no code of its own (the original includes an in-browser pendulum demo). The pendulum environment is
Pendulumincode/wm_envs.py. - Previous: Tutorial hub / Next: 02. Probability and Maximum Likelihood
A simulator in your head
People play out the outcome in their heads before actually trying something. The original gives three examples.
- Catching a ball: An outfielder predicts where a hit ball will land and runs there. Processing visual information takes time, so chasing only the ball's currently visible position would be too late.
- Driving: Knowing that braking on a wet road makes the car slide farther than usual, a driver slows down early, before ever skidding.
- Setting down a cup: You know that a cup placed half over the edge of a table will fall, without having to break one to find out.
In his 1943 book The Nature of Explanation, Kenneth Craik argued that if an organism carries a "small-scale model" of the external world and of its own possible actions in its head, it can try out various actions mentally, pick the best one, and prepare for future situations in advance. Today this is called a mental model.
A World Model is the "small game engine" in an agent's head. In a real game engine, programmers write the laws of physics into code; a world model learns those laws from experience (data) alone. Once trained, it can be used instead of the real world to ask "what happens if I do this?" many times, cheaply and safely.
- Definition
- A world model is a model trained on data to predict "what happens next if I take this action in the current situation". With such a predictor, an agent can imagine the future without actually acting, and use that imagination to plan or to learn behavior.
Agent and environment
Basic terms
The terms below are restated formally in Chapter 5.
| Term | Symbol | Meaning |
| Agent | The entity that chooses actions: a person, a robot, a program controlling a game character, and so on. | |
| Environment | The world around the agent. It changes in response to the agent's actions. | |
| State | \(s_t\) | All the information describing exactly what the world looks like at time \(t\). For the pendulum, the angle and angular velocity. |
| Observation | \(o_t\) | The information the agent can actually see. It may be the full state or only part of it (e.g. a single camera image). |
| Action | \(a_t\) | What the agent chooses. For the pendulum, the torque applied at the pivot. |
| Reward | \(r_t\) | The score received as a result of an action. The agent's goal is to maximize the sum of rewards. |
| \(\pi\) | The rule that picks an action given an observation (or state). | |
| Episode | One run from start to finish. A pendulum episode is 200 steps (10 seconds). |
The agent and the environment form a loop (Figure 1-1 in the original). The agent sends an action \(a_t\) chosen by its policy \(\pi\), the environment (the real world) changes according to the laws of physics, and the agent receives the next observation \(o_{t+1}\) and the reward \(r_t\). After this loop runs 200 times, a pendulum episode ends.
The pendulum environment
The whole tutorial uses the pendulum swing-up task. The physics is the same as Gym's Pendulum-v1.
| Term | For the pendulum |
| State \(s_t\) | Angle \(\theta\) and angular velocity \(\dot\theta\). These two numbers fully determine the future motion. \(\theta = 0\) is up (standing upright) and \(\pm\pi\) is down (hanging). |
| Observation \(o_t\) | In Chapters 6 and 7, \([\cos\theta, \sin\theta, \dot\theta]\) (the full state is visible). From Chapter 8 on, a 64×64 grayscale image (the angular velocity is not visible). |
| Action \(a_t\) | Torque \(u \in [-2, 2]\). Positive values turn the pendulum clockwise, negative values counterclockwise. |
| Reward \(r_t\) | \(-(\theta^2 + 0.1\,\dot\theta^2 + 0.001\,u^2)\). Standing still at the top gives 0 (the maximum); hanging at the bottom gives about −9.9. |
| Episode | 200 steps. Each step is 0.05 seconds, so an episode lasts 10 seconds. |
The environment constants are \(g = 10\), \(m = 1\), \(l = 1\), \(dt = 0.05\), a maximum angular velocity of 8 rad/s, and a maximum torque of 2. Pendulum.step() in wm_envs.py is this environment's true transition function \(f\) and reward function.
def step(self, action) -> tuple[np.ndarray, float, bool]:
u = float(np.clip(np.asarray(action, dtype=np.float64).reshape(-1)[0], -self.max_torque, self.max_torque))
th, thdot = self.th, self.thdot
# 보상은 '움직이기 전' 상태로 계산한다 (Gym 과 같다).
cost = angle_normalize(th) ** 2 + 0.1 * thdot**2 + 0.001 * u**2
# 물리: θ̈ = (3g / 2l)·sin θ + (3 / m l²)·u (6장에서 유도한다)
# 오일러 적분: 속도를 먼저 갱신하고, 새 속도로 각도를 갱신한다 (semi-implicit Euler).
thdot_new = thdot + (3.0 * self.g / (2.0 * self.l) * np.sin(th) + 3.0 / (self.m * self.l**2) * u) * self.dt
thdot_new = float(np.clip(thdot_new, -self.max_speed, self.max_speed))
th_new = th + thdot_new * self.dt
self.th, self.thdot = th_new, thdot_new
self.t += 1
done = self.t >= self.max_steps
return self._obs(), float(-cost), done
What swinging up the pendulum reveals
The original includes an in-browser demo that ports the same physics to JavaScript. The task is to start with the pole hanging down and get it upright within 10 seconds. Trying it by hand reveals three things.
- Pushing in one direction only does not get it up. The maximum torque of 2 is less than half the torque with which gravity pulls the pole down (at most \(mgl/2 = 5\), when the pole is horizontal). Applying +2 continuously from the bottom raises the pole only to about 49° from the bottom before it falls back.
- You have to swing it like a swing. Pushing in the direction the pole is already moving makes it rise a little higher with each back-and-forth.
- Near the top, you have to do the opposite. You must brake so it does not tip over, and then balance it. The same "torque +2" can be a good action at the bottom and a bad one at the top.
The demo's autopilot implements exactly this strategy. It defines the energy as \(E = \dot\theta^2/6 + 5\cos\theta\) (5 when at rest at the top). Near the top, where \(\vert\theta\vert \lt 0.5\), it balances with \(u = -(10\theta + 2\dot\theta)\) clipped to \([-2, 2]\); elsewhere, it applies torque 2 in the direction of motion if \(E \lt 5\) and 0 otherwise. Starting from the bottom, it gets the pole upright in about 3 seconds (62 steps), and the total reward for one episode is about −373.
A judgment like "pushing now looks costly, but it pays off a few seconds later" requires being able to picture the future. Swinging up the pendulum is a problem where greedily chasing the immediate reward fails, which makes it a good showcase for the advantages of imagining the future with a world model.
Models in reinforcement learning
In reinforcement learning, a model means exactly these two things.
- Transition model (dynamics model)
- What the next state \(s_{t+1}\) will be if action \(a_t\) is taken in the current state \(s_t\).
- Reward model
- How much reward \(r_t\) is received if action \(a_t\) is taken in the current state \(s_t\).
If the world always responds the same way (deterministic), the transition model is a single function.
\(s_{t+1} = f(s_t, a_t), \qquad r_t = R(s_t, a_t)\)
Since the world does not always respond the same way (dice, the direction the car ahead takes at an intersection), the transition model is generally written as a probability distribution.
\(s_{t+1} \sim p(s_{t+1} \mid s_t, a_t)\)
Read this as "the next state \(s_{t+1}\) is drawn from the distribution \(p\) given that the current state is \(s_t\) and the action is \(a_t\)". What comes after the vertical bar is what is already known. A deterministic world is the special case where this distribution puts all of its probability, 1, on the single point \(f(s_t, a_t)\). Conditional probability notation is covered in Chapter 2.
A world model is a learned model that imitates the true \(f\) (or \(p\)) from data, without knowing it. The learned numbers (the neural network's weights) are collected into \(\theta\), and the model is written \(f_\theta\) or \(p_\theta\).
\(\hat{s}_{t+1} = f_\theta(s_t, a_t) \approx f(s_t, a_t)\)
Learning from data: a small example
Suppose the true rule of a one-dimensional world is \(s_{t+1} = s_t + 0.5\,a_t\), but we do not know it and have only three experiences.
| Experience | \(s_t\) | \(a_t\) | \(s_{t+1}\) (observed outcome) |
| 1 | 0 | 2 | 1.0 |
| 2 | 1 | \(-2\) | 0.0 |
| 3 | 3 | 1 | 3.5 |
Fix the form of the model as \(f_\theta(s, a) = s + \theta\,a\) and learn just one number, \(\theta\). The error is measured as the mean of the squared errors (mean squared error, MSE). Squaring keeps negative and positive errors from canceling each other out.
\(\begin{align} \theta = 1:\quad & \hat{s}^{(1)} = 0 + 1\cdot 2 = 2, \quad \hat{s}^{(2)} = 1 + 1\cdot(-2) = -1, \quad \hat{s}^{(3)} = 3 + 1\cdot 1 = 4 \\ & \hat{s} - s_{t+1}:\quad 2 - 1 = 1, \quad -1 - 0 = -1, \quad 4 - 3.5 = 0.5 \\ & \text{MSE}(\theta{=}1) = \frac{1^2 + (-1)^2 + 0.5^2}{3} = \frac{1 + 1 + 0.25}{3} = 0.75 \\ \theta = 0.5:\quad & \text{MSE}(\theta{=}0.5) = \frac{(1-1)^2 + (0-0)^2 + (3.5-3.5)^2}{3} = 0 \end{align}\)
The value that minimizes the error is \(\theta^\ast = 0.5\), which gives \(f_{\theta^\ast}(s, a) = s + 0.5\,a\), exactly the true rule. This is what "learning a model from data" means.
A real world model has tens of thousands to millions of values in \(\theta\) (the neural network's weights), and instead of trying values by hand, it reduces the error with gradient descent (Chapter 4). Chapter 2 shows why the error is measured as a square: it is the "maximum likelihood under a Gaussian distribution".
When only images are visible: world models in latent space
If the pendulum is seen only through a camera image instead of \((\theta, \dot\theta)\), a 64×64 image is 4096 numbers. Predicting all 4096 pixels of the next image is hard and wasteful, since the only information in the image is "what angle the pole is at". So a world model that works from images is usually split into three parts.
- Encoder: compresses the image \(o_t\) into a small vector \(z_t\) (a latent variable). Covered by the VAE in Chapter 8.
- Latent dynamics: predicts what comes next in the small vector space. \(z_{t+1} \sim p_\theta(z_{t+1} \mid z_t, a_t, \text{기억})\) (the last argument is memory). Covered in Chapter 9 and Chapter 11.
- Decoder: when needed, expands the small vector back into an image. Covered in Chapter 8.
This way, imagination runs quickly inside the small vector space. Ha's World Models (Chapter 10), PlaNet (Chapter 11), and Dreamer (Chapter 12) all use this structure.
Model-free and model-based
Reinforcement learning algorithms fall broadly into two groups depending on whether they use a world model.
- Model-free RL: does not learn how the world works; it learns directly, by trial and error, that "this action was good/bad in this situation". Examples: DQN, PPO, SAC. It is like learning to ride a bicycle by falling over.
- Model-based RL: first learns the rules of the world (a world model), then uses that model to plan or to practice in imagination. Examples: PETS, PlaNet, Dreamer, MuZero. It is like a chess player reading several moves ahead before making one.
| Aspect | model-free | model-based (World Model) |
| Sample efficiency (how sparingly real experience is used) | Low. Requires a great deal of interaction with the real environment (millions of steps is common). | High. Learns a model from the experience collected once, then generates and uses plenty of imagined experience. |
| Computation | Low. Only the policy and value networks are trained. | High. The model must also be trained, and planning or imagination must run inside the model. |
| Asymptotic performance (final skill with unlimited data) | Traditionally high. Since only real experience is used, model error has no effect. | Can lose out to the extent the model is wrong. Recent methods (DreamerV3 and others) match or beat model-free methods on many tasks. |
| Sources of error | Bias and variance in value estimates, insufficient exploration. | The causes on the left, plus model error (model bias). A plan made inside a wrong model fails in reality. |
| Reuse | When the goal (reward) changes, learning often has to start over from scratch. | The rules of the world do not depend on the goal, so the same model can plan for a new goal by swapping only the reward. |
| In this tutorial | Basics only, in Chapter 5 | All of Chapters 6–13 |
The difference in sample efficiency is large. The PETS paper (Chua et al., 2018) reported reaching similar performance on the HalfCheetah running task with about 8 times fewer samples than the model-free method SAC and about 125 times fewer than PPO. In the real world, where moving a robot arm once takes several seconds and a fall can break it, this difference is decisive.
The two approaches are not opposed; they are often used together. Dreamer runs actor-critic (a typical model-free method) on imagined experience generated by a world model. The original version of this "mix" is Dyna, described below.
Five things a world model offers
- Sample efficiency: Real experience is expensive. Robots wear out, accidents with self-driving cars cannot be undone, and drug experiments take months. A world model lets an agent learn as much as possible from each real experience.
- Planning: Before acting, run several candidate actions inside the model and pick the best one. MPC and CEM in Chapter 7, PlaNet in Chapter 11.
- Learning in imagination: Instead of planning every time, train a policy in advance inside the "dream" the model generates, then use that policy directly in the real environment. World Models in Chapter 10, Dreamer in Chapter 12.
- Simulators and data generation: A good enough world model can itself serve as a game engine or a driving simulator. It can generate dangerous situations, such as the moments just before an accident, to train or test other AI systems. GameNGen, Genie, and Cosmos in Chapter 13.
- Understanding and reuse: Predicting the future well requires grasping the structure of the world (what the objects are and how they move), so the representations a world model learns are useful for other tasks too. Since the rules of the world stay the same when the goal changes, the model can be reused.
Dyna: mixing real and imagined experience
An architecture proposed by Richard Sutton in 1990–1991.
- Act in the real environment to obtain an experience \((s, a, r, s')\) (where \(s'\) is the next state).
- Use this real experience to update the policy and value directly (as in model-free methods), and at the same time update the model.
- Ask the model "what if this action is taken in this state?" to generate several simulated experiences, and update the policy and value with those simulated experiences in the same way.
The components and flow are as follows (Figure 1-2 in the original). The action chosen by the policy/value goes to the real environment (expensive and slow); ① the real experience trains the policy/value directly, and ② the same experience trains the world model (cheap and fast). ③ The model generates simulated experiences \((s, a, r, s')\) through imagination, and ④ those simulated experiences train the policy/value several more times.
The skeleton (pseudocode) that uses imagined experience \(n\) times for each real experience is shown below.
for step in range(total_steps):
a = policy(s) # 1. 행동을 고른다
s_next, r = real_env.step(a) # 2. 진짜 세상에서 해 본다 (비싸다)
update_policy(s, a, r, s_next) # 3. 진짜 경험으로 배운다
model.update(s, a, r, s_next) # 4. 세상의 규칙도 배운다
for _ in range(n): # 5. 상상 경험으로 n 번 더 배운다 (싸다)
s_im, a_im = sample_seen_state_action()
s_im_next, r_im = model.predict(s_im, a_im)
update_policy(s_im, a_im, r_im, s_im_next)
s = s_next
- With \(n = 0\) it is the same as model-free learning; the larger \(n\) is, the more is learned from the same real experience.
- If the model is wrong, the imagined experience is wrong too and actually does harm. This trade-off is a concern throughout model-based reinforcement learning.
- MBPO (2019) revived Dyna with neural network ensembles, and Dreamer can be seen as an extreme form of Dyna that learns its policy from imagined experience alone.
History
The idea of "learning a model that predicts the world and using it to act" is old, but handling high-dimensional images became possible thanks to deep learning.
| Year | Work | Key idea | Tutorial |
| 1943 | Craik, The Nature of Explanation | The hypothesis that organisms keep a small-scale model of the world in their heads and test things on it in advance | Ch. 1 |
| 1990 | Schmidhuber, "Making the World Differentiable" | Model the environment with a recurrent neural network and train a controller by differentiating through that model | Ch. 9 and 10 (same idea) |
| 1990–1991 | Sutton, Dyna | Learning and planning from a mix of real experience and simulated experience generated by a model | Ch. 1 |
| 2011 | Deisenroth & Rasmussen, PILCO | A dynamics model that captures uncertainty using Gaussian processes; learns control from very few trials | Ch. 7 (the idea of uncertainty) |
| 2018 | Ha & Schmidhuber, "World Models" | VAE (eyes) + MDN-RNN (memory) + a small controller; trains a policy inside the model's dream | Ch. 8, 9, and 10 |
| 2018 | Chua et al., PETS | High sample efficiency from an ensemble of probabilistic neural networks + CEM planning | Ch. 7 |
| 2019 | Hafner et al., PlaNet | RSSM: latent dynamics combining deterministic and stochastic paths; learns from images and plans in latent space | Ch. 11 |
| 2019–20 | Schrittwieser et al., MuZero | A model that predicts only reward, value, and policy without reconstructing observations, plus tree search; Go, chess, shogi, Atari | Ch. 13 |
| 2020 | Hafner et al., Dreamer | Actor-critic training in imagination within latent space | Ch. 12 |
| 2021 | Hafner et al., DreamerV2 | Categorical (discrete) latent variables; a policy learned inside the world model reaches human level on 55 Atari games | Ch. 12 |
| 2021 | Ye et al., EfficientZero | An improved MuZero; average performance above human scores on the Atari 100k benchmark, which allows only 100,000 steps per game | Ch. 13 |
| 2022 | LeCun, "A Path Towards Autonomous Machine Intelligence" | A position paper proposing the JEPA architecture, which predicts "representations" rather than pixels | Ch. 13 |
| 2022 and 2024 | Hansen et al., TD-MPC and TD-MPC2 | World models for continuous control that plan with MPC in latent space without a decoder | Ch. 13 |
| 2023 (Nature 2025) | Hafner et al., DreamerV3 | Learns over 150 tasks with a single fixed configuration; collects diamonds in Minecraft without human data | Ch. 12 |
| 2023 | Micheli et al., IRIS | A world model that turns images into discrete tokens and predicts the next token with a Transformer | Ch. 13 |
| 2024 | OpenAI, Sora | A diffusion Transformer that uses spacetime patches as tokens; argues that video generation models can become simulators of the physical world | Ch. 13 |
| 2024 | Bruce et al., Genie | Learns "latent actions" from game videos without action labels and generates controllable 2D worlds | Ch. 13 |
| 2024 | Alonso et al., DIAMOND | Trains Atari agents inside a world model built from a diffusion model | Ch. 13 |
| 2024 | Valevski et al., GameNGen | A diffusion model generates frames of the game DOOM in real time; a neural network replaces the game engine | Ch. 13 |
| 2024 | Google DeepMind, Genie 2 | Generates a 3D world, controllable with keyboard and mouse, from a single image | Ch. 13 |
| 2025 | NVIDIA, Cosmos | Releases a "world foundation model" platform for physical AI such as robotics and autonomous driving | Ch. 13 |
| 2025 | Meta, V-JEPA 2 | A world model, trained with self-supervised learning on video, that predicts "representations" rather than images; used for robot planning | Ch. 13 |
| 2025 | Google DeepMind, Genie 3 | Generates a 3D world described in a text prompt in real time and lets the user move around inside it | Ch. 13 |
| 2025 | Hafner et al., Dreamer 4 | Reported obtaining diamonds in Minecraft from offline data alone, without interacting with the environment | Ch. 13 |
| 2026 | Google DeepMind, Genie 4 | Released as a research preview; also generates sound matching the world in real time | Ch. 13 |
The progression can be summed up in three stages: models that predict a small state (1990s–2010s), models that predict in a latent space compressed from images (2018–2021), and general-purpose video and 3D world models trained on huge datasets (2023 onward). The tutorial follows this order.
Ways to classify world models
Quite different things are grouped under the name "world model". Dividing them along five criteria makes clear where each piece of work fits.
Criterion 1. What does it take as input?
- State-based: takes already-structured numbers such as position and velocity. The model is small and easy to train. Used for things like robot joint control (Chapters 6 and 7).
- Observation (image)-based: takes raw data such as camera images. Useful information must first be extracted from the images (Chapters 8–12).
Criterion 2. What is it trained to predict?
| Type | Training objective | Pros | Cons | Examples |
| Reconstruction | Reconstruct or generate the future observations (images) themselves | Rich training signal (every pixel is a target). Results can be checked by eye. | Spends capacity on details irrelevant to the task (background patterns, etc.). | Ha's World Models, PlaNet, Dreamer, video generation models |
| Value-equivalent | Only needs to predict future reward, value, and policy (observations are not reconstructed). | Learns only what the task needs. | Weak training signal when rewards are sparse. Hard to reuse for other tasks. | MuZero, (partly) TD-MPC |
| Representation prediction | Predict the "representation (summary vector)" of future observations | Discards unpredictable details and predicts only meaningful structure. | Needs a mechanism to prevent "collapse", where everything maps to the same vector. | JEPA family (V-JEPA 2) |
Criterion 3. Deterministic or stochastic?
- Deterministic models: always output a single prediction for the same input. Simple, but when the future can branch several ways, they predict the average (see "Stochasticity and multimodality" below). Chapter 6.
- Stochastic models: predict a distribution over the future, so several possibilities can be sampled. Chapters 7, 9, 11, and 12.
Criterion 4. What is the model used for?
- Planning: searches the future inside the model at every action. PETS, PlaNet, MuZero, TD-MPC (Chapters 7 and 11).
- Policy learning in imagination: trains a policy inside the model in advance. World Models, Dreamer, IRIS (Chapters 10 and 12).
- Simulation and generation: the model itself serves as a world for people or other AI systems to use. GameNGen, Genie, Cosmos (Chapter 13).
Criterion 5. What neural network architecture?
The field has progressed from multilayer perceptrons (MLP, Chapter 6) to recurrent neural networks (RNN and RSSM, Chapters 9 and 11) to Transformers and diffusion models (Chapter 13). Whatever the architecture, the job is the same: "look at the past and the actions, and predict the future".
Four things that make world models hard
The later chapters of the tutorial tackle the four problems below one at a time.
Compounding error
Even if a one-step prediction is only slightly wrong, feeding that wrong prediction back in as input to predict the next step makes the errors pile up. This is compounding error.
Let \(e_k\) be the size of the error after \(k\) steps and \(\varepsilon\) the new error the model introduces at each step, and assume that existing error is amplified by a factor of \(L\) by the dynamics of the world. \(L \gt 1\) describes a world where small differences keep growing, as when the pendulum is standing at the top.
\(e_1 = \varepsilon, \qquad e_{k+1} = L\, e_k + \varepsilon\)
Solving the recurrence gives the sum of a geometric series.
\(\begin{align} e_2 &= L\varepsilon + \varepsilon = \varepsilon(1 + L) \\ e_3 &= L\varepsilon(1+L) + \varepsilon = \varepsilon(1 + L + L^2) \\ e_k &= \varepsilon\,(1 + L + L^2 + \cdots + L^{k-1}) = \varepsilon \sum_{j=0}^{k-1} L^j \end{align}\)
Let the sum be \(S = \sum_{j=0}^{k-1} L^j\). Then \(LS = L + L^2 + \cdots + L^k\), and subtracting the two cancels all the middle terms, leaving \(LS - S = L^k - 1\). Therefore \(S = (L^k - 1)/(L - 1)\) (\(L \ne 1\)), and
\(e_k = \varepsilon\,\frac{L^k - 1}{L - 1}\)
- If \(L \gt 1\), the \(L^k\) term makes the error grow exponentially.
- If \(L = 1\), then \(S = k\), so \(e_k = k\varepsilon\) grows linearly.
The values for \(\varepsilon = 0.01\) and \(L = 1.2\) are as follows.
| \(k\) | \(L^k\) | \(e_k\) |
| 10 | \(1.2^{10} \approx 6.19\) | \(0.01 \times (6.19 - 1)/0.2 \approx 0.26\) |
| 20 | \(1.2^{20} \approx 38.34\) | \(0.01 \times (38.34 - 1)/0.2 \approx 1.87\) |
An error of 0.01 at one step becomes almost 190 times larger after 20 steps. Chapter 6 confirms this with an actually trained pendulum model, and Chapter 7 avoids it by imagining only a short way ahead and replanning often (MPC).
Model exploitation
Planning and policy learning look for actions that "maximize reward inside the model". If the model is wrong somewhere and gives more reward than reality would, optimization will find exactly that wrong spot. This is model exploitation. It resembles finding a bug in a game to rack up points, except that the bug exists only in the model and not in the real world, so the behavior fails when actually tried. The World Models paper in Chapter 10 called this "cheating" in the dream.
There are three remedies.
- Have the model also predict how confident it is, and avoid uncertain regions (ensembles in Chapter 7).
- Imagine only a short way ahead.
- Keep collecting real data to correct the model.
Partial observability
A single image cannot tell which way or how fast the pendulum is currently turning, just as a still photo cannot tell whether a ball is going up or coming down. A situation where observations show only part of the state is called partial observability, and inferring the state requires remembering several frames. Covered by POMDPs in Chapter 5, RNNs in Chapter 9, and RSSM in Chapter 11.
Stochasticity and multimodality
Even in the same situation, the outcome can branch several ways. Suppose a coin flip yields heads (+1) or tails (−1) with equal probability. If the outcome \(y\) is predicted with a single number \(c\) chosen to minimize the mean squared error \(\mathbb{E}[(y - c)^2]\), the prediction becomes the mean.
\(\begin{align} J(c) &= \mathbb{E}[(y - c)^2] = \mathbb{E}[y^2 - 2cy + c^2] \\ &= \mathbb{E}[y^2] - 2c\,\mathbb{E}[y] + c^2 \\ \frac{dJ}{dc} &= -2\,\mathbb{E}[y] + 2c = 0 \;\Rightarrow\; c^\ast = \mathbb{E}[y] \end{align}\)
The second line uses the linearity of expectation (proved in Chapter 2), and \(\mathbb{E}[y^2]\) is a constant independent of \(c\), so its derivative is 0.
The mean of the coin is \(0.5 \times (+1) + 0.5 \times (-1) = 0\). A deterministic model trained with squared error predicts 0, an outcome that never happens. In image terms, when a car at a fork goes either left or right, the model draws a blurry ghost car that is half of each image overlaid. A distribution whose possible outcomes split into several peaks is called a multimodal distribution.
The remedy is to predict the future as a distribution rather than a single number. Gaussian outputs in Chapter 7, mixture density networks in Chapter 9, stochastic latent variables in Chapters 11 and 12, and diffusion models in Chapter 13 all take this approach.
The model-based learning loop
Every world model agent built in the tutorial repeats the following four steps, though the details differ (Figure 1-3 in the original).
- Collect experience: add \((o, a, r)\) to the dataset \(D\).
- Train the world model: reduce the prediction error on \(D\).
- Plan or learn in imagination: roll out the future inside the model.
- Act for real: act in the environment with the improved policy.
Newly collected experience corrects the model, the corrected model yields better actions, and better actions collect more useful experience.
D = collect_episodes(env, policy=None) # 처음엔 무작위 행동으로 경험을 모은다
for iteration in range(num_iterations):
world_model.train(D) # ② 모은 경험으로 세상의 규칙을 배운다
policy = plan_or_learn_in_imagination( # ③ 모델 안에서 계획하거나 정책을 연습한다
world_model)
new_episodes = collect_episodes(env, policy) # ④ 실제 환경에서 행동해 본다
D = D + new_episodes # ① 새 경험을 데이터에 더한다
There are three reasons to repeat the loop rather than do everything in one pass.
- Data collected with random actions at the start contains almost no "pole standing upright" scenes. The model is likely to be wrong in such regions.
- Acting with a slightly better policy collects data from the places the policy actually visits (e.g. near the top), and the model becomes accurate there.
- The true outcomes in regions where the model was exploited (where the model said things were good but they were actually bad) enter the data, and the model is corrected.
Each chapter implements step ③ differently.
| Chapter | World Model | ③ Planning / learning in imagination |
| 6–7 | MLP over states (deterministic → probabilistic ensemble) | Search over action sequences inside the model at every action (random shooting, CEM, MPC) |
| 8–10 | VAE + MDN-RNN | Train a small controller in the dream with an evolution strategy (CMA-ES) |
| 11 | RSSM | CEM planning in latent space (PlaNet) |
| 12 | RSSM | Actor-critic in imagination (Dreamer) |
What you build in this tutorial
Using a single pendulum throughout, the tutorial implements world models step by step. Before that, Chapters 2–5 cover the necessary tools (probability, information theory, deep learning, reinforcement learning).
- Chapter 6: a neural network that takes 3 numbers and predicts the next 3. You will see that it gets single steps right but drifts off when imagining far ahead (compounding error).
- Chapter 7: search over action sequences inside that model to swing up the real pendulum. Making the model aware of its own uncertainty reduces model exploitation.
- Chapters 8, 9, and 10: compress images into 32-dimensional vectors (VAE), predict the future of those vectors (MDN-RNN), and train a controller inside that "dream".
- Chapters 11 and 12: implement PlaNet and Dreamer from scratch to swing up the pendulum from images alone.
Summary
- A world model is a model trained on data to predict "state and action → next state and reward"; in other words, a simulator in the agent's head.
- A "model" in reinforcement learning means two things: a transition model \(p(s_{t+1} \mid s_t, a_t)\) and a reward model \(r_t = R(s_t, a_t)\).
- Model-based methods are sample-efficient and easy to reuse, at the cost of more computation and the risk of model error. Dyna is the original architecture for mixing real and imagined experience.
- World models can be classified by input (state/image), training objective (reconstruction/value equivalence/representation prediction), stochasticity, use (planning/learning in imagination/simulation), and architecture.
- There are four difficulties: compounding error (\(e_k = \varepsilon (L^k - 1)/(L - 1)\)), model exploitation, partial observability, and stochasticity and multimodality (squared-error predictions go to the mean).
- The agents in every chapter repeat the loop "collect experience → train the model → plan or learn in imagination → act for real".
Review questions
- 1. Which of the following is NOT an advantage of model-based reinforcement learning? (a) It saves real experience (b) The model can be reused when the reward changes (c) It requires little computation (d) It can plan before acting
- (c). Because it must train a model and also plan or imagine inside it, model-based RL usually needs more computation than model-free RL. In exchange, it saves expensive real experience.
- 2. Read \(p(s_{t+1} \mid s_t, a_t)\) as a sentence, and describe the shape of this distribution in a deterministic world.
- "The probability (density) that the next state is \(s_{t+1}\), given that the current state is \(s_t\) and action \(a_t\) was taken." In a deterministic world, it is a distribution with all of its probability, 1, concentrated on the single point \(f(s_t, a_t)\).
- 3. In the compounding error model, what is \(e_{10}\) when \(\varepsilon = 0.01\) and \(L = 1.1\)? (\(1.1^{10} \approx 2.594\))
- \(e_{10} = 0.01 \times (2.594 - 1)/0.1 = 0.01 \times 15.94 \approx 0.159\), about 16 times the one-step error. Lowering \(L\) only slightly, from 1.2 to 1.1, cuts \(e_{10}\) substantially, from about 0.26 to 0.16.
- 4. If the next outcome is 10 with probability 0.8 and 0 with probability 0.2, what does a deterministic model trained with squared error predict, and what is the problem?
- It predicts the mean, \(0.8 \times 10 + 0.2 \times 0 = 8\). In reality only 10 or 0 ever occurs, never 8. Summarizing a multimodal distribution with a single number leads the model to imagine a future that cannot happen.
- 5. In Dyna, what does setting the number of imagination steps \(n\) to 0 give you, and what is the risk of making it very large?
- With \(n = 0\) it is equivalent to model-free learning that does not use the model. With a very large \(n\), the policy is trained almost entirely on imagined experience, so the risk of learning faulty experience from regions where the model is wrong (model exploitation, model bias) grows.
See also
- World Models:Tutorial
- World Models
- Reinforcement learning
- Markov decision process
- Recurrent World Models Facilitate Policy Evolution
- Genie (world model)
- GameNGen
- Sora
- Physical AI