WorldModels:Tutorial:14-Appendix
lastmod 2026-11-18created 2026-11-172 edits
Appendix: Debugging, Glossary, and Next Steps is chapter 14 of the World Model tutorial. It collects reference material to consult when you are building a World Model yourself and the loss turns into NaN, reconstructions come out blurry, or the agent does well in imagination but poorly in reality. It covers a debugging checklist and remedies by symptom, derivations of numerical stability formulas, hyperparameters from the original papers and from the tutorial, key equations by chapter, a Korean–English glossary, a reading order for the papers, public implementations, frequently asked questions, and follow-up projects.
- Original: 14. Appendix: Debugging, Glossary, Next Steps
- Code: there is no dedicated file, only the short check snippets in the text. The environment comes from
code/wm_envs.py. - Previous: 13. The Modern World Model Landscape / Next: none (final chapter)
Checklist for the first 30 minutes
A World Model has many parts (encoder, dynamics, decoder, reward, policy), and reinforcement learning results have high variance, so bugs are hard to locate. When something looks wrong, go through the steps below before touching any hyperparameters. Most bugs are caught in the first three steps.
- Look at the data. Save a few observation images and open them, and print the minimum, maximum, and mean of the values. Check whether images are in 0–255 or 0–1, and whether the action range is correct (the environment uses [−2, 2]; the agents in chapters 11 and 12 output [−1, 1]).
- Check shapes. Write the expected shape of each tensor in a comment and verify it with
assert. Pay particular attention to the order of the time axis \(T\) and the batch axis \(B\), and to how observations (\(T+1\) of them) pair with actions (\(T\) of them). - Overfit a tiny dataset. Training repeatedly on a single batch (e.g., 4 sequences) should drive the loss close to 0. If it does not, there is a bug in the model or the loss code.
- Log each loss term separately. If you only watch the sum of the reconstruction, reward, KL, actor, and critic losses, you cannot tell which term is the problem.
- Compare with baselines. Check that the model beats simple baselines such as a model that predicts "next state = current state" or the return of a random policy (the baselines from chapters 5 and 6).
- Log gradient magnitudes. A gradient norm of 0 means the gradient is cut off (a
.detach()mistake); a sudden jump into the thousands means it is exploding. - Fix the seed, then run several seeds. A good result from one seed may be luck. Look at the mean and range over 3 – 5 seeds.
- Only then change hyperparameters. One at a time, starting with the learning rate.
Here are the checks above as code. Put them in the training loop to catch problems early.
import random
import numpy as np
import torch
def set_seed(seed):
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed) # CPU · GPU 모두
def check(name, x, shape=None):
"""모양과 유한성(NaN · inf 없음)을 확인한다."""
if shape is not None:
assert tuple(x.shape) == tuple(shape), f"{name}: {tuple(x.shape)} != {shape}"
assert torch.isfinite(x).all(), f"{name} 에 NaN 또는 inf 가 있습니다"
def grad_norm(model):
"""모든 매개변수 기울기를 한 벡터로 이었을 때의 길이."""
total = 0.0
for p in model.parameters():
if p.grad is not None:
total += p.grad.detach().pow(2).sum().item()
return total ** 0.5
# 학습 루프 안에서
# check("obs", obs, (B, L, 64, 64)) # 11장의 배치: 길이 L, 채널 축 없음
# loss.backward()
# print("grad norm", grad_norm(model))
# torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=100.0)
# NaN 이 어디서 처음 생기는지 찾을 때만 켠다 (느려진다)
# torch.autograd.set_detect_anomaly(True)
The most common silent bug: broadcasting
If the prediction pred has shape (B, 1) and the target target has shape (B,), then (pred - target) ** 2 silently becomes a (B, B) matrix. Every prediction is compared with every target, so the model learns to predict only the mean. The loss still appears to go down, which makes the bug hard to notice. Check that both tensors have the same shape right before computing the loss.
Symptoms, causes, and remedies
This table collects common symptoms, grouped by component. The numbers under "Chapters" are the tutorial chapters that contain detailed explanations and derivations.
Data and shapes
| Symptom | Common causes | Remedy | Chapters |
| The loss is absurdly large or small from the start | Inputs were not normalized, or were normalized twice. Images were fed in raw as 0–255 | Print the input range. Images: /255 (−0.5); numeric states: standardization | 06 · 08 |
| The model always predicts roughly the same value (the mean) | Broadcasting turns the loss into (B, B). Inputs and targets are misaligned | Check shapes right before the loss; cross-check a few data points by hand | 04 |
| Predictions are shifted by one step | There are \(T{+}1\) observations but only \(T\) actions and rewards, and the indices are paired incorrectly | Draw out the pairing \((o_t, a_t) \to o_{t+1}\), \(r_t\) (see the description of | 06 · 11 |
| Shape error in a convolutional layer | Images are (B, H, W, C) but PyTorch expects (B, C, H, W) | | 04 · 08 |
| Predictions jump near angles of ±π | The angle \(\theta\) is used directly as input and output, so \(\pi\) and \(-\pi\) become numbers far apart | Represent it as \((\cos\theta, \sin\theta)\) | 06 |
Numerical stability
| Symptom | Common causes | Remedy | Chapters |
| The loss becomes NaN or inf | \(\log 0\), \(e^{x}\) overflow, zero standard deviation, learning rate too large | The stable formulas ( | 02 · 09 |
| The Gaussian NLL keeps decreasing without bound (toward −∞) | The predicted standard deviation goes to \(\sigma \to 0\), overfitting a few points | \(\sigma = \operatorname{softplus}(x) + \sigma_{\min}\); bound the range of \(\log\sigma^2\) | 07 · 11 |
| Sudden NaN while training the MDN | The \(\log\sum\exp\) over mixture weights is computed naively and overflows | Use | 09 |
| BCE is inf | Applying sigmoid first and then taking the \(\log\), which gives \(\log 0\) | | 08 |
Training dynamics
| Symptom | Common causes | Remedy | Chapters |
| The loss does not decrease at all | Learning rate too small, gradient cut off ( | Small-data overfitting test, log the gradient norm, check | 04 |
| The loss spikes and then diverges | Learning rate too large for the curvature (\(\eta \gt 2/a\)), exploding gradients in the RNN | Cut the learning rate to 1/3 at a time, gradient clipping | 04 · 09 |
| Training loss falls but results on new data get worse | Overfitting (too little data) | Collect more data, early stopping on validation loss, weight decay, a smaller model | 06 |
| Results vary widely between seeds | The high variance of reinforcement learning, luck in the initial data | Judge by the mean and range over several seeds; use more evaluation episodes | 05 |
Latent variable models (VAE, RSSM)
| Symptom | Common causes | Remedy | Chapters |
| KL sticks to 0 and reconstructions are always the same “average image” | Posterior collapse: the decoder ignores the latent; the KL weight is too large | Lower β, ramp the KL weight up slowly from 0 (warm-up), free nats / free bits | 08 · 11 · 12 |
| Reconstructions are blurry | MSE fits the average of the plausible images, insufficient decoder capacity, β too large | Enlarge the decoder, tune β, use a model that can represent several outcomes (MDN, stochastic latents, diffusion) | 08 · 09 · 13 |
| Reconstructions are good but imagined (prior) rollouts are a mess | Only the posterior is learned well; the transition (prior) does not keep up | Inspect open-loop imagined images, KL balancing, train longer, latent overshooting | 11 · 12 |
| Small objects (the pendulum rod) disappear from reconstructions | Most of the image is background, so the object contributes little to the loss | Increase the resolution and decoder capacity; always inspect output images by eye | 08 |
| KL stays large and does not decrease | The prior cannot catch up with the posterior; the standard deviation floor is too small | KL balancing (more weight on the prior side), adjust the minimum standard deviation | 12 |
Prediction, planning, and policy
| Symptom | Common causes | Remedy | Chapters |
| One-step predictions are good but multi-step predictions diverge | Compounding error (with a Lipschitz constant \(L \gt 1\) the error grows exponentially) | Train with a multi-step loss, predict the change \(\Delta o\), ensembles, a shorter imagination horizon | 06 |
| Imagined returns are high but real returns are low | Model exploitation: the policy exploits the model's errors | Ensemble uncertainty penalty, shorter imagination, collect more data, raise the dream temperature | 10 · 12 |
| MPC is too slow | The model runs (number of candidates × imagination length × iterations) times | Evaluate candidates as one batch on the GPU, use fewer candidates, start from the previous solution shifted by one step (warm start), shorten the horizon with a policy prior or value function | 07 · 13 |
| The actor's actions are always at ±max | tanh saturation; entropy shrinks too quickly | Increase the entropy bonus; check the action range and scale | 12 |
| The critic loss keeps growing | Large reward magnitudes; a constantly moving target | Scale rewards down (×0.1 in chapter 11), a slow (target) critic, symlog, two-hot, return normalization | 11 · 12 |
| Settings must be retuned for every task | A design that is sensitive to the scale of rewards and returns | DreamerV3's symlog and return normalization (percentiles) | 12 |
Numerical stability formulas that prevent NaN
float32 cannot hold numbers larger than about \(3.4 \times 10^{38}\); they become inf. Since \(e^{89} \approx 4.5 \times 10^{38}\), exponentials overflow easily. Below are derivations of the formulas used internally by the stable functions in PyTorch.
Stable softplus
With the definition \(\operatorname{softplus}(x) := \log(1 + e^{x})\), a large \(x\) (e.g., \(x = 1000\)) makes \(e^x\) overflow to inf. Split into cases and factor out \(e^x\).
\(\begin{align} x \ge 0: \quad \log(1 + e^{x}) &= \log\big(e^{x}(e^{-x} + 1)\big) = x + \log(1 + e^{-x}) \\ x \lt 0: \quad \log(1 + e^{x}) &= 0 + \log(1 + e^{-\lvert x\rvert}) \end{align}\)
Combining the two cases into a single expression gives:
\(\operatorname{softplus}(x) = \max(x, 0) + \log\big(1 + e^{-\lvert x\rvert}\big)\)
The exponent is always 0 or less, so nothing overflows for any \(x\). For \(x = 1000\), it computes exactly \(1000 + \log(1 + e^{-1000}) = 1000\).
Binary cross-entropy from logits
When a network outputs a probability \(p = \sigma(x)\) as a logit \(x\), as in the Bernoulli decoder of chapter 8, computing \(\sigma(x)\) first and then taking the \(\log\) can round \(\sigma(x)\) to 0 and give \(\log 0 = -\infty\). The formula that works directly from the logit is derived in the following steps.
\(\begin{align} \log \sigma(x) &= \log\frac{1}{1 + e^{-x}} = -\log(1 + e^{-x}) = -\operatorname{softplus}(-x) \\ \log\big(1 - \sigma(x)\big) &= \log\frac{e^{-x}}{1 + e^{-x}} = \log\frac{1}{e^{x} + 1} = -\operatorname{softplus}(x) \\ \operatorname{softplus}(-x) &= \log(1 + e^{-x}) = \log\big(e^{-x}(e^{x} + 1)\big) = -x + \operatorname{softplus}(x) \end{align}\)
Substituting the first two results into the BCE definition from chapter 2, the signs cancel.
\(\text{BCE} = -\big[y \log\sigma(x) + (1 - y)\log(1 - \sigma(x))\big] = y\,\operatorname{softplus}(-x) + (1 - y)\,\operatorname{softplus}(x)\)
Substituting the third line and simplifying gives the result.
\(\text{BCE}(x, y) = y\big(\operatorname{softplus}(x) - x\big) + (1 - y)\operatorname{softplus}(x) = \operatorname{softplus}(x) - y\,x\)
- There is nowhere for \(\log 0\) to appear, and because softplus is computed with the stable formula above, the result is finite for any logit. PyTorch's
binary_cross_entropy_with_logitscomputes it this way. - Example: with \(x = 50\) and \(y = 0\), \(\sigma(50)\) is exactly 1 in float32, so \(\log(1 - 1) = -\infty\), but this formula gives \(\operatorname{softplus}(50) - 0 = 50\).
Gradient clipping
clip_grad_norm_(params, max_norm=c) multiplies the gradient vector \(g\) (the gradients of all parameters concatenated into one vector) by a single positive number.
\(g' = g \cdot \min\!\Big(1,\ \frac{c}{\lVert g \rVert}\Big)\)
- If \(\lVert g \rVert \le c\), then \(c/\lVert g\rVert \ge 1\), so \(\min(\cdot) = 1\) and \(g' = g\).
- If \(\lVert g \rVert \gt c\), then \(\lVert g' \rVert = \frac{c}{\lVert g \rVert}\,\lVert g \rVert = c\) (multiplying by a positive \(k\) scales the length by \(k\)).
\(\lVert g' \rVert = \min(\lVert g \rVert,\ c), \qquad \frac{g'}{\lVert g' \rVert} = \frac{g}{\lVert g \rVert}\)
Because the gradient is multiplied by a positive number, its direction is unchanged; only its length is capped at \(c\). Example: with \(g = (30, 40)\) and \(c = 10\), \(\lVert g\rVert = 50\) and \(g' = (30, 40) \times \frac{10}{50} = (6, 8)\). The descent direction stays correct; only steps that jump too far are shortened.
A standard deviation floor and the NLL floor
The 1D Gaussian NLL is the log density derived in chapter 2 with its sign flipped.
\(\text{NLL} = \frac{(y - \mu)^2}{2\sigma^2} + \log\sigma + \frac{1}{2}\log(2\pi)\)
The first term is nonnegative and equals 0 when the prediction is exact (\(y = \mu\)), so \(\text{NLL} \ge \log\sigma + \frac{1}{2}\log(2\pi)\). Since \(\log\) is an increasing function, a lower bound on \(\sigma\) gives the NLL a floor.
\(\sigma \ge \sigma_{\min} \;\Rightarrow\; \text{NLL} \ge \log\sigma_{\min} + \frac{1}{2}\log(2\pi)\)
- Without the lower bound, the model can push \(\sigma \to 0\) at points it predicts correctly, sending \(\log\sigma \to -\infty\) and reducing the loss without bound.
- With \(\sigma_{\min} = 0.1\), the floor is \(-2.303 + 0.919 = -1.384\).
-
min_logvarin chapter 7 andlog_sigma.clamp(-7, 2)in chapter 9 serve this purpose.min_std = 0.1in chapter 11 applies the same idea to the \(\sigma\) of the latent distribution.
Hyperparameter reference
Values from the original papers worked well at those papers' environments and scale, so treat them as starting points. Values can differ between versions of the same paper and its official code; if you need an exact reproduction, check the paper's appendix and the configuration files in the official code.
Values from the original papers
| Method | Model | Training | Planning / behavior learning | Data collection |
| PETS | Ensemble of 5 probabilistic networks (3 fully connected layers, 500 units per layer (250 for HalfCheetah), Swish) | Gaussian NLL, upper and lower bounds on \(\log\sigma\) | CEM-MPC, trajectory sampling with 20 particles (the planning horizon, number of candidates, and number of elites are not given in the paper; the public code sets them per task) | Retrain the model every episode |
| World Models | VAE \(z\) 32-dim (64 for Doom), 64×64 color · MDN-RNN: LSTM 256 (512 for Doom), 5 Gaussians | VAE → MDN-RNN → controller, each trained separately in that order | Linear controller (867 parameters for CarRacing), CMA-ES population 64, 16 rollouts per individual · Doom dream training temperature \(\tau = 1.15\) | 10,000 rollouts from a random policy |
| PlaNet | RSSM: \(h\) 200 (GRU), \(z\) 30 | Batch 50 × length 50, Adam learning rate \(10^{-3}\), \(\epsilon = 10^{-4}\), gradient clipping 1000, free nats 3 | CEM: planning horizon 12, 10 iterations, 1000 candidates, 100 elites | 5 seed episodes, 1 episode every 100 training steps, exploration noise \(\mathcal{N}(0, 0.3^2)\), action repeat 2 – 8 |
| Dreamer (V1) | Same RSSM as PlaNet (200 / 30) | Learning rates: model \(6 \times 10^{-4}\), actor and value \(8 \times 10^{-5}\); gradient clipping 100, free nats 3 | Imagination horizon \(H = 15\), \(\gamma = 0.99\), \(\lambda = 0.95\), actor trained by backpropagating through the dynamics | Action repeat 2, exploration noise 0.3 |
| DreamerV2 (Atari) | 32 categorical latents × 32 classes, GRU 600 | KL balancing \(\alpha = 0.8\), KL weight 0.1 (1.0 for continuous control), learning rates: model \(2\times10^{-4}\) · actor \(4\times10^{-5}\) · critic \(10^{-4}\), batch 50 × length 50 (arXiv v4, appendix Table D.1) | \(H = 15\), \(\gamma = 0.999\) (the value given in Section 2.1 of the main text; the appendix table of the same paper lists 0.995), \(\lambda = 0.95\), entropy \(10^{-3}\) (\(10^{-4}\) for continuous control), REINFORCE gradients, slow critic updated every 100 steps | Standard Atari settings (frame skip 4) |
| DreamerV3 | Categorical latents 32 × 32 + 1% unimix, models in several sizes (from small up to hundreds of millions of parameters) | Free bits 1, KL weights \(\beta_{\text{rep}} = 0.1\) · \(\beta_{\text{dyn}}\) differs by version (arXiv v1, January 2023: 0.5 / revised v2, April 2024: 1), symlog · two-hot, batch 16 × 64, learning rates also differ by version (v1: model \(10^{-4}\), actor and critic \(3\times10^{-5}\); revision: \(4\times10^{-5}\)) | \(H = 15\), discount horizon 333 (\(\gamma \approx 0.997\)), \(\lambda = 0.95\), entropy \(3\times10^{-4}\), return normalization (5th – 95th percentile), critic EMA | The key point is using the same settings across all domains |
Settings in this tutorial
These values are scaled down to run in minutes to tens of minutes on a personal computer. They can be changed through the argparse defaults and Config in each exercise file.
| Chapter · file | Key settings |
| 06 · | 50 random episodes, 100 epochs |
| 07 · | Planning horizon 25, 1000 random-shooting candidates, CEM 400 candidates · 40 elites · 5 iterations, 8 rounds of iterative training |
| 07 · | Ensemble of 5 (hidden size 128), 200 candidates, 10 particles, planning horizon 25 |
| 08 · | \(z\) 32-dim, \(\beta = 1\), 200 episodes, 10 epochs |
| 09 · | LSTM 256, 5 Gaussians, 6000 steps |
| 10 · | Controller \(a = 2\tanh(W[z; h] + b)\) (289 parameters), CMA-ES 60 generations · population 32 · 4 rollouts · \(\sigma_0 = 0.1\), dream temperatures 0.1 · 1.0 · 1.5 |
| 11 · | \(h\) 200, \(z\) 30, batch 50 × 50, learning rate \(6\times10^{-4}\), gradient clipping 100, free nats 3, reward × 0.1, CEM 12 · 1000 · 10 · 100, action repeat 2, 5 seed episodes, exploration noise 0.3 |
| 12 · | Chapter 11 settings + \(H = 15\), \(\gamma = 0.99\), \(\lambda = 0.95\), actor and critic learning rate \(8\times10^{-5}\), entropy \(10^{-4}\), slow critic updated every 100 steps, actor and critic MLP 300 × 3 |
| 13 · | 100 noise steps, \(\beta\) 0.0001 → 0.15, MLP 256 × 3, 8000 steps |
What to change first
- When things are not working, the rough order of impact is ① the amount of data (episodes, training steps), ② the learning rate, ③ model size, ④ KL-related values (β, free nats), and ⑤ the imagination and planning horizons.
- The discount factor \(\gamma\) determines how many steps ahead the agent looks (about \(\frac{1}{1-\gamma}\) steps, chapter 5), so match it to the time scale of the task. The pendulum in chapter 12 makes 100 decisions with action repeat 2, so \(\gamma = 0.99\) (about 100 steps) is enough.
Key equations by chapter
This table collects the key equations derived in the tutorial, in chapter order. The derivations are on each chapter's page, and the notation follows the conventions in chapter 00 of the tutorial.
Foundations (chapters 02 – 05)
| Equation | Meaning | Chapter |
| \(p(y \mid x) = \dfrac{p(x \mid y)\,p(y)}{p(x)}\) | Bayes' theorem | 02 |
| \(p(x_{1:N}) = \prod_{i=1}^{N} p(x_i \mid x_{1:i-1})\) | Chain rule of probability (the product rule applied repeatedly) | 02 |
| \(\mathbb{E}[X] = \sum_y p(y)\,\mathbb{E}[X \mid y]\) | Law of total expectation (averaging over cases; used to derive the Bellman equation in chapter 5) | 02 |
| \(\operatorname{Var}[X] = \mathbb{E}[X^2] - (\mathbb{E}[X])^2\), \(\ \operatorname{Var}[aX + b] = a^2\operatorname{Var}[X]\) | Properties of variance | 02 |
| \(\log \mathcal{N}(x; \mu, \sigma^2) = -\tfrac12\log(2\pi) - \log\sigma - \dfrac{(x-\mu)^2}{2\sigma^2}\) | Gaussian log density | 02 |
| \(\hat\mu = \tfrac{1}{N}\sum_i x_i\), \(\ \hat\sigma^2 = \tfrac1N\sum_i (x_i - \hat\mu)^2\) | Gaussian maximum likelihood estimation | 02 |
| \(\operatorname*{arg\,min}_\theta \text{NLL} = \operatorname*{arg\,min}_\theta \text{MSE}\) (\(\sigma\) fixed) | MSE is the NLL of a fixed-variance Gaussian | 02 |
| \(\text{BCE} = -[y\log p + (1-y)\log(1-p)]\), \(\ \text{CE} = -l_y + \log\sum_j e^{l_j}\) | Bernoulli and categorical NLL | 02 |
| \(H(p) = -\sum_x p(x)\log p(x)\), \(\ H(p, q) = -\sum_x p(x)\log q(x)\) | Entropy, cross-entropy | 03 |
| \(D_{\mathrm{KL}}(p\,\Vert \,q) = H(p, q) - H(p) = \mathbb{E}_p\!\big[\log\tfrac{p}{q}\big] \ge 0\) | KL divergence and its properties | 03 |
| \(D_{\mathrm{KL}}\big(\mathcal{N}(\mu_1,\sigma_1^2)\,\Vert \,\mathcal{N}(\mu_2,\sigma_2^2)\big) = \log\tfrac{\sigma_2}{\sigma_1} + \dfrac{\sigma_1^2 + (\mu_1-\mu_2)^2}{2\sigma_2^2} - \tfrac12\) | KL between two Gaussians | 03 |
| \(f(\mathbb{E}[X]) \ge \mathbb{E}[f(X)]\) (\(f\) concave) | Jensen's inequality | 03 |
| \(\dfrac{dy}{dx} = \dfrac{dy}{du}\cdot\dfrac{du}{dx}\) | Chain rule (the basis of backpropagation) | 04 |
| \(\theta \leftarrow \theta - \eta\,\nabla_\theta L\), \(\ 0 \lt \eta \lt 2/a\) | Gradient descent and its convergence condition (curvature \(a\)) | 04 |
| \(\sigma'(z) = \sigma(z)(1 - \sigma(z))\), \(\ \tanh'(z) = 1 - \tanh^2 z\) | Derivatives of activation functions | 04 |
| \(G_t = \sum_{k \ge 0}\gamma^k r_{t+k} = r_t + \gamma G_{t+1}\), \(\ \sum_{k\ge0}\gamma^k = \dfrac{1}{1-\gamma}\) | Discounted return and its recursion | 05 |
| \(V^\pi(s) = \sum_a \pi(a\mid s)\sum_{s'}p(s'\mid s,a)\big[r(s,a,s') + \gamma V^\pi(s')\big]\) | Bellman expectation equation | 05 |
| \(V^\ast(s) = \max_a \sum_{s'}p(s'\mid s,a)\big[r(s,a,s') + \gamma V^\ast(s')\big]\) | Bellman optimality equation (the basis of value iteration) | 05 |
| \(\nabla_\psi J = \mathbb{E}_{\tau}\big[R(\tau)\sum_t \nabla_\psi\log\pi_\psi(a_t\mid s_t)\big]\) | Policy gradient theorem (log-derivative trick, REINFORCE) | 05 |
A first World Model (chapters 06 – 07)
| Equation | Meaning | Chapter |
| \(I = \tfrac13 ml^2\), \(\ \ddot\theta = \dfrac{3g}{2l}\sin\theta + \dfrac{3}{ml^2}u\) | Moment of inertia and equation of motion of the pendulum | 06 |
| \(\dot\theta \leftarrow \dot\theta + \ddot\theta\,\Delta t\), \(\ \theta \leftarrow \theta + \dot\theta\,\Delta t\) | (Semi-implicit) Euler integration | 06 |
| \(\tilde x = (x - \mu)/\sigma\) | Standardization | 06 |
| \(e_k \le \varepsilon\,\dfrac{L^k - 1}{L - 1}\) | Upper bound on compounding error in an open-loop rollout | 06 |
| \(a^\ast_{t:t+H-1} = \operatorname*{arg\,max} \sum_{k=0}^{H-1} r(\hat o_{t+k}, a_{t+k})\), \(\ \hat o_{t+k+1} = f_\theta(\hat o_{t+k}, a_{t+k})\) | Planning objective of MPC | 07 |
| \(\mu \leftarrow \tfrac{1}{K}\sum_{k \in \text{elites}} a^{(k)}\), \(\ \sigma^2 \leftarrow \tfrac1K\sum_k (a^{(k)} - \mu)^2\) | CEM: refit the Gaussian to the elites (maximum likelihood) | 07 |
| \(\mathcal{L}(\mu, s) = (y - \mu)^2 e^{-s} + s\), \(\ s = \log\sigma^2\) | Gaussian NLL that also predicts the variance (constants and ½ omitted, PETS) | 07 |
| \(\operatorname{Var}[y] = \tfrac1E\sum_e \sigma_e^2 + \tfrac1E\sum_e(\mu_e - \bar\mu)^2\) | Ensemble uncertainty = aleatoric (mean of the variances) + epistemic (spread of the means) | 07 |
World Models that see images (chapters 08 – 10)
| Equation | Meaning | Chapter |
| \(p_\theta(x) = \int p_\theta(x\mid z)\,p(z)\,dz\) | Marginal likelihood of a latent variable model | 08 |
| \(\log p_\theta(x) \ge \mathbb{E}_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)] - D_{\mathrm{KL}}(q_\phi(z\mid x)\,\Vert \,p(z))\) | ELBO | 08 |
| \(\log p_\theta(x) = \text{ELBO} + D_{\mathrm{KL}}(q_\phi(z\mid x)\,\Vert \,p_\theta(z\mid x))\) | Gap between the ELBO and the true log-likelihood | 08 |
| \(z = \mu + \sigma\odot\epsilon\), \(\ \epsilon\sim\mathcal{N}(0, I)\) | Reparameterization trick | 08 |
| \(D_{\mathrm{KL}}(\mathcal{N}(\mu,\sigma^2)\,\Vert \,\mathcal{N}(0,1)) = \tfrac12(\mu^2 + \sigma^2 - 1 - \log\sigma^2)\) | KL term of the VAE (summed over dimensions) | 08 |
| \(\dfrac{\partial L_T}{\partial h_k} = \dfrac{\partial L_T}{\partial h_T}\prod_{t=k+1}^{T}\operatorname{diag}(1 - h_t^2)\,W_h\) | Gradient from the distant past in an RNN (the cause of vanishing and exploding gradients) | 09 |
| \(\log p(y\mid x) = \operatorname{logsumexp}_k\big(\log\pi_k + \log\mathcal{N}(y;\mu_k,\sigma_k^2)\big)\) | Log-likelihood of a mixture density network | 09 |
| \(\log\sum_k e^{a_k} = m + \log\sum_k e^{a_k - m}\), \(\ m = \max_k a_k\) | log-sum-exp stabilization | 09 |
| \(a_t = 2\tanh(W_c[z_t; h_t] + b_c)\) | The controller in this tutorial (Ha's linear controller \(W_c[z_t; h_t] + b_c\) wrapped in \(2\tanh\) to match the torque range) | 10 |
| \(\nabla_\theta\,\mathbb{E}_\epsilon[F(\theta + \sigma\epsilon)] = \tfrac{1}{\sigma}\mathbb{E}_\epsilon[F(\theta + \sigma\epsilon)\,\epsilon]\) | Gradient estimate of evolution strategies (Gaussian smoothing) | 10 |
| \(C_\mu = \sum_{i=1}^{\mu} w_i\, y_{i:\lambda}\, y_{i:\lambda}^\top\) | CMA-ES: refit the covariance to the successful steps (weighted maximum likelihood) | 10 |
Learning in imagination and modern models (chapters 11 – 13)
| Equation | Meaning | Chapter |
| \(h_t = f(h_{t-1}, z_{t-1}, a_{t-1})\), \(\ z_t \sim p(z_t\mid h_t)\), \(\ z_t \sim q(z_t\mid h_t, o_t)\) | Deterministic, prior, and posterior paths of the RSSM | 11 |
| \(\sum_t \Big(\mathbb{E}_q[\log p(o_t\mid h_t,z_t)] + \mathbb{E}_q[\log p(r_t\mid h_t,z_t)] - \mathbb{E}_q\big[D_{\mathrm{KL}}(q(z_t\mid h_t,o_t)\,\Vert \,p(z_t\mid h_t))\big]\Big)\) | Sequential ELBO | 11 |
| \(\max(D_{\mathrm{KL}}, c)\) | Free nats | 11 |
| \(V^\lambda_t = \hat r_t + \gamma\big[(1-\lambda)\,v(s_{t+1}) + \lambda\,V^\lambda_{t+1}\big]\) | Recursion of the λ-return | 12 |
| \(\tfrac12\big(v_\xi(s_t) - \operatorname{sg}(V^\lambda_t)\big)^2\) | Critic loss | 12 |
| \(z = \operatorname{onehot} + p - \operatorname{sg}(p)\) (value = one-hot, \(\partial\mathcal{L}/\partial p = \partial\mathcal{L}/\partial z\)) | Straight-through estimator | 12 |
| \(\alpha\,D_{\mathrm{KL}}(\operatorname{sg}(q)\,\Vert \,p) + (1-\alpha)\,D_{\mathrm{KL}}(q\,\Vert \,\operatorname{sg}(p))\) | KL balancing (DreamerV2, \(\alpha = 0.8\)) | 12 |
| \(\operatorname{symlog}(x) = \operatorname{sign}(x)\log(\lvert x\rvert + 1)\), \(\ \operatorname{symexp}(x) = \operatorname{sign}(x)(e^{\lvert x\rvert} - 1)\) | symlog and its inverse | 12 |
| \(w_{k+1} = \dfrac{y - b_k}{b_{k+1} - b_k}\), \(\ w_k = \dfrac{b_{k+1} - y}{b_{k+1} - b_k}\) | two-hot weights | 12 |
| \(z_t = \sum_{i=0}^{n-1}\gamma^i r_{t+i} + \gamma^n v(s_{t+n})\) | n-step value target (MuZero). Here \(z_t\) is the value target, not a latent variable | 13 |
| \(Q + P\,\dfrac{\sqrt{\sum_b N}}{1 + N}\big(c_1 + \log\tfrac{\sum_b N + c_2 + 1}{c_2}\big)\) | pUCT score in MCTS | 13 |
| \(q^\ast \propto p\,\exp(J/\tau)\), \(\ w_i = \dfrac{\exp(J_i/\tau)}{\sum_j\exp(J_j/\tau)}\) | MPPI weights (\(\tau\) is the temperature) | 13 |
| \(\operatorname{softmax}\!\big(QK^\top/\sqrt{d}\big)V\), \(\ \operatorname{Var}(q\cdot k) = d\) | Attention, and why it divides by \(\sqrt d\) | 13 |
| \(x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon\) | Closed form of the forward diffusion process | 13 |
| \(\tilde\mu_t = \dfrac{1}{\sqrt{\alpha_t}}\Big(x_t - \dfrac{\beta_t}{\sqrt{1-\bar\alpha_t}}\epsilon\Big)\), \(\ \tilde\beta_t = \dfrac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_t\) | Mean and variance of the true reverse distribution | 13 |
| \(\mathbb{E}\big\lVert\epsilon - \epsilon_\theta(\sqrt{\bar\alpha_t}x_0 + \sqrt{1-\bar\alpha_t}\epsilon,\ t)\big\rVert^2\) | Training loss of a diffusion model | 13 |
| \(\bar\theta \leftarrow \tau\bar\theta + (1-\tau)\theta\) | EMA target encoder (JEPA, TD-MPC). Here \(\tau\) is the mixing rate | 13 |
Glossary
Terms used in the (Korean-language) tutorial, listed in Korean alphabetical order. "Chapter" is the chapter that first defines and explains the term in detail (chapters where only the name appears earlier are not listed).
Korean terms
| Korean term | English | Meaning | Chapter |
| 가능도 | likelihood | The probability (density) of the observed data with the parameters held fixed, viewed as a function of the parameters | Ch. 2 |
| 가우시안 분포 (정규 분포) | Gaussian / normal distribution | The bell-shaped distribution N(μ, σ²) with mean μ and variance σ² | Ch. 2 |
| 가우시안 평활화 | Gaussian smoothing | A technique that optimizes E[F(θ + σε)] instead of F so that the gradient can be estimated from samples | Ch. 10 |
| 가치 등가 | value equivalence | The World Model design principle that only the quantities used for planning (reward, value, policy) need to match reality (MuZero) | Ch. 13 |
| 가치 반복 | value iteration | A method that finds the optimal value by repeatedly applying the Bellman optimality equation | Ch. 5 |
| 가치 함수 | value function | The expected return V(s) when following a policy from state s | Ch. 5 |
| 강화학습 | reinforcement learning, RL | Learning, by trial and error, the actions that maximize reward | Ch. 5 |
| 결합 확률 | joint probability | The probability p(x, y) that several events occur together | Ch. 2 |
| 경사 하강법 | gradient descent | A method that reduces the loss by moving the parameters a little at a time in the direction opposite to the gradient | Ch. 4 |
| 계획 | planning | Using a model to think through the future before choosing an action | Ch. 5 |
| 곡률 | curvature | How sharply the loss surface bends. The larger it is, the smaller the learning rate must be | Ch. 4 |
| 공통 난수 | common random numbers | A technique that evaluates candidates with the same seed and the same initial state to reduce noise in the comparison | Ch. 10 |
| 과적합 | overfitting | Fitting the training data well but getting new data wrong | Ch. 4 · 6 |
| 관성 모멘트 | moment of inertia | How hard something is to rotate. ml²/3 for a rod rotating about one end | Ch. 6 |
| 관측 | observation | What the agent actually sees, o_t (a numeric vector or an image) | Ch. 1 |
| 교차 엔트로피 | cross-entropy | The average information when data from p is encoded using q, H(p, q) = −Σ p log q | Ch. 3 |
| 궤적 | trajectory | States, actions, and rewards laid out in time order, τ | Ch. 5 |
| 궤적 표본 전파 | trajectory sampling, TS | PETS's method of propagating uncertainty into the future by sampling an ensemble member and noise for each particle | Ch. 7 |
| 기댓값 | expectation | The probability-weighted average E[f(x)] | Ch. 2 |
| 기울기 소실 | vanishing gradient | The gradient approaching 0 as it flows back through layers or time, so that learning stalls | Ch. 4 |
| 기울기 자르기 | gradient clipping | A stabilization technique that shrinks the gradient vector to length c when its length exceeds c | Ch. 9 |
| 기울기 차단 | stop-gradient, sg | An operation that uses a value as is but lets no gradient flow through it (.detach()) | Ch. 12 |
| 기준선 | baseline | A value subtracted from the return to reduce the variance of the policy gradient. Subtracting it leaves the expectation unchanged | Ch. 5 |
| 꿈 | dream | Imagined trajectories generated by the World Model itself. Policies are sometimes trained inside them | Ch. 9 |
| 노출 편향 | exposure bias | The mismatch that arises from receiving true inputs during training but the model's own outputs as inputs during generation | Ch. 9 |
| 다루기 어렵다 | intractable | So expensive to compute exactly that it effectively cannot be computed | Ch. 8 |
| 다봉 분포 | multimodal distribution | A distribution with several peaks. Its mean may resemble none of the peaks | Ch. 1 |
| 다양체 가설 | manifold hypothesis | The assumption that high-dimensional data (images) actually lies near a low-dimensional surface | Ch. 8 |
| 다층 퍼셉트론 | multilayer perceptron, MLP | The most basic neural network, built by alternating linear layers and activation functions | Ch. 4 |
| 대각 가우시안 | diagonal Gaussian | A multivariate Gaussian whose dimensions are independent. Its covariance matrix is diagonal | Ch. 2 |
| 독립 | independent | p(x, y) = p(x) p(y). Knowing one variable gives no information about the other | Ch. 2 |
| 드리프트 | drift | Scenes gradually going wrong as errors accumulate while the model keeps generating from its own outputs | Ch. 13 |
| 라그랑주 승수 | Lagrange multiplier | A variable added to turn a constrained optimization into an unconstrained problem | Ch. 13 |
| 로그 미분 트릭 | log-derivative trick | A method that uses ∇p = p ∇log p to estimate the gradient of an expectation from samples | Ch. 5 |
| 로짓 | logit | A score before softmax or sigmoid is applied (an unnormalized log probability) | Ch. 2 |
| 리턴 | return | The sum of (discounted) future rewards, G_t | Ch. 5 |
| 리플레이 버퍼 | replay buffer | Storage that holds the experience (observations, actions, rewards) collected by the agent, sampled at random during training. Denoted 𝒟 | Ch. 11 |
| 립시츠 상수 | Lipschitz constant | An upper bound L on how many times larger a difference in the input can become in the output | Ch. 6 |
| 마르코프 결정 과정 | Markov decision process, MDP | A decision problem defined by states, actions, transition probabilities, rewards, and a discount factor | Ch. 5 |
| 마르코프 성질 | Markov property | The property that the next state depends only on the current state and action, not on anything earlier | Ch. 5 |
| 매개변수 | parameter | The numbers in a neural network that training adjusts (weights and biases) | Ch. 4 |
| 모델 기반 강화학습 | model-based RL | Reinforcement learning that learns a model of the environment (a World Model) and uses it for planning or imagination | Ch. 1 |
| 모델 없는 강화학습 | model-free RL | Reinforcement learning that learns a policy or value directly from experience, without a model | Ch. 1 |
| 모델 예측 제어 | model predictive control, MPC | A method that plans ahead with the model at every step, executes only the first action, and then replans | Ch. 7 |
| 모델 착취 | model exploitation | A policy exploiting errors in the World Model to score highly only in imagination | Ch. 1 |
| 목표 신경망 | target network | A slowly following copy of a network (periodic copy or EMA) used to compute training targets. It keeps the target from shifting at every step. Examples: Dreamer's slow critic, the target encoder in TD-MPC and JEPA | Ch. 5 · 12 |
| 몬테카를로 추정 | Monte Carlo estimation | Approximating an expectation by drawing samples and averaging them | Ch. 2 |
| 미니배치 | minibatch | A small random subset drawn at each step instead of the full dataset. Its gradient is an unbiased estimate of the full gradient | Ch. 4 |
| 미분 | derivative | The rate at which the output changes when the input changes very slightly | Ch. 4 |
| 미분 방정식 | differential equation | An equation relating a value to its rate of change (derivative). Example: the pendulum's equation of motion | Ch. 6 |
| 믿음 상태 | belief state | The probability distribution b_t(s) over the current state under partial observability | Ch. 5 |
| 밴딧 | bandit | The simplest reinforcement learning problem: choosing actions with no state | Ch. 5 |
| 베르누이 분포 | Bernoulli distribution | A distribution that outputs 0 or 1, with probability p of outputting 1 | Ch. 2 |
| 베이즈 정리 | Bayes' theorem | p(y | x) = p(x | y) p(y) / p(x). Reverses a conditional probability between cause and effect | Ch. 2 |
| 벨만 방정식 | Bellman equation || The recursion value = current reward + discounted value of the next state | Ch. 5 | |
| 벨만 최적 방정식 | Bellman optimality equation | Optimal value = (reward + discounted next value) of the best action | Ch. 5 |
| 변분 추론 | variational inference | A method that approximates an intractable posterior with a tractable distribution q | Ch. 8 |
| 보상 | reward | The score r_t that the environment gives for an action | Ch. 1 |
| 보상 모델 | reward model | A model that predicts the reward from the state and action | Ch. 1 |
| 보편 근사 정리 | universal approximation theorem | The theorem that a sufficiently wide neural network can imitate any continuous function as closely as desired | Ch. 4 |
| 볼록 | convex | A function shape in which the line segment joining any two points lies above the graph (e.g., x²) | Ch. 3 |
| 부분 관측 | partial observability | A situation where a single observation does not reveal the whole state. A single image does not show the angular velocity | Ch. 1 |
| 부트스트랩 | bootstrap | ① Statistics: resampling the data with replacement so that each model gets slightly different training data (chapter 7) ② Reinforcement learning: building training targets from one's own value estimates (TD targets, chapters 5 and 12) | Ch. 7 · 12 |
| 분리 표현 | disentanglement | A representation in which each component of the latent variable corresponds to a single meaningful factor | Ch. 8 |
| 분산 | variance | How spread out values are around the mean, Var[X] = E[(X − E[X])²] | Ch. 2 |
| 분할 상환 추론 | amortized inference | An approach in which a single network (the encoder) outputs q(z | x) directly instead of optimizing separately for each data point | Ch. 8 |
| 블랙박스 최적화 | black-box optimization | Optimization that finds an optimum by looking only at function values, without gradients | Ch. 10 |
| 사람 기준 정규화 점수 | human-normalized score, HNS | (agent − random)/(human − random). Used to compare Atari results | Ch. 13 |
| 사전 확률 | prior | The belief before seeing the data, p(z) | Ch. 2 |
| 사후 붕괴 | posterior collapse | The VAE's q(z | x) becomes identical to the prior, so the latent variable carries no information | Ch. 8 |
| 사후 확률 (사후 분포) | posterior | The belief after seeing the data, p(z | x) | Ch. 2 |
| 상태 | state | The true situation of the environment, s_t. For the pendulum, (θ, θ̇) | Ch. 1 |
| 상태 공간 모델 | state-space model, SSM | A model in which a hidden state changes over time and observations are generated from that state | Ch. 11 |
| 상호정보량 | mutual information | How much knowing one variable reduces uncertainty about another, I(X; Y) | Ch. 3 |
| 선형 층 | linear layer || y = Wx + b. A fully connected layer | Ch. 4 | |
| 소프트맥스 | softmax | A function that exponentiates scores and turns them into probabilities that sum to 1 | Ch. 2 |
| 손실 함수 | loss function | A value that measures how wrong a prediction is. Training reduces it | Ch. 4 |
| 순환 신경망 | recurrent neural network, RNN | A neural network that processes a sequence one step at a time while carrying a hidden state | Ch. 9 |
| 스트라이드 | stride | How many cells the convolution kernel moves at a time. With 2, the output size is roughly halved | Ch. 4 |
| 신경망 | neural network | A function built by composing many layers of simple parameterized functions (layers) | Ch. 4 |
| 앙상블 | ensemble | Training several models of the same architecture separately and using them together. Their disagreement measures uncertainty | Ch. 7 |
| 액터 | actor | The policy network π_ψ that chooses actions | Ch. 5 |
| 어텐션 | attention | An operation that mixes values according to the similarity between queries and keys (softmax weights). The core of the Transformer | Ch. 13 |
| 에너지 | energy | A value that measures how poorly two inputs match. In JEPA, the distance between representations | Ch. 13 |
| 에이전트 | agent | The entity that looks at observations and chooses actions | Ch. 1 |
| 에폭 | epoch | One full pass over the training data | Ch. 6 |
| 에피소드 | episode | One attempt from start to finish. 200 steps for the pendulum | Ch. 1 |
| 엔트로피 | entropy | The average information (uncertainty) of a distribution, H(p) = −Σ p log p | Ch. 3 |
| 역전파 | backpropagation | A method that uses the chain rule to compute the gradients of all parameters backward from the loss at the output | Ch. 4 |
| 열린 고리 롤아웃 | open-loop rollout | Rolling forward from only the initial state by feeding the model's own predictions back in | Ch. 6 |
| 오목 | concave | A function shaped the opposite way from a convex one (e.g., log x) | Ch. 3 |
| 오차 누적 | compounding error | Small one-step errors accumulating and growing during multi-step prediction | Ch. 1 |
| 오토인코더 | autoencoder, AE | A neural network trained to compress its input into a small code and then reconstruct it | Ch. 8 |
| 온도 | temperature | A value τ that sets how flat (random) a distribution is | Ch. 9 |
| 우연적 불확실성 | aleatoric uncertainty | Randomness in the world itself. It does not shrink with more data (the mean of the variances) | Ch. 7 |
| 은닉 상태 | hidden state | The memory h_t that an RNN carries along | Ch. 9 |
| 음의 로그 가능도 | negative log-likelihood, NLL | −log p(data). Maximum likelihood estimation = NLL minimization | Ch. 2 |
| 이점 | advantage | A(s, a) = Q(s, a) − V(s). How much better this action is than average | Ch. 5 |
| 인과 마스크 | causal mask | Masking scores with −∞ so that the Transformer cannot see future tokens | Ch. 13 |
| 인식적 불확실성 | epistemic uncertainty | Not knowing because of insufficient data. It shrinks as data grows (the spread of the ensemble means) | Ch. 7 |
| 일반화 | generalization | Predicting well on new data that was not used in training | Ch. 4 |
| 자기회귀 | autoregressive | Building a sequence by repeatedly predicting the next item from the ones before it | Ch. 13 |
| 자동 미분 | automatic differentiation, autograd | A feature that records the computation and then computes gradients automatically with the chain rule | Ch. 4 |
| 자유 내트 | free nats | A technique that stops learning from the KL term while the KL is below a set value, max(KL, c) | Ch. 11 |
| 잠재 과대 예측 | latent overshooting | An auxiliary objective in PlaNet that also fits priors several steps ahead to the posteriors | Ch. 11 |
| 잠재 변수 | latent variable | A hidden variable z that is not directly observed and explains the data | Ch. 1 |
| 잠재 행동 모델 | latent action model, LAM | A model that learns discrete actions explaining frame-to-frame changes from video without action labels (Genie) | Ch. 13 |
| 재매개변수화 트릭 | reparameterization trick || A method that writes z = μ + σ·ε so that gradients can flow past the sampling step | Ch. 8 | |
| 전이 모델 | transition model, dynamics model | A model that predicts the next state from the current state and action | Ch. 1 |
| 전체 기댓값의 법칙 | law of total expectation | Splitting into cases and adding up each case's expectation weighted by its probability gives the overall expectation. E[X] = Σ_y p(y) E[X | y] | Ch. 2 |
| 전치 합성곱 | transposed convolution | A convolution that enlarges the image. Used in the VAE decoder | Ch. 4 · 8 |
| 정보 병목 | information bottleneck | A design that deliberately narrows the information that can pass through, so only essential information remains | Ch. 13 |
| 정보량 | information content, surprisal | The surprise −log p(x), larger for rarer events | Ch. 3 |
| 정신 모형 | mental model | A scaled-down model of the world that a person holds in their head | Ch. 1 |
| 정예 | elite | The top-scoring candidates in CEM | Ch. 7 |
| 정책 | policy | The rule π(a | s) for choosing an action from the state (observation) | Ch. 1 |
| 젠센 부등식 | Jensen's inequality | For a concave function f, f(E[X]) ≥ E[f(X)]. The key step in deriving the ELBO | Ch. 3 |
| 조건부 기댓값 | conditional expectation | The expectation E[X | Y = y] given a condition. It is the prediction that minimizes MSE | Ch. 2 |
| 조건부 확률 | conditional probability | The probability of x given y, p(x | y) = p(x, y)/p(y) | Ch. 2 |
| 주변 확률 | marginal probability | The probability of a single variable, obtained by summing (integrating) all other variables out of the joint probability | Ch. 2 |
| 주변화 | marginalization | Removing variables of no interest by summing (integrating) them out, p(x) = Σ_y p(x, y) | Ch. 2 |
| 증거 | evidence | The denominator p(x) in Bayes' theorem: the probability of the observation summed over all causes | Ch. 2 |
| 지도 학습 | supervised learning | Learning an input → answer function from pairs of inputs and correct answers | Ch. 4 |
| 직선 통과 추정기 | straight-through estimator | A trick that passes discrete values forward and passes gradients backward unchanged | Ch. 12 |
| 진화 경로 | evolution path | A vector in which CMA-ES accumulates past mean shifts, used to adjust the step size and covariance | Ch. 10 |
| 진화 전략 | evolution strategies, ES | Black-box optimization that sets the next distribution from the scores of candidates made by adding noise to the parameters | Ch. 10 |
| 최대우도 추정 | maximum likelihood estimation, MLE | A method that chooses the parameters that make the likelihood of the data as large as possible | Ch. 2 |
| 커널 (필터) | kernel, filter | A small set of weights that slides over the image and is multiplied in during a convolution | Ch. 4 |
| 코드북 | codebook | The list of vectors {e_1, …, e_K} that a VQ-VAE can choose from | Ch. 13 |
| 크리틱 | critic | A network v_ξ that estimates the value of a state | Ch. 5 |
| 큰 수의 법칙 | law of large numbers | The more samples there are, the closer the sample mean gets to the expectation | Ch. 2 |
| 텐서 | tensor | An array of numbers laid out along several axes. PyTorch's basic data type | Ch. 4 |
| 토크 | torque | A force that turns something around an axis. The pendulum's action u | Ch. 6 |
| 토큰 | token | A single discrete symbol (an integer) that the model handles. Images also become token sequences via VQ | Ch. 13 |
| 패딩 | padding | Values (usually 0) added around the border of the image before convolution. Controls the output size | Ch. 4 |
| 편미분 | partial derivative | Differentiating with respect to one variable while holding the others fixed | Ch. 4 |
| 표준화 | standardization | Preprocessing with (x − mean)/standard deviation to give mean 0 and standard deviation 1 | Ch. 4 · 6 |
| 표현 붕괴 | representation collapse | A failure in which all inputs collapse into the same representation and only the loss gets smaller | Ch. 13 |
| 필터링 | filtering | Estimating the distribution of the current state from the observations so far | Ch. 11 |
| 할인율 | discount factor | A value γ (0 – 1) that sets how much future rewards are discounted | Ch. 5 |
| 합성 함수 | composite function | A function inside a function, f(g(x)). What the chain rule applies to | Ch. 4 |
| 합성곱 신경망 | convolutional neural network, CNN | A neural network that slides small filters over an image to extract local features | Ch. 4 |
| 행동 | action | What the agent does to the environment, a_t. For the pendulum, the torque u | Ch. 1 |
| 행동 반복 | action repeat | A technique that repeats the same action for k steps to reduce the number of decisions | Ch. 11 |
| 혼합 밀도 네트워크 | mixture density network, MDN | A neural network that outputs the weights, means, and standard deviations of a Gaussian mixture | Ch. 9 |
| 확률밀도함수 | probability density function, PDF | The probability density of a continuous variable. Integrating it over an interval gives a probability | Ch. 2 |
| 확률변수 | random variable | A variable whose value is determined by the outcome of an experiment | Ch. 2 |
| 확률적 경사 하강법 | stochastic gradient descent, SGD | Gradient descent that updates the parameters with gradients estimated from minibatches | Ch. 4 |
| 확률질량함수 | probability mass function, PMF | The probability that a discrete variable takes each value | Ch. 2 |
| 확산 모델 | diffusion model | A generative model that learns to reverse the process of adding noise to data, producing data from noise | Ch. 13 |
| 환경 | environment | The world the agent interacts with. It takes an action and returns the next observation and reward | Ch. 1 |
| 활성 함수 | activation function | A nonlinear function placed between layers (ReLU, tanh, SiLU …) | Ch. 4 |
| 후퇴 지평 | receding horizon | Planning H steps ahead at every step, executing only the first action, then sliding the window forward by one and replanning | Ch. 7 |
| 1차 테일러 근사 | first-order Taylor approximation | f(x + Δ) ≈ f(x) + f'(x)Δ. The reason gradient descent reduces the loss | Ch. 4 |
| KL 균형 | KL balancing | A DreamerV2 technique that splits the KL gradient between the prior side (α) and the posterior side (1−α) | Ch. 12 |
| KL 발산 | Kullback–Leibler divergence | How different two distributions are, KL(p‖q) = Σ p log(p/q) ≥ 0 | Ch. 3 |
| n-스텝 리턴 | n-step return | n real rewards + γⁿ × (the value estimate n steps later) | Ch. 12 |
| β-VAE | beta-VAE | A VAE that multiplies the KL term by β to balance reconstruction against compression | Ch. 8 |
| λ-리턴 | λ-return | A value target that mixes several n-step returns with weights (1−λ)λ^(n−1) | Ch. 12 |
English abbreviations and names
| Abbreviation / English term | Meaning | Chapter |
| Adam | An adaptive learning-rate optimizer. Adjusts the step using moving averages of the gradient and its magnitude | Ch. 4 |
| BPTT | backpropagation through time. Backpropagation through an RNN unrolled over time | Ch. 9 |
| CEM | cross-entropy method. An optimizer that repeatedly samples candidates and refits a Gaussian to the top ones (the elites) | Ch. 7 |
| CMA-ES | covariance matrix adaptation evolution strategy. An evolution strategy that also adapts the covariance | Ch. 10 |
| DDPM | denoising diffusion probabilistic model. A diffusion model trained by predicting noise (Ho et al., 2020) | Ch. 13 |
| Dyna | An architecture that learns a model from real experience and also trains values and policies on simulated experience generated by the model (Sutton, 1990 – 1991) | Ch. 1 |
| ELBO | evidence lower bound. A lower bound on log p(x) = reconstruction − KL | Ch. 8 |
| EMA | exponential moving average. A slowly following average, θ̄ ← τθ̄ + (1−τ)θ | Ch. 4 · 13 |
| GRU | gated recurrent unit. An RNN that controls its memory with two gates (update and reset) | Ch. 9 |
| JEPA | joint-embedding predictive architecture. An architecture that predicts in representation space instead of pixel space | Ch. 13 |
| log-sum-exp | A way to compute log Σ exp(a_k) stably by subtracting the maximum | Ch. 2 |
| LSTM | long short-term memory. An RNN that keeps long-term memory with a cell state and three gates | Ch. 9 |
| MCTS | Monte Carlo tree search. A search that digs down promising branches one step at a time (MuZero) | Ch. 13 |
| MDP | Markov decision process. A decision problem defined by states, actions, transitions, rewards, and a discount factor | Ch. 5 |
| MPPI | model predictive path integral. A planning method that averages candidates with exp(J/τ) weights | Ch. 13 |
| MSE | mean squared error. The mean of the squared errors | Ch. 4 |
| PETS | probabilistic ensembles with trajectory sampling. Probabilistic ensemble + trajectory sampling + CEM-MPC | Ch. 7 |
| PlaNet | deep planning network. An agent that plans with CEM in the RSSM latent space | Ch. 11 |
| POMDP | partially observable MDP. An MDP in which observations alone do not reveal the whole state | Ch. 5 |
| pUCT | A score in MCTS that mixes exploitation (Q) and exploration (prior probability and visit count) | Ch. 13 |
| REINFORCE | A policy gradient method that differentiates the policy directly via ∇J = E[∇ log π(a|s) · G] | Ch. 5 |
| RSSM | recurrent state-space model. Latent dynamics that combine a deterministic path h_t and a stochastic path z_t | Ch. 11 |
| SiLU (Swish) | x · sigmoid(x). A smooth, ReLU-like activation function | Ch. 6 |
| SimNorm | A TD-MPC2 technique that normalizes the latent vector with a softmax over each group | Ch. 13 |
| softplus | log(1 + eˣ). An always-positive smooth ReLU. Used to produce standard deviations | Ch. 7 |
| symlog | sign(x) log(|x| + 1). A transform that squashes values of widely varying magnitude (DreamerV3) | Ch. 12 |
| TD error | temporal difference error. δ = r + γV(s') − V(s). Its expectation is the advantage | Ch. 5 |
| two-hot | A categorical encoding that splits a single real number's weight between two neighboring bins (DreamerV3) | Ch. 12 |
| unimix | A technique that mixes a little uniform distribution (1% in DreamerV3) into a categorical distribution so that no probability is exactly 0 | Ch. 12 |
| VAE | variational autoencoder. A latent variable model and an encoder (variational inference) trained together with the ELBO | Ch. 8 |
| VQ-VAE | vector-quantized VAE. Turns latents into tokens by replacing each with the nearest codebook vector | Ch. 13 |
Paper map
Rather than reading a paper from start to finish, read ① the abstract and Figure 1, ② the key equations in the method section, and ③ the main tables in the experiments, in that order, and go back to the corresponding tutorial chapter for any equation you do not follow. The order below follows the flow of the tutorial.
| Order | Paper | Why read it | Chapters |
| 0 | Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed.), chapters 3 – 4 and 8 | MDPs and the Bellman equations, plus Dyna (the classic of model-based reinforcement learning) | 05 |
| 1 | Kingma & Welling, “Auto-Encoding Variational Bayes”, 1312.6114 | The original source of the VAE and the reparameterization trick | 08 |
| 2 | Graves, “Generating Sequences With Recurrent Neural Networks”, 1308.0850 | How to generate sequences with an MDN-RNN | 09 |
| 3 | Ha & Schmidhuber, “World Models”, 1803.10122 (+ interactive article) | The V · M · C architecture and learning inside dreams. Easy to read, with good figures | 10 |
| 4 | Chua et al., “PETS”, 1805.12114 | Probabilistic ensembles and uncertainty-aware MPC | 07 |
| 5 | Janner et al., “MBPO: When to Trust Your Model”, 1906.08253 | Using short model rollouts to help model-free learning; an analysis of compounding error | 06 · 07 |
| 6 | Salimans et al., “Evolution Strategies”, 1703.03864 · Hansen, “CMA-ES Tutorial”, 1604.00772 | Evolution strategies that find policies without gradients | 10 |
| 7 | Hafner et al., “PlaNet”, 1811.04551 | The RSSM and planning in latent space | 11 |
| 8 | Hafner et al., “Dreamer”, 1912.01603 | Actor-critic in imagination and the λ-return | 12 |
| 9 | Hafner et al., “DreamerV2”, 2010.02193 · “DreamerV3”, 2301.04104 | Categorical latents and KL balancing, plus a single set of settings across many domains | 12 |
| 10 | Schrittwieser et al., “MuZero”, 1911.08265 | Value-equivalent models and MCTS | 13 |
| 11 | Hansen et al., “TD-MPC2”, 2310.16828 | The current form of decoder-free latent planning | 13 |
| 12 | Micheli et al., “IRIS”, 2209.00588 | Tokens + a Transformer World Model | 13 |
| 13 | Ho et al., “DDPM”, 2006.11239 → Alonso et al., “DIAMOND”, 2405.12399 | Diffusion models and diffusion World Models | 13 |
| 14 | LeCun, “A Path Towards Autonomous Machine Intelligence” → “V-JEPA 2”, 2506.09985 | A different path: prediction in representation space | 13 |
| 15 | Bruce et al., “Genie”, 2402.15391 → Hafner et al., “Dreamer 4”, 2509.24527 | Large-scale video World Models and training agents inside them | 13 |
| + | Moerland et al., “Model-based RL: A Survey”, 2006.16712 | A survey covering model-based reinforcement learning as a whole. Useful when you want to redraw the map | 01 |
Public implementations and tools
Once you have learned the principles from the tutorial's small implementations, start from well-tested public implementations when you take on larger tasks.
| Implementation | Description |
| Official DreamerV3 implementation (JAX). There are also dreamerv2 and dreamer repositories | |
| Community PyTorch reimplementation of DreamerV3. Useful if you want to read it in PyTorch, like this tutorial | |
| Official PlaNet implementation (TensorFlow) | |
| Experiment code for Ha & Schmidhuber's World Models | |
| Model-based reinforcement learning library including PETS and MBPO (PyTorch) | |
| Official TD-MPC2 implementation and trained checkpoints | |
| Readable community PyTorch reimplementation of MuZero | |
| Official implementations of the IRIS (Transformer) and DIAMOND (diffusion) World Models | |
| V-JEPA 2 code and pretrained models | |
| NVIDIA Cosmos World Foundation Model repositories | |
| Experiment environments. This tutorial's pendulum has the same physics as Gymnasium's |
Porting the tutorial code to Gymnasium environments
- The environments in
wm_envs.pyreturn three values,obs, reward, done = env.step(action), whereas Gymnasium returns five,obs, reward, terminated, truncated, info = env.step(action). Convert withdone = terminated or truncated. - Once you also account for
reset()returning(obs, info), most of the exercise code can be used as is. - For image observations, create the environment with
render_mode="rgb_array", convert the RGB image returned byenv.render()to grayscale (1 channel), and resize it to 64×64. The models in the text take 1-channel input.
Frequently asked questions
- Can I follow along to the end without a GPU?
- Up to chapter 10, and chapter 13, yes. The code is kept small enough to run on a CPU, and each chapter lists the actual time it took (about 1 – 20 minutes on a CPU for chapters 06, 07, and 13). The settings in the text for chapters 11 and 12 (PlaNet and Dreamer, which see images) take more than a day on a CPU, which is impractical. First confirm that the code runs with
--quick, then either run the full training on a GPU (a free cloud notebook works) or reduce the number of training steps and look only at the trends. - How does a World Model differ from a simulator (physics engine)?
- A physics engine contains rules (equations) written in by people; a World Model learns those rules from data. Chapter 6 learns the pendulum with a neural network even though its equations are known, because in the real world (robots, games, driving) the equations are usually unknown or too complex. A learned model can be wrong, so always keep uncertainty (chapter 7) and model exploitation (chapter 10) in mind.
- Is model-based reinforcement learning always better than model-free reinforcement learning?
- No. When interacting with the environment is expensive (real robots, slow simulators), sample efficiency is a big advantage, but it comes at a price: the compute needed to learn the model, and model error. When the simulator is very fast and cheap, model-free methods are often simpler and reach better final performance. That said, methods such as DreamerV3 that aim for the advantages of both are becoming more common.
- Why use only the pendulum from start to finish?
- Keeping the world the same lets you compare only the effect of changing the model. The pendulum's physics fits in three lines, so results can be checked against the ground truth; seen as images, it becomes a partially observable problem in which velocity cannot be seen; and, like a swing, it requires building up energy, so a simple greedy policy cannot solve it. It is small, but it contains all the core difficulties of World Models.
- What to change when making images larger or switching to color
- Change the number of convolutional layers and the input channels (grayscale 1 → color 3) in the encoder and decoder, and recompute the sizes layer by layer following the output-size calculations in chapters 8 and 11. More pixels increase the magnitude of the reconstruction loss, so rebalance it against the KL (β, free nats) as well.
- When Dreamer does not perform as well as in the paper
- First compare the number of training steps. The papers use hundreds of thousands to millions of steps, while the tutorial uses far fewer. Then check "the gap between imagined and real returns", "open-loop imagined images", and "KL magnitude" in the symptom tables. If reproduction is the goal, comparing against a public implementation with the same settings is the fastest route.
- Are large language models (LLMs) also World Models?
- LLMs clearly hold a great deal of knowledge about the world, but they do not exactly fit the tutorial's narrow definition ("given an action, predict the next state, and use that to plan or learn"). Research that combines language with World Models is active, such as using an LLM as an agent's planner (imagining the outcomes of actions in text) or video World Models that create and modify worlds from text, like Genie 3 and 4 in chapter 13. Researchers have not agreed on how far the term World Model should extend.
- When the math gets stuck
- Go back to the chapter that the derivation of the equation points to (usually chapters 2 and 3). Then plug small numbers into the equation and compute it yourself. Every key equation in the tutorial comes with a numerical example and code that checks those numbers.
Next steps: projects to try
Projects worth trying, in order of difficulty, using the tutorial code as a starting point.
| Difficulty | Project | What you learn |
| ★☆☆ | Change the pendulum's reward so the goal is to “spin fast while hanging upside down”, and run the chapter 7 MPC on it | The advantage of model-based methods: solving a new task by changing only the reward and keeping the World Model as is |
| ★☆☆ | Change the chapter 6 dynamics model to predict the “next observation” directly instead of the “change” (turning off trick 2), and compare one-step error and compounding error | How the design of the output affects multi-step prediction |
| ★★☆ | Port the chapter 6 and 7 exercises to Gymnasium's | Environment interfaces, and the exploration problem in sparse-reward tasks |
| ★★☆ | Add DreamerV3's symlog, two-hot, free bits, and return normalization to the chapter 12 Dreamer one at a time and compare their effects | Confirming the role of each technique by experiment (ablation study) |
| ★★☆ | Add the disagreement of the chapter 7 ensemble as a penalty on the chapter 12 imagined rewards to reduce model exploitation | Uncertainty-aware learning in imagination |
| ★★★ | Scale the chapter 13 diffusion exercise up to “next-frame prediction for 64×64 pendulum images” | Conditional diffusion World Models, drift, and noise augmentation |
| ★★★ | Remove the decoder from the chapter 11 RSSM and replace it with a TD-MPC-style latent consistency + TD loss | Reconstruction-free World Models and representation collapse |
| ★★★ | Train a Genie-style latent action model on pendulum videos (without action logs) and check whether the codes correspond to the direction of the torque | Learning World Models without action labels, information bottleneck |
Summary
- Start debugging by checking the data, checking shapes, and overfitting a small dataset, in that order; change hyperparameters last, one at a time.
- Log losses term by term and gradients by their norm. A norm of 0 means a cut-off gradient; values jumping into the thousands mean an explosion.
- NaN mostly comes from \(\log 0\), exponential overflow, and zero standard deviation. Prevent it with a stable softplus, logit BCE (\(\operatorname{softplus}(x) - yx\)),
logsumexp, a standard deviation floor, and gradient clipping. - Hyperparameters from the original papers are starting points and differ by version and official code. If things do not work, adjust the amount of data, the learning rate, model size, KL-related values, and the imagination and planning horizons, in that order.
- The tutorial as a whole starts from a one-sentence definition, "a learned simulator that predicts what comes next given a state and an action", and goes from an MLP that predicts three numbers to Dreamer, which learns to swing the pendulum upright in imagination from images alone.
- Today's large World Models are also different answers to the same three questions: what to predict, how to represent an uncertain future, and how to turn imagination into action.
Review questions
- 1. The result of clipping the gradient \(g = (3, 4)\) to a maximum length of \(c = 1\), and the result when \(c = 10\)
- \(\lVert g\rVert = 5\). With \(c = 1\), \(g' = (3, 4)/5 = (0.6, 0.8)\). With \(c = 10\), \(5 \le 10\), so it stays \((3, 4)\).
- 2. For logit \(x = 0\) and label \(y = 1\), compare the BCE computed as \(\operatorname{softplus}(x) - yx\) with the definition \(-\log\sigma(0)\)
- \(\operatorname{softplus}(0) - 0 = \log 2 \approx 0.693\). By the definition, \(\sigma(0) = 0.5\), so \(-\log 0.5 = \log 2\), the same value.
- 3. Diagnosis when the VAE's KL term is nearly 0 and the reconstruction is nearly the same blurry image no matter which image goes in
- This is posterior collapse. The encoder's \(q(z\mid x)\) has become equal to the prior \(\mathcal{N}(0, I)\), so the latent carries no information and the decoder draws only the average image. Lower β or ramp the KL weight up slowly from 0 (warm-up), use free bits, and if the decoder is too powerful (e.g., an autoregressive decoder), weaken it (chapters 8, 11, and 12).
- 4. The range that \(\gamma = 0.9\) covers in the pendulum environment with action repeat 2 (100 decisions per episode)
- \(\frac{1}{1 - 0.9} = 10\) steps, or about \(10 \times 2 \times 0.05 = 1\) second. Swinging the pendulum up takes a few seconds, so this may be too short. With \(\gamma = 0.99\), it is about 100 steps (10 seconds).
See also
- World Models:Tutorial
- World Models
- Recurrent World Models Facilitate Policy Evolution
- Reinforcement learning
- VAE
- Diffusion model
- Genie (world model)
- PyTorch
Favorite site
- 14. Appendix: Debugging, Glossary, Next Steps
- Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed.)
- Chua et al., PETS (2018)
- Ha & Schmidhuber, World Models (2018)
- Hafner et al., PlaNet (2019)
- Hafner et al., Dreamer (2020)
- Hafner et al., DreamerV2 (2021)
- Hafner et al., DreamerV3 (2023)
- Moerland et al., Model-based RL: A Survey
- danijar/dreamerv3