Skip to content
Search

WorldModels:Tutorial:14-Appendix

lastmod 2026-11-18created 2026-11-172 edits

Appendix: Debugging, Glossary, and Next Steps is chapter 14 of the World Model tutorial. It collects reference material to consult when you are building a World Model yourself and the loss turns into NaN, reconstructions come out blurry, or the agent does well in imagination but poorly in reality. It covers a debugging checklist and remedies by symptom, derivations of numerical stability formulas, hyperparameters from the original papers and from the tutorial, key equations by chapter, a Korean–English glossary, a reading order for the papers, public implementations, frequently asked questions, and follow-up projects.

Checklist for the first 30 minutes

A World Model has many parts (encoder, dynamics, decoder, reward, policy), and reinforcement learning results have high variance, so bugs are hard to locate. When something looks wrong, go through the steps below before touching any hyperparameters. Most bugs are caught in the first three steps.

  1. Look at the data. Save a few observation images and open them, and print the minimum, maximum, and mean of the values. Check whether images are in 0–255 or 0–1, and whether the action range is correct (the environment uses [−2, 2]; the agents in chapters 11 and 12 output [−1, 1]).
  2. Check shapes. Write the expected shape of each tensor in a comment and verify it with assert. Pay particular attention to the order of the time axis \(T\) and the batch axis \(B\), and to how observations (\(T+1\) of them) pair with actions (\(T\) of them).
  3. Overfit a tiny dataset. Training repeatedly on a single batch (e.g., 4 sequences) should drive the loss close to 0. If it does not, there is a bug in the model or the loss code.
  4. Log each loss term separately. If you only watch the sum of the reconstruction, reward, KL, actor, and critic losses, you cannot tell which term is the problem.
  5. Compare with baselines. Check that the model beats simple baselines such as a model that predicts "next state = current state" or the return of a random policy (the baselines from chapters 5 and 6).
  6. Log gradient magnitudes. A gradient norm of 0 means the gradient is cut off (a .detach() mistake); a sudden jump into the thousands means it is exploding.
  7. Fix the seed, then run several seeds. A good result from one seed may be luck. Look at the mean and range over 3 – 5 seeds.
  8. Only then change hyperparameters. One at a time, starting with the learning rate.

Here are the checks above as code. Put them in the training loop to catch problems early.

import random
import numpy as np
import torch

def set_seed(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)          # CPU · GPU 모두

def check(name, x, shape=None):
    """모양과 유한성(NaN · inf 없음)을 확인한다."""
    if shape is not None:
        assert tuple(x.shape) == tuple(shape), f"{name}: {tuple(x.shape)} != {shape}"
    assert torch.isfinite(x).all(), f"{name} 에 NaN 또는 inf 가 있습니다"

def grad_norm(model):
    """모든 매개변수 기울기를 한 벡터로 이었을 때의 길이."""
    total = 0.0
    for p in model.parameters():
        if p.grad is not None:
            total += p.grad.detach().pow(2).sum().item()
    return total ** 0.5

# 학습 루프 안에서
# check("obs", obs, (B, L, 64, 64))   # 11장의 배치: 길이 L, 채널 축 없음
# loss.backward()
# print("grad norm", grad_norm(model))
# torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=100.0)

# NaN 이 어디서 처음 생기는지 찾을 때만 켠다 (느려진다)
# torch.autograd.set_detect_anomaly(True)

The most common silent bug: broadcasting

If the prediction pred has shape (B, 1) and the target target has shape (B,), then (pred - target) ** 2 silently becomes a (B, B) matrix. Every prediction is compared with every target, so the model learns to predict only the mean. The loss still appears to go down, which makes the bug hard to notice. Check that both tensors have the same shape right before computing the loss.

Symptoms, causes, and remedies

This table collects common symptoms, grouped by component. The numbers under "Chapters" are the tutorial chapters that contain detailed explanations and derivations.

Data and shapes

Symptom

Common causes

Remedy

Chapters

The loss is absurdly large or small from the start

Inputs were not normalized, or were normalized twice. Images were fed in raw as 0–255

Print the input range. Images: /255 (−0.5); numeric states: standardization

06 · 08

The model always predicts roughly the same value (the mean)

Broadcasting turns the loss into (B, B). Inputs and targets are misaligned

Check shapes right before the loss; cross-check a few data points by hand

04

Predictions are shifted by one step

There are \(T{+}1\) observations but only \(T\) actions and rewards, and the indices are paired incorrectly

Draw out the pairing \((o_t, a_t) \to o_{t+1}\), \(r_t\) (see the description of collect_episodes)

06 · 11

Shape error in a convolutional layer

Images are (B, H, W, C) but PyTorch expects (B, C, H, W)

permute, or unsqueeze(1) (grayscale)

04 · 08

Predictions jump near angles of ±π

The angle \(\theta\) is used directly as input and output, so \(\pi\) and \(-\pi\) become numbers far apart

Represent it as \((\cos\theta, \sin\theta)\)

06

Numerical stability

Symptom

Common causes

Remedy

Chapters

The loss becomes NaN or inf

\(\log 0\), \(e^{x}\) overflow, zero standard deviation, learning rate too large

The stable formulas (softplus, logit BCE) in the section "Numerical stability formulas that prevent NaN" below, logsumexp from chapter 9, a minimum standard deviation, gradient clipping, set_detect_anomaly

02 · 09

The Gaussian NLL keeps decreasing without bound (toward −∞)

The predicted standard deviation goes to \(\sigma \to 0\), overfitting a few points

\(\sigma = \operatorname{softplus}(x) + \sigma_{\min}\); bound the range of \(\log\sigma^2\)

07 · 11

Sudden NaN while training the MDN

The \(\log\sum\exp\) over mixture weights is computed naively and overflows

Use torch.logsumexp; bound the range of \(\log\sigma\)

09

BCE is inf

Applying sigmoid first and then taking the \(\log\), which gives \(\log 0\)

binary_cross_entropy_with_logits, which computes directly from logits

08

Training dynamics

Symptom

Common causes

Remedy

Chapters

The loss does not decrease at all

Learning rate too small, gradient cut off (.detach(), torch.no_grad), parameters not passed to the optimizer

Small-data overfitting test, log the gradient norm, check optimizer.param_groups

04

The loss spikes and then diverges

Learning rate too large for the curvature (\(\eta \gt 2/a\)), exploding gradients in the RNN

Cut the learning rate to 1/3 at a time, gradient clipping

04 · 09

Training loss falls but results on new data get worse

Overfitting (too little data)

Collect more data, early stopping on validation loss, weight decay, a smaller model

06

Results vary widely between seeds

The high variance of reinforcement learning, luck in the initial data

Judge by the mean and range over several seeds; use more evaluation episodes

05

Latent variable models (VAE, RSSM)

Symptom

Common causes

Remedy

Chapters

KL sticks to 0 and reconstructions are always the same “average image”

Posterior collapse: the decoder ignores the latent; the KL weight is too large

Lower β, ramp the KL weight up slowly from 0 (warm-up), free nats / free bits

08 · 11 · 12

Reconstructions are blurry

MSE fits the average of the plausible images, insufficient decoder capacity, β too large

Enlarge the decoder, tune β, use a model that can represent several outcomes (MDN, stochastic latents, diffusion)

08 · 09 · 13

Reconstructions are good but imagined (prior) rollouts are a mess

Only the posterior is learned well; the transition (prior) does not keep up

Inspect open-loop imagined images, KL balancing, train longer, latent overshooting

11 · 12

Small objects (the pendulum rod) disappear from reconstructions

Most of the image is background, so the object contributes little to the loss

Increase the resolution and decoder capacity; always inspect output images by eye

08

KL stays large and does not decrease

The prior cannot catch up with the posterior; the standard deviation floor is too small

KL balancing (more weight on the prior side), adjust the minimum standard deviation

12

Prediction, planning, and policy

Symptom

Common causes

Remedy

Chapters

One-step predictions are good but multi-step predictions diverge

Compounding error (with a Lipschitz constant \(L \gt 1\) the error grows exponentially)

Train with a multi-step loss, predict the change \(\Delta o\), ensembles, a shorter imagination horizon

06

Imagined returns are high but real returns are low

Model exploitation: the policy exploits the model's errors

Ensemble uncertainty penalty, shorter imagination, collect more data, raise the dream temperature

10 · 12

MPC is too slow

The model runs (number of candidates × imagination length × iterations) times

Evaluate candidates as one batch on the GPU, use fewer candidates, start from the previous solution shifted by one step (warm start), shorten the horizon with a policy prior or value function

07 · 13

The actor's actions are always at ±max

tanh saturation; entropy shrinks too quickly

Increase the entropy bonus; check the action range and scale

12

The critic loss keeps growing

Large reward magnitudes; a constantly moving target

Scale rewards down (×0.1 in chapter 11), a slow (target) critic, symlog, two-hot, return normalization

11 · 12

Settings must be retuned for every task

A design that is sensitive to the scale of rewards and returns

DreamerV3's symlog and return normalization (percentiles)

12

Numerical stability formulas that prevent NaN

float32 cannot hold numbers larger than about \(3.4 \times 10^{38}\); they become inf. Since \(e^{89} \approx 4.5 \times 10^{38}\), exponentials overflow easily. Below are derivations of the formulas used internally by the stable functions in PyTorch.

Stable softplus

With the definition \(\operatorname{softplus}(x) := \log(1 + e^{x})\), a large \(x\) (e.g., \(x = 1000\)) makes \(e^x\) overflow to inf. Split into cases and factor out \(e^x\).

\(\begin{align} x \ge 0: \quad \log(1 + e^{x}) &= \log\big(e^{x}(e^{-x} + 1)\big) = x + \log(1 + e^{-x}) \\ x \lt 0: \quad \log(1 + e^{x}) &= 0 + \log(1 + e^{-\lvert x\rvert}) \end{align}\)

Combining the two cases into a single expression gives:

\(\operatorname{softplus}(x) = \max(x, 0) + \log\big(1 + e^{-\lvert x\rvert}\big)\)

The exponent is always 0 or less, so nothing overflows for any \(x\). For \(x = 1000\), it computes exactly \(1000 + \log(1 + e^{-1000}) = 1000\).

Binary cross-entropy from logits

When a network outputs a probability \(p = \sigma(x)\) as a logit \(x\), as in the Bernoulli decoder of chapter 8, computing \(\sigma(x)\) first and then taking the \(\log\) can round \(\sigma(x)\) to 0 and give \(\log 0 = -\infty\). The formula that works directly from the logit is derived in the following steps.

\(\begin{align} \log \sigma(x) &= \log\frac{1}{1 + e^{-x}} = -\log(1 + e^{-x}) = -\operatorname{softplus}(-x) \\ \log\big(1 - \sigma(x)\big) &= \log\frac{e^{-x}}{1 + e^{-x}} = \log\frac{1}{e^{x} + 1} = -\operatorname{softplus}(x) \\ \operatorname{softplus}(-x) &= \log(1 + e^{-x}) = \log\big(e^{-x}(e^{x} + 1)\big) = -x + \operatorname{softplus}(x) \end{align}\)

Substituting the first two results into the BCE definition from chapter 2, the signs cancel.

\(\text{BCE} = -\big[y \log\sigma(x) + (1 - y)\log(1 - \sigma(x))\big] = y\,\operatorname{softplus}(-x) + (1 - y)\,\operatorname{softplus}(x)\)

Substituting the third line and simplifying gives the result.

\(\text{BCE}(x, y) = y\big(\operatorname{softplus}(x) - x\big) + (1 - y)\operatorname{softplus}(x) = \operatorname{softplus}(x) - y\,x\)

  • There is nowhere for \(\log 0\) to appear, and because softplus is computed with the stable formula above, the result is finite for any logit. PyTorch's binary_cross_entropy_with_logits computes it this way.
  • Example: with \(x = 50\) and \(y = 0\), \(\sigma(50)\) is exactly 1 in float32, so \(\log(1 - 1) = -\infty\), but this formula gives \(\operatorname{softplus}(50) - 0 = 50\).

Gradient clipping

clip_grad_norm_(params, max_norm=c) multiplies the gradient vector \(g\) (the gradients of all parameters concatenated into one vector) by a single positive number.

\(g' = g \cdot \min\!\Big(1,\ \frac{c}{\lVert g \rVert}\Big)\)

  • If \(\lVert g \rVert \le c\), then \(c/\lVert g\rVert \ge 1\), so \(\min(\cdot) = 1\) and \(g' = g\).
  • If \(\lVert g \rVert \gt c\), then \(\lVert g' \rVert = \frac{c}{\lVert g \rVert}\,\lVert g \rVert = c\) (multiplying by a positive \(k\) scales the length by \(k\)).

\(\lVert g' \rVert = \min(\lVert g \rVert,\ c), \qquad \frac{g'}{\lVert g' \rVert} = \frac{g}{\lVert g \rVert}\)

Because the gradient is multiplied by a positive number, its direction is unchanged; only its length is capped at \(c\). Example: with \(g = (30, 40)\) and \(c = 10\), \(\lVert g\rVert = 50\) and \(g' = (30, 40) \times \frac{10}{50} = (6, 8)\). The descent direction stays correct; only steps that jump too far are shortened.

A standard deviation floor and the NLL floor

The 1D Gaussian NLL is the log density derived in chapter 2 with its sign flipped.

\(\text{NLL} = \frac{(y - \mu)^2}{2\sigma^2} + \log\sigma + \frac{1}{2}\log(2\pi)\)

The first term is nonnegative and equals 0 when the prediction is exact (\(y = \mu\)), so \(\text{NLL} \ge \log\sigma + \frac{1}{2}\log(2\pi)\). Since \(\log\) is an increasing function, a lower bound on \(\sigma\) gives the NLL a floor.

\(\sigma \ge \sigma_{\min} \;\Rightarrow\; \text{NLL} \ge \log\sigma_{\min} + \frac{1}{2}\log(2\pi)\)

  • Without the lower bound, the model can push \(\sigma \to 0\) at points it predicts correctly, sending \(\log\sigma \to -\infty\) and reducing the loss without bound.
  • With \(\sigma_{\min} = 0.1\), the floor is \(-2.303 + 0.919 = -1.384\).
  • min_logvar in chapter 7 and log_sigma.clamp(-7, 2) in chapter 9 serve this purpose. min_std = 0.1 in chapter 11 applies the same idea to the \(\sigma\) of the latent distribution.

Hyperparameter reference

Values from the original papers worked well at those papers' environments and scale, so treat them as starting points. Values can differ between versions of the same paper and its official code; if you need an exact reproduction, check the paper's appendix and the configuration files in the official code.

Values from the original papers

Method

Model

Training

Planning / behavior learning

Data collection

PETS
Chua et al., 2018
1805.12114

Ensemble of 5 probabilistic networks (3 fully connected layers, 500 units per layer (250 for HalfCheetah), Swish)

Gaussian NLL, upper and lower bounds on \(\log\sigma\)

CEM-MPC, trajectory sampling with 20 particles (the planning horizon, number of candidates, and number of elites are not given in the paper; the public code sets them per task)

Retrain the model every episode

World Models
Ha & Schmidhuber, 2018
1803.10122

VAE \(z\) 32-dim (64 for Doom), 64×64 color · MDN-RNN: LSTM 256 (512 for Doom), 5 Gaussians

VAE → MDN-RNN → controller, each trained separately in that order

Linear controller (867 parameters for CarRacing), CMA-ES population 64, 16 rollouts per individual · Doom dream training temperature \(\tau = 1.15\)

10,000 rollouts from a random policy

PlaNet
Hafner et al., 2019
1811.04551

RSSM: \(h\) 200 (GRU), \(z\) 30

Batch 50 × length 50, Adam learning rate \(10^{-3}\), \(\epsilon = 10^{-4}\), gradient clipping 1000, free nats 3

CEM: planning horizon 12, 10 iterations, 1000 candidates, 100 elites

5 seed episodes, 1 episode every 100 training steps, exploration noise \(\mathcal{N}(0, 0.3^2)\), action repeat 2 – 8

Dreamer (V1)
Hafner et al., 2020
1912.01603

Same RSSM as PlaNet (200 / 30)

Learning rates: model \(6 \times 10^{-4}\), actor and value \(8 \times 10^{-5}\); gradient clipping 100, free nats 3

Imagination horizon \(H = 15\), \(\gamma = 0.99\), \(\lambda = 0.95\), actor trained by backpropagating through the dynamics

Action repeat 2, exploration noise 0.3

DreamerV2 (Atari)
Hafner et al., 2021
2010.02193

32 categorical latents × 32 classes, GRU 600

KL balancing \(\alpha = 0.8\), KL weight 0.1 (1.0 for continuous control), learning rates: model \(2\times10^{-4}\) · actor \(4\times10^{-5}\) · critic \(10^{-4}\), batch 50 × length 50 (arXiv v4, appendix Table D.1)

\(H = 15\), \(\gamma = 0.999\) (the value given in Section 2.1 of the main text; the appendix table of the same paper lists 0.995), \(\lambda = 0.95\), entropy \(10^{-3}\) (\(10^{-4}\) for continuous control), REINFORCE gradients, slow critic updated every 100 steps

Standard Atari settings (frame skip 4)

DreamerV3
Hafner et al., 2023 · Nature 2025
2301.04104

Categorical latents 32 × 32 + 1% unimix, models in several sizes (from small up to hundreds of millions of parameters)

Free bits 1, KL weights \(\beta_{\text{rep}} = 0.1\) · \(\beta_{\text{dyn}}\) differs by version (arXiv v1, January 2023: 0.5 / revised v2, April 2024: 1), symlog · two-hot, batch 16 × 64, learning rates also differ by version (v1: model \(10^{-4}\), actor and critic \(3\times10^{-5}\); revision: \(4\times10^{-5}\))

\(H = 15\), discount horizon 333 (\(\gamma \approx 0.997\)), \(\lambda = 0.95\), entropy \(3\times10^{-4}\), return normalization (5th – 95th percentile), critic EMA

The key point is using the same settings across all domains

Settings in this tutorial

These values are scaled down to run in minutes to tens of minutes on a personal computer. They can be changed through the argparse defaults and Config in each exercise file.

Chapter · file

Key settings

06 · ch06_dynamics.py

50 random episodes, 100 epochs

07 · ch07_mpc.py

Planning horizon 25, 1000 random-shooting candidates, CEM 400 candidates · 40 elites · 5 iterations, 8 rounds of iterative training

07 · ch07_pets.py

Ensemble of 5 (hidden size 128), 200 candidates, 10 particles, planning horizon 25

08 · ch08_vae.py

\(z\) 32-dim, \(\beta = 1\), 200 episodes, 10 epochs

09 · ch09_mdn_rnn.py

LSTM 256, 5 Gaussians, 6000 steps

10 · ch10_world_models.py

Controller \(a = 2\tanh(W[z; h] + b)\) (289 parameters), CMA-ES 60 generations · population 32 · 4 rollouts · \(\sigma_0 = 0.1\), dream temperatures 0.1 · 1.0 · 1.5

11 · ch11_rssm_planet.py

\(h\) 200, \(z\) 30, batch 50 × 50, learning rate \(6\times10^{-4}\), gradient clipping 100, free nats 3, reward × 0.1, CEM 12 · 1000 · 10 · 100, action repeat 2, 5 seed episodes, exploration noise 0.3

12 · ch12_dreamer.py

Chapter 11 settings + \(H = 15\), \(\gamma = 0.99\), \(\lambda = 0.95\), actor and critic learning rate \(8\times10^{-5}\), entropy \(10^{-4}\), slow critic updated every 100 steps, actor and critic MLP 300 × 3

13 · ch13_diffusion_toy.py

100 noise steps, \(\beta\) 0.0001 → 0.15, MLP 256 × 3, 8000 steps

What to change first

  • When things are not working, the rough order of impact is ① the amount of data (episodes, training steps), ② the learning rate, ③ model size, ④ KL-related values (β, free nats), and ⑤ the imagination and planning horizons.
  • The discount factor \(\gamma\) determines how many steps ahead the agent looks (about \(\frac{1}{1-\gamma}\) steps, chapter 5), so match it to the time scale of the task. The pendulum in chapter 12 makes 100 decisions with action repeat 2, so \(\gamma = 0.99\) (about 100 steps) is enough.

Key equations by chapter

This table collects the key equations derived in the tutorial, in chapter order. The derivations are on each chapter's page, and the notation follows the conventions in chapter 00 of the tutorial.

Foundations (chapters 02 – 05)

Equation

Meaning

Chapter

\(p(y \mid x) = \dfrac{p(x \mid y)\,p(y)}{p(x)}\)

Bayes' theorem

02

\(p(x_{1:N}) = \prod_{i=1}^{N} p(x_i \mid x_{1:i-1})\)

Chain rule of probability (the product rule applied repeatedly)

02

\(\mathbb{E}[X] = \sum_y p(y)\,\mathbb{E}[X \mid y]\)

Law of total expectation (averaging over cases; used to derive the Bellman equation in chapter 5)

02

\(\operatorname{Var}[X] = \mathbb{E}[X^2] - (\mathbb{E}[X])^2\), \(\ \operatorname{Var}[aX + b] = a^2\operatorname{Var}[X]\)

Properties of variance

02

\(\log \mathcal{N}(x; \mu, \sigma^2) = -\tfrac12\log(2\pi) - \log\sigma - \dfrac{(x-\mu)^2}{2\sigma^2}\)

Gaussian log density

02

\(\hat\mu = \tfrac{1}{N}\sum_i x_i\), \(\ \hat\sigma^2 = \tfrac1N\sum_i (x_i - \hat\mu)^2\)

Gaussian maximum likelihood estimation

02

\(\operatorname*{arg\,min}_\theta \text{NLL} = \operatorname*{arg\,min}_\theta \text{MSE}\) (\(\sigma\) fixed)

MSE is the NLL of a fixed-variance Gaussian

02

\(\text{BCE} = -[y\log p + (1-y)\log(1-p)]\), \(\ \text{CE} = -l_y + \log\sum_j e^{l_j}\)

Bernoulli and categorical NLL

02

\(H(p) = -\sum_x p(x)\log p(x)\), \(\ H(p, q) = -\sum_x p(x)\log q(x)\)

Entropy, cross-entropy

03

\(D_{\mathrm{KL}}(p\,\Vert \,q) = H(p, q) - H(p) = \mathbb{E}_p\!\big[\log\tfrac{p}{q}\big] \ge 0\)

KL divergence and its properties

03

\(D_{\mathrm{KL}}\big(\mathcal{N}(\mu_1,\sigma_1^2)\,\Vert \,\mathcal{N}(\mu_2,\sigma_2^2)\big) = \log\tfrac{\sigma_2}{\sigma_1} + \dfrac{\sigma_1^2 + (\mu_1-\mu_2)^2}{2\sigma_2^2} - \tfrac12\)

KL between two Gaussians

03

\(f(\mathbb{E}[X]) \ge \mathbb{E}[f(X)]\) (\(f\) concave)

Jensen's inequality

03

\(\dfrac{dy}{dx} = \dfrac{dy}{du}\cdot\dfrac{du}{dx}\)

Chain rule (the basis of backpropagation)

04

\(\theta \leftarrow \theta - \eta\,\nabla_\theta L\), \(\ 0 \lt \eta \lt 2/a\)

Gradient descent and its convergence condition (curvature \(a\))

04

\(\sigma'(z) = \sigma(z)(1 - \sigma(z))\), \(\ \tanh'(z) = 1 - \tanh^2 z\)

Derivatives of activation functions

04

\(G_t = \sum_{k \ge 0}\gamma^k r_{t+k} = r_t + \gamma G_{t+1}\), \(\ \sum_{k\ge0}\gamma^k = \dfrac{1}{1-\gamma}\)

Discounted return and its recursion

05

\(V^\pi(s) = \sum_a \pi(a\mid s)\sum_{s'}p(s'\mid s,a)\big[r(s,a,s') + \gamma V^\pi(s')\big]\)

Bellman expectation equation

05

\(V^\ast(s) = \max_a \sum_{s'}p(s'\mid s,a)\big[r(s,a,s') + \gamma V^\ast(s')\big]\)

Bellman optimality equation (the basis of value iteration)

05

\(\nabla_\psi J = \mathbb{E}_{\tau}\big[R(\tau)\sum_t \nabla_\psi\log\pi_\psi(a_t\mid s_t)\big]\)

Policy gradient theorem (log-derivative trick, REINFORCE)

05

A first World Model (chapters 06 – 07)

Equation

Meaning

Chapter

\(I = \tfrac13 ml^2\), \(\ \ddot\theta = \dfrac{3g}{2l}\sin\theta + \dfrac{3}{ml^2}u\)

Moment of inertia and equation of motion of the pendulum

06

\(\dot\theta \leftarrow \dot\theta + \ddot\theta\,\Delta t\), \(\ \theta \leftarrow \theta + \dot\theta\,\Delta t\)

(Semi-implicit) Euler integration

06

\(\tilde x = (x - \mu)/\sigma\)

Standardization

06

\(e_k \le \varepsilon\,\dfrac{L^k - 1}{L - 1}\)

Upper bound on compounding error in an open-loop rollout

06

\(a^\ast_{t:t+H-1} = \operatorname*{arg\,max} \sum_{k=0}^{H-1} r(\hat o_{t+k}, a_{t+k})\), \(\ \hat o_{t+k+1} = f_\theta(\hat o_{t+k}, a_{t+k})\)

Planning objective of MPC

07

\(\mu \leftarrow \tfrac{1}{K}\sum_{k \in \text{elites}} a^{(k)}\), \(\ \sigma^2 \leftarrow \tfrac1K\sum_k (a^{(k)} - \mu)^2\)

CEM: refit the Gaussian to the elites (maximum likelihood)

07

\(\mathcal{L}(\mu, s) = (y - \mu)^2 e^{-s} + s\), \(\ s = \log\sigma^2\)

Gaussian NLL that also predicts the variance (constants and ½ omitted, PETS)

07

\(\operatorname{Var}[y] = \tfrac1E\sum_e \sigma_e^2 + \tfrac1E\sum_e(\mu_e - \bar\mu)^2\)

Ensemble uncertainty = aleatoric (mean of the variances) + epistemic (spread of the means)

07

World Models that see images (chapters 08 – 10)

Equation

Meaning

Chapter

\(p_\theta(x) = \int p_\theta(x\mid z)\,p(z)\,dz\)

Marginal likelihood of a latent variable model

08

\(\log p_\theta(x) \ge \mathbb{E}_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)] - D_{\mathrm{KL}}(q_\phi(z\mid x)\,\Vert \,p(z))\)

ELBO

08

\(\log p_\theta(x) = \text{ELBO} + D_{\mathrm{KL}}(q_\phi(z\mid x)\,\Vert \,p_\theta(z\mid x))\)

Gap between the ELBO and the true log-likelihood

08

\(z = \mu + \sigma\odot\epsilon\), \(\ \epsilon\sim\mathcal{N}(0, I)\)

Reparameterization trick

08

\(D_{\mathrm{KL}}(\mathcal{N}(\mu,\sigma^2)\,\Vert \,\mathcal{N}(0,1)) = \tfrac12(\mu^2 + \sigma^2 - 1 - \log\sigma^2)\)

KL term of the VAE (summed over dimensions)

08

\(\dfrac{\partial L_T}{\partial h_k} = \dfrac{\partial L_T}{\partial h_T}\prod_{t=k+1}^{T}\operatorname{diag}(1 - h_t^2)\,W_h\)

Gradient from the distant past in an RNN (the cause of vanishing and exploding gradients)

09

\(\log p(y\mid x) = \operatorname{logsumexp}_k\big(\log\pi_k + \log\mathcal{N}(y;\mu_k,\sigma_k^2)\big)\)

Log-likelihood of a mixture density network

09

\(\log\sum_k e^{a_k} = m + \log\sum_k e^{a_k - m}\), \(\ m = \max_k a_k\)

log-sum-exp stabilization

09

\(a_t = 2\tanh(W_c[z_t; h_t] + b_c)\)

The controller in this tutorial (Ha's linear controller \(W_c[z_t; h_t] + b_c\) wrapped in \(2\tanh\) to match the torque range)

10

\(\nabla_\theta\,\mathbb{E}_\epsilon[F(\theta + \sigma\epsilon)] = \tfrac{1}{\sigma}\mathbb{E}_\epsilon[F(\theta + \sigma\epsilon)\,\epsilon]\)

Gradient estimate of evolution strategies (Gaussian smoothing)

10

\(C_\mu = \sum_{i=1}^{\mu} w_i\, y_{i:\lambda}\, y_{i:\lambda}^\top\)

CMA-ES: refit the covariance to the successful steps (weighted maximum likelihood)

10

Learning in imagination and modern models (chapters 11 – 13)

Equation

Meaning

Chapter

\(h_t = f(h_{t-1}, z_{t-1}, a_{t-1})\), \(\ z_t \sim p(z_t\mid h_t)\), \(\ z_t \sim q(z_t\mid h_t, o_t)\)

Deterministic, prior, and posterior paths of the RSSM

11

\(\sum_t \Big(\mathbb{E}_q[\log p(o_t\mid h_t,z_t)] + \mathbb{E}_q[\log p(r_t\mid h_t,z_t)] - \mathbb{E}_q\big[D_{\mathrm{KL}}(q(z_t\mid h_t,o_t)\,\Vert \,p(z_t\mid h_t))\big]\Big)\)

Sequential ELBO

11

\(\max(D_{\mathrm{KL}}, c)\)

Free nats

11

\(V^\lambda_t = \hat r_t + \gamma\big[(1-\lambda)\,v(s_{t+1}) + \lambda\,V^\lambda_{t+1}\big]\)

Recursion of the λ-return

12

\(\tfrac12\big(v_\xi(s_t) - \operatorname{sg}(V^\lambda_t)\big)^2\)

Critic loss

12

\(z = \operatorname{onehot} + p - \operatorname{sg}(p)\) (value = one-hot, \(\partial\mathcal{L}/\partial p = \partial\mathcal{L}/\partial z\))

Straight-through estimator

12

\(\alpha\,D_{\mathrm{KL}}(\operatorname{sg}(q)\,\Vert \,p) + (1-\alpha)\,D_{\mathrm{KL}}(q\,\Vert \,\operatorname{sg}(p))\)

KL balancing (DreamerV2, \(\alpha = 0.8\))

12

\(\operatorname{symlog}(x) = \operatorname{sign}(x)\log(\lvert x\rvert + 1)\), \(\ \operatorname{symexp}(x) = \operatorname{sign}(x)(e^{\lvert x\rvert} - 1)\)

symlog and its inverse

12

\(w_{k+1} = \dfrac{y - b_k}{b_{k+1} - b_k}\), \(\ w_k = \dfrac{b_{k+1} - y}{b_{k+1} - b_k}\)

two-hot weights

12

\(z_t = \sum_{i=0}^{n-1}\gamma^i r_{t+i} + \gamma^n v(s_{t+n})\)

n-step value target (MuZero). Here \(z_t\) is the value target, not a latent variable

13

\(Q + P\,\dfrac{\sqrt{\sum_b N}}{1 + N}\big(c_1 + \log\tfrac{\sum_b N + c_2 + 1}{c_2}\big)\)

pUCT score in MCTS

13

\(q^\ast \propto p\,\exp(J/\tau)\), \(\ w_i = \dfrac{\exp(J_i/\tau)}{\sum_j\exp(J_j/\tau)}\)

MPPI weights (\(\tau\) is the temperature)

13

\(\operatorname{softmax}\!\big(QK^\top/\sqrt{d}\big)V\), \(\ \operatorname{Var}(q\cdot k) = d\)

Attention, and why it divides by \(\sqrt d\)

13

\(x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon\)

Closed form of the forward diffusion process

13

\(\tilde\mu_t = \dfrac{1}{\sqrt{\alpha_t}}\Big(x_t - \dfrac{\beta_t}{\sqrt{1-\bar\alpha_t}}\epsilon\Big)\), \(\ \tilde\beta_t = \dfrac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_t\)

Mean and variance of the true reverse distribution

13

\(\mathbb{E}\big\lVert\epsilon - \epsilon_\theta(\sqrt{\bar\alpha_t}x_0 + \sqrt{1-\bar\alpha_t}\epsilon,\ t)\big\rVert^2\)

Training loss of a diffusion model

13

\(\bar\theta \leftarrow \tau\bar\theta + (1-\tau)\theta\)

EMA target encoder (JEPA, TD-MPC). Here \(\tau\) is the mixing rate

13

Glossary

Terms used in the (Korean-language) tutorial, listed in Korean alphabetical order. "Chapter" is the chapter that first defines and explains the term in detail (chapters where only the name appears earlier are not listed).

Korean terms

Korean term

English

Meaning

Chapter

가능도

likelihood

The probability (density) of the observed data with the parameters held fixed, viewed as a function of the parameters

Ch. 2

가우시안 분포 (정규 분포)

Gaussian / normal distribution

The bell-shaped distribution N(μ, σ²) with mean μ and variance σ²

Ch. 2

가우시안 평활화

Gaussian smoothing

A technique that optimizes E[F(θ + σε)] instead of F so that the gradient can be estimated from samples

Ch. 10

가치 등가

value equivalence

The World Model design principle that only the quantities used for planning (reward, value, policy) need to match reality (MuZero)

Ch. 13

가치 반복

value iteration

A method that finds the optimal value by repeatedly applying the Bellman optimality equation

Ch. 5

가치 함수

value function

The expected return V(s) when following a policy from state s

Ch. 5

강화학습

reinforcement learning, RL

Learning, by trial and error, the actions that maximize reward

Ch. 5

결합 확률

joint probability

The probability p(x, y) that several events occur together

Ch. 2

경사 하강법

gradient descent

A method that reduces the loss by moving the parameters a little at a time in the direction opposite to the gradient

Ch. 4

계획

planning

Using a model to think through the future before choosing an action

Ch. 5

곡률

curvature

How sharply the loss surface bends. The larger it is, the smaller the learning rate must be

Ch. 4

공통 난수

common random numbers

A technique that evaluates candidates with the same seed and the same initial state to reduce noise in the comparison

Ch. 10

과적합

overfitting

Fitting the training data well but getting new data wrong

Ch. 4 · 6

관성 모멘트

moment of inertia

How hard something is to rotate. ml²/3 for a rod rotating about one end

Ch. 6

관측

observation

What the agent actually sees, o_t (a numeric vector or an image)

Ch. 1

교차 엔트로피

cross-entropy

The average information when data from p is encoded using q, H(p, q) = −Σ p log q

Ch. 3

궤적

trajectory

States, actions, and rewards laid out in time order, τ

Ch. 5

궤적 표본 전파

trajectory sampling, TS

PETS's method of propagating uncertainty into the future by sampling an ensemble member and noise for each particle

Ch. 7

기댓값

expectation

The probability-weighted average E[f(x)]

Ch. 2

기울기 소실

vanishing gradient

The gradient approaching 0 as it flows back through layers or time, so that learning stalls

Ch. 4

기울기 자르기

gradient clipping

A stabilization technique that shrinks the gradient vector to length c when its length exceeds c

Ch. 9

기울기 차단

stop-gradient, sg

An operation that uses a value as is but lets no gradient flow through it (.detach())

Ch. 12

기준선

baseline

A value subtracted from the return to reduce the variance of the policy gradient. Subtracting it leaves the expectation unchanged

Ch. 5

꿈

dream

Imagined trajectories generated by the World Model itself. Policies are sometimes trained inside them

Ch. 9

노출 편향

exposure bias

The mismatch that arises from receiving true inputs during training but the model's own outputs as inputs during generation

Ch. 9

다루기 어렵다

intractable

So expensive to compute exactly that it effectively cannot be computed

Ch. 8

다봉 분포

multimodal distribution

A distribution with several peaks. Its mean may resemble none of the peaks

Ch. 1

다양체 가설

manifold hypothesis

The assumption that high-dimensional data (images) actually lies near a low-dimensional surface

Ch. 8

다층 퍼셉트론

multilayer perceptron, MLP

The most basic neural network, built by alternating linear layers and activation functions

Ch. 4

대각 가우시안

diagonal Gaussian

A multivariate Gaussian whose dimensions are independent. Its covariance matrix is diagonal

Ch. 2

독립

independent

p(x, y) = p(x) p(y). Knowing one variable gives no information about the other

Ch. 2

드리프트

drift

Scenes gradually going wrong as errors accumulate while the model keeps generating from its own outputs

Ch. 13

라그랑주 승수

Lagrange multiplier

A variable added to turn a constrained optimization into an unconstrained problem

Ch. 13

로그 미분 트릭

log-derivative trick

A method that uses ∇p = p ∇log p to estimate the gradient of an expectation from samples

Ch. 5

로짓

logit

A score before softmax or sigmoid is applied (an unnormalized log probability)

Ch. 2

리턴

return

The sum of (discounted) future rewards, G_t

Ch. 5

리플레이 버퍼

replay buffer

Storage that holds the experience (observations, actions, rewards) collected by the agent, sampled at random during training. Denoted 𝒟

Ch. 11

립시츠 상수

Lipschitz constant

An upper bound L on how many times larger a difference in the input can become in the output

Ch. 6

마르코프 결정 과정

Markov decision process, MDP

A decision problem defined by states, actions, transition probabilities, rewards, and a discount factor

Ch. 5

마르코프 성질

Markov property

The property that the next state depends only on the current state and action, not on anything earlier

Ch. 5

매개변수

parameter

The numbers in a neural network that training adjusts (weights and biases)

Ch. 4

모델 기반 강화학습

model-based RL

Reinforcement learning that learns a model of the environment (a World Model) and uses it for planning or imagination

Ch. 1

모델 없는 강화학습

model-free RL

Reinforcement learning that learns a policy or value directly from experience, without a model

Ch. 1

모델 예측 제어

model predictive control, MPC

A method that plans ahead with the model at every step, executes only the first action, and then replans

Ch. 7

모델 착취

model exploitation

A policy exploiting errors in the World Model to score highly only in imagination

Ch. 1

목표 신경망

target network

A slowly following copy of a network (periodic copy or EMA) used to compute training targets. It keeps the target from shifting at every step. Examples: Dreamer's slow critic, the target encoder in TD-MPC and JEPA

Ch. 5 · 12

몬테카를로 추정

Monte Carlo estimation

Approximating an expectation by drawing samples and averaging them

Ch. 2

미니배치

minibatch

A small random subset drawn at each step instead of the full dataset. Its gradient is an unbiased estimate of the full gradient

Ch. 4

미분

derivative

The rate at which the output changes when the input changes very slightly

Ch. 4

미분 방정식

differential equation

An equation relating a value to its rate of change (derivative). Example: the pendulum's equation of motion

Ch. 6

믿음 상태

belief state

The probability distribution b_t(s) over the current state under partial observability

Ch. 5

밴딧

bandit

The simplest reinforcement learning problem: choosing actions with no state

Ch. 5

베르누이 분포

Bernoulli distribution

A distribution that outputs 0 or 1, with probability p of outputting 1

Ch. 2

베이즈 정리

Bayes' theorem

p(y | x) = p(x | y) p(y) / p(x). Reverses a conditional probability between cause and effect

Ch. 2

벨만 방정식

Bellman equation || The recursion value = current reward + discounted value of the next state

Ch. 5

벨만 최적 방정식

Bellman optimality equation

Optimal value = (reward + discounted next value) of the best action

Ch. 5

변분 추론

variational inference

A method that approximates an intractable posterior with a tractable distribution q

Ch. 8

보상

reward

The score r_t that the environment gives for an action

Ch. 1

보상 모델

reward model

A model that predicts the reward from the state and action

Ch. 1

보편 근사 정리

universal approximation theorem

The theorem that a sufficiently wide neural network can imitate any continuous function as closely as desired

Ch. 4

볼록

convex

A function shape in which the line segment joining any two points lies above the graph (e.g., x²)

Ch. 3

부분 관측

partial observability

A situation where a single observation does not reveal the whole state. A single image does not show the angular velocity

Ch. 1

부트스트랩

bootstrap

① Statistics: resampling the data with replacement so that each model gets slightly different training data (chapter 7) ② Reinforcement learning: building training targets from one's own value estimates (TD targets, chapters 5 and 12)

Ch. 7 · 12

분리 표현

disentanglement

A representation in which each component of the latent variable corresponds to a single meaningful factor

Ch. 8

분산

variance

How spread out values are around the mean, Var[X] = E[(X − E[X])²]

Ch. 2

분할 상환 추론

amortized inference

An approach in which a single network (the encoder) outputs q(z | x) directly instead of optimizing separately for each data point

Ch. 8

블랙박스 최적화

black-box optimization

Optimization that finds an optimum by looking only at function values, without gradients

Ch. 10

사람 기준 정규화 점수

human-normalized score, HNS

(agent − random)/(human − random). Used to compare Atari results

Ch. 13

사전 확률

prior

The belief before seeing the data, p(z)

Ch. 2

사후 붕괴

posterior collapse

The VAE's q(z | x) becomes identical to the prior, so the latent variable carries no information

Ch. 8

사후 확률 (사후 분포)

posterior

The belief after seeing the data, p(z | x)

Ch. 2

상태

state

The true situation of the environment, s_t. For the pendulum, (θ, θ̇)

Ch. 1

상태 공간 모델

state-space model, SSM

A model in which a hidden state changes over time and observations are generated from that state

Ch. 11

상호정보량

mutual information

How much knowing one variable reduces uncertainty about another, I(X; Y)

Ch. 3

선형 층

linear layer || y = Wx + b. A fully connected layer

Ch. 4

소프트맥스

softmax

A function that exponentiates scores and turns them into probabilities that sum to 1

Ch. 2

손실 함수

loss function

A value that measures how wrong a prediction is. Training reduces it

Ch. 4

순환 신경망

recurrent neural network, RNN

A neural network that processes a sequence one step at a time while carrying a hidden state

Ch. 9

스트라이드

stride

How many cells the convolution kernel moves at a time. With 2, the output size is roughly halved

Ch. 4

신경망

neural network

A function built by composing many layers of simple parameterized functions (layers)

Ch. 4

앙상블

ensemble

Training several models of the same architecture separately and using them together. Their disagreement measures uncertainty

Ch. 7

액터

actor

The policy network π_ψ that chooses actions

Ch. 5

어텐션

attention

An operation that mixes values according to the similarity between queries and keys (softmax weights). The core of the Transformer

Ch. 13

에너지

energy

A value that measures how poorly two inputs match. In JEPA, the distance between representations

Ch. 13

에이전트

agent

The entity that looks at observations and chooses actions

Ch. 1

에폭

epoch

One full pass over the training data

Ch. 6

에피소드

episode

One attempt from start to finish. 200 steps for the pendulum

Ch. 1

엔트로피

entropy

The average information (uncertainty) of a distribution, H(p) = −Σ p log p

Ch. 3

역전파

backpropagation

A method that uses the chain rule to compute the gradients of all parameters backward from the loss at the output

Ch. 4

열린 고리 롤아웃

open-loop rollout

Rolling forward from only the initial state by feeding the model's own predictions back in

Ch. 6

오목

concave

A function shaped the opposite way from a convex one (e.g., log x)

Ch. 3

오차 누적

compounding error

Small one-step errors accumulating and growing during multi-step prediction

Ch. 1

오토인코더

autoencoder, AE

A neural network trained to compress its input into a small code and then reconstruct it

Ch. 8

온도

temperature

A value τ that sets how flat (random) a distribution is

Ch. 9

우연적 불확실성

aleatoric uncertainty

Randomness in the world itself. It does not shrink with more data (the mean of the variances)

Ch. 7

은닉 상태

hidden state

The memory h_t that an RNN carries along

Ch. 9

음의 로그 가능도

negative log-likelihood, NLL

−log p(data). Maximum likelihood estimation = NLL minimization

Ch. 2

이점

advantage

A(s, a) = Q(s, a) − V(s). How much better this action is than average

Ch. 5

인과 마스크

causal mask

Masking scores with −∞ so that the Transformer cannot see future tokens

Ch. 13

인식적 불확실성

epistemic uncertainty

Not knowing because of insufficient data. It shrinks as data grows (the spread of the ensemble means)

Ch. 7

일반화

generalization

Predicting well on new data that was not used in training

Ch. 4

자기회귀

autoregressive

Building a sequence by repeatedly predicting the next item from the ones before it

Ch. 13

자동 미분

automatic differentiation, autograd

A feature that records the computation and then computes gradients automatically with the chain rule

Ch. 4

자유 내트

free nats

A technique that stops learning from the KL term while the KL is below a set value, max(KL, c)

Ch. 11

잠재 과대 예측

latent overshooting

An auxiliary objective in PlaNet that also fits priors several steps ahead to the posteriors

Ch. 11

잠재 변수

latent variable

A hidden variable z that is not directly observed and explains the data

Ch. 1

잠재 행동 모델

latent action model, LAM

A model that learns discrete actions explaining frame-to-frame changes from video without action labels (Genie)

Ch. 13

재매개변수화 트릭

reparameterization trick || A method that writes z = μ + σ·ε so that gradients can flow past the sampling step

Ch. 8

전이 모델

transition model, dynamics model

A model that predicts the next state from the current state and action

Ch. 1

전체 기댓값의 법칙

law of total expectation

Splitting into cases and adding up each case's expectation weighted by its probability gives the overall expectation. E[X] = Σ_y p(y) E[X | y]

Ch. 2

전치 합성곱

transposed convolution

A convolution that enlarges the image. Used in the VAE decoder

Ch. 4 · 8

정보 병목

information bottleneck

A design that deliberately narrows the information that can pass through, so only essential information remains

Ch. 13

정보량

information content, surprisal

The surprise −log p(x), larger for rarer events

Ch. 3

정신 모형

mental model

A scaled-down model of the world that a person holds in their head

Ch. 1

정예

elite

The top-scoring candidates in CEM

Ch. 7

정책

policy

The rule π(a | s) for choosing an action from the state (observation)

Ch. 1

젠센 부등식

Jensen's inequality

For a concave function f, f(E[X]) ≥ E[f(X)]. The key step in deriving the ELBO

Ch. 3

조건부 기댓값

conditional expectation

The expectation E[X | Y = y] given a condition. It is the prediction that minimizes MSE

Ch. 2

조건부 확률

conditional probability

The probability of x given y, p(x | y) = p(x, y)/p(y)

Ch. 2

주변 확률

marginal probability

The probability of a single variable, obtained by summing (integrating) all other variables out of the joint probability

Ch. 2

주변화

marginalization

Removing variables of no interest by summing (integrating) them out, p(x) = Σ_y p(x, y)

Ch. 2

증거

evidence

The denominator p(x) in Bayes' theorem: the probability of the observation summed over all causes

Ch. 2

지도 학습

supervised learning

Learning an input → answer function from pairs of inputs and correct answers

Ch. 4

직선 통과 추정기

straight-through estimator

A trick that passes discrete values forward and passes gradients backward unchanged

Ch. 12

진화 경로

evolution path

A vector in which CMA-ES accumulates past mean shifts, used to adjust the step size and covariance

Ch. 10

진화 전략

evolution strategies, ES

Black-box optimization that sets the next distribution from the scores of candidates made by adding noise to the parameters

Ch. 10

최대우도 추정

maximum likelihood estimation, MLE

A method that chooses the parameters that make the likelihood of the data as large as possible

Ch. 2

커널 (필터)

kernel, filter

A small set of weights that slides over the image and is multiplied in during a convolution

Ch. 4

코드북

codebook

The list of vectors {e_1, …, e_K} that a VQ-VAE can choose from

Ch. 13

크리틱

critic

A network v_ξ that estimates the value of a state

Ch. 5

큰 수의 법칙

law of large numbers

The more samples there are, the closer the sample mean gets to the expectation

Ch. 2

텐서

tensor

An array of numbers laid out along several axes. PyTorch's basic data type

Ch. 4

토크

torque

A force that turns something around an axis. The pendulum's action u

Ch. 6

토큰

token

A single discrete symbol (an integer) that the model handles. Images also become token sequences via VQ

Ch. 13

패딩

padding

Values (usually 0) added around the border of the image before convolution. Controls the output size

Ch. 4

편미분

partial derivative

Differentiating with respect to one variable while holding the others fixed

Ch. 4

표준화

standardization

Preprocessing with (x − mean)/standard deviation to give mean 0 and standard deviation 1

Ch. 4 · 6

표현 붕괴

representation collapse

A failure in which all inputs collapse into the same representation and only the loss gets smaller

Ch. 13

필터링

filtering

Estimating the distribution of the current state from the observations so far

Ch. 11

할인율

discount factor

A value γ (0 – 1) that sets how much future rewards are discounted

Ch. 5

합성 함수

composite function

A function inside a function, f(g(x)). What the chain rule applies to

Ch. 4

합성곱 신경망

convolutional neural network, CNN

A neural network that slides small filters over an image to extract local features

Ch. 4

행동

action

What the agent does to the environment, a_t. For the pendulum, the torque u

Ch. 1

행동 반복

action repeat

A technique that repeats the same action for k steps to reduce the number of decisions

Ch. 11

혼합 밀도 네트워크

mixture density network, MDN

A neural network that outputs the weights, means, and standard deviations of a Gaussian mixture

Ch. 9

확률밀도함수

probability density function, PDF

The probability density of a continuous variable. Integrating it over an interval gives a probability

Ch. 2

확률변수

random variable

A variable whose value is determined by the outcome of an experiment

Ch. 2

확률적 경사 하강법

stochastic gradient descent, SGD

Gradient descent that updates the parameters with gradients estimated from minibatches

Ch. 4

확률질량함수

probability mass function, PMF

The probability that a discrete variable takes each value

Ch. 2

확산 모델

diffusion model

A generative model that learns to reverse the process of adding noise to data, producing data from noise

Ch. 13

환경

environment

The world the agent interacts with. It takes an action and returns the next observation and reward

Ch. 1

활성 함수

activation function

A nonlinear function placed between layers (ReLU, tanh, SiLU …)

Ch. 4

후퇴 지평

receding horizon

Planning H steps ahead at every step, executing only the first action, then sliding the window forward by one and replanning

Ch. 7

1차 테일러 근사

first-order Taylor approximation

f(x + Δ) ≈ f(x) + f'(x)Δ. The reason gradient descent reduces the loss

Ch. 4

KL 균형

KL balancing

A DreamerV2 technique that splits the KL gradient between the prior side (α) and the posterior side (1−α)

Ch. 12

KL 발산

Kullback–Leibler divergence

How different two distributions are, KL(p‖q) = Σ p log(p/q) ≥ 0

Ch. 3

n-스텝 리턴

n-step return

n real rewards + γⁿ × (the value estimate n steps later)

Ch. 12

β-VAE

beta-VAE

A VAE that multiplies the KL term by β to balance reconstruction against compression

Ch. 8

λ-리턴

λ-return

A value target that mixes several n-step returns with weights (1−λ)λ^(n−1)

Ch. 12

English abbreviations and names

Abbreviation / English term

Meaning

Chapter

Adam

An adaptive learning-rate optimizer. Adjusts the step using moving averages of the gradient and its magnitude

Ch. 4

BPTT

backpropagation through time. Backpropagation through an RNN unrolled over time

Ch. 9

CEM

cross-entropy method. An optimizer that repeatedly samples candidates and refits a Gaussian to the top ones (the elites)

Ch. 7

CMA-ES

covariance matrix adaptation evolution strategy. An evolution strategy that also adapts the covariance

Ch. 10

DDPM

denoising diffusion probabilistic model. A diffusion model trained by predicting noise (Ho et al., 2020)

Ch. 13

Dyna

An architecture that learns a model from real experience and also trains values and policies on simulated experience generated by the model (Sutton, 1990 – 1991)

Ch. 1

ELBO

evidence lower bound. A lower bound on log p(x) = reconstruction − KL

Ch. 8

EMA

exponential moving average. A slowly following average, θ̄ ← τθ̄ + (1−τ)θ

Ch. 4 · 13

GRU

gated recurrent unit. An RNN that controls its memory with two gates (update and reset)

Ch. 9

JEPA

joint-embedding predictive architecture. An architecture that predicts in representation space instead of pixel space

Ch. 13

log-sum-exp

A way to compute log Σ exp(a_k) stably by subtracting the maximum

Ch. 2

LSTM

long short-term memory. An RNN that keeps long-term memory with a cell state and three gates

Ch. 9

MCTS

Monte Carlo tree search. A search that digs down promising branches one step at a time (MuZero)

Ch. 13

MDP

Markov decision process. A decision problem defined by states, actions, transitions, rewards, and a discount factor

Ch. 5

MPPI

model predictive path integral. A planning method that averages candidates with exp(J/τ) weights

Ch. 13

MSE

mean squared error. The mean of the squared errors

Ch. 4

PETS

probabilistic ensembles with trajectory sampling. Probabilistic ensemble + trajectory sampling + CEM-MPC

Ch. 7

PlaNet

deep planning network. An agent that plans with CEM in the RSSM latent space

Ch. 11

POMDP

partially observable MDP. An MDP in which observations alone do not reveal the whole state

Ch. 5

pUCT

A score in MCTS that mixes exploitation (Q) and exploration (prior probability and visit count)

Ch. 13

REINFORCE

A policy gradient method that differentiates the policy directly via ∇J = E[∇ log π(a|s) · G]

Ch. 5

RSSM

recurrent state-space model. Latent dynamics that combine a deterministic path h_t and a stochastic path z_t

Ch. 11

SiLU (Swish)

x · sigmoid(x). A smooth, ReLU-like activation function

Ch. 6

SimNorm

A TD-MPC2 technique that normalizes the latent vector with a softmax over each group

Ch. 13

softplus

log(1 + eˣ). An always-positive smooth ReLU. Used to produce standard deviations

Ch. 7

symlog

sign(x) log(|x| + 1). A transform that squashes values of widely varying magnitude (DreamerV3)

Ch. 12

TD error

temporal difference error. δ = r + γV(s') − V(s). Its expectation is the advantage

Ch. 5

two-hot

A categorical encoding that splits a single real number's weight between two neighboring bins (DreamerV3)

Ch. 12

unimix

A technique that mixes a little uniform distribution (1% in DreamerV3) into a categorical distribution so that no probability is exactly 0

Ch. 12

VAE

variational autoencoder. A latent variable model and an encoder (variational inference) trained together with the ELBO

Ch. 8

VQ-VAE

vector-quantized VAE. Turns latents into tokens by replacing each with the nearest codebook vector

Ch. 13

Paper map

Rather than reading a paper from start to finish, read ① the abstract and Figure 1, ② the key equations in the method section, and ③ the main tables in the experiments, in that order, and go back to the corresponding tutorial chapter for any equation you do not follow. The order below follows the flow of the tutorial.

Order

Paper

Why read it

Chapters

0

Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed.), chapters 3 – 4 and 8

MDPs and the Bellman equations, plus Dyna (the classic of model-based reinforcement learning)

05

1

Kingma & Welling, “Auto-Encoding Variational Bayes”, 1312.6114

The original source of the VAE and the reparameterization trick

08

2

Graves, “Generating Sequences With Recurrent Neural Networks”, 1308.0850

How to generate sequences with an MDN-RNN

09

3

Ha & Schmidhuber, “World Models”, 1803.10122 (+ interactive article)

The V · M · C architecture and learning inside dreams. Easy to read, with good figures

10

4

Chua et al., “PETS”, 1805.12114

Probabilistic ensembles and uncertainty-aware MPC

07

5

Janner et al., “MBPO: When to Trust Your Model”, 1906.08253

Using short model rollouts to help model-free learning; an analysis of compounding error

06 · 07

6

Salimans et al., “Evolution Strategies”, 1703.03864 · Hansen, “CMA-ES Tutorial”, 1604.00772

Evolution strategies that find policies without gradients

10

7

Hafner et al., “PlaNet”, 1811.04551

The RSSM and planning in latent space

11

8

Hafner et al., “Dreamer”, 1912.01603

Actor-critic in imagination and the λ-return

12

9

Hafner et al., “DreamerV2”, 2010.02193 · “DreamerV3”, 2301.04104

Categorical latents and KL balancing, plus a single set of settings across many domains

12

10

Schrittwieser et al., “MuZero”, 1911.08265

Value-equivalent models and MCTS

13

11

Hansen et al., “TD-MPC2”, 2310.16828

The current form of decoder-free latent planning

13

12

Micheli et al., “IRIS”, 2209.00588

Tokens + a Transformer World Model

13

13

Ho et al., “DDPM”, 2006.11239 → Alonso et al., “DIAMOND”, 2405.12399

Diffusion models and diffusion World Models

13

14

LeCun, “A Path Towards Autonomous Machine Intelligence” → “V-JEPA 2”, 2506.09985

A different path: prediction in representation space

13

15

Bruce et al., “Genie”, 2402.15391 → Hafner et al., “Dreamer 4”, 2509.24527

Large-scale video World Models and training agents inside them

13

+

Moerland et al., “Model-based RL: A Survey”, 2006.16712

A survey covering model-based reinforcement learning as a whole. Useful when you want to redraw the map

01

Public implementations and tools

Once you have learned the principles from the tutorial's small implementations, start from well-tested public implementations when you take on larger tasks.

Implementation

Description

danijar/dreamerv3

Official DreamerV3 implementation (JAX). There are also dreamerv2 and dreamer repositories

NM512/dreamerv3-torch

Community PyTorch reimplementation of DreamerV3. Useful if you want to read it in PyTorch, like this tutorial

google-research/planet

Official PlaNet implementation (TensorFlow)

hardmaru/WorldModelsExperiments

Experiment code for Ha & Schmidhuber's World Models

facebookresearch/mbrl-lib

Model-based reinforcement learning library including PETS and MBPO (PyTorch)

nicklashansen/tdmpc2

Official TD-MPC2 implementation and trained checkpoints

werner-duvaud/muzero-general

Readable community PyTorch reimplementation of MuZero

eloialonso/iris · eloialonso/diamond

Official implementations of the IRIS (Transformer) and DIAMOND (diffusion) World Models

facebookresearch/vjepa2

V-JEPA 2 code and pretrained models

nvidia-cosmos

NVIDIA Cosmos World Foundation Model repositories

Gymnasium · DeepMind Control Suite · MuJoCo

Experiment environments. This tutorial's pendulum has the same physics as Gymnasium's Pendulum-v1, and PlaNet and Dreamer were evaluated on the DeepMind Control Suite

Porting the tutorial code to Gymnasium environments

  • The environments in wm_envs.py return three values, obs, reward, done = env.step(action), whereas Gymnasium returns five, obs, reward, terminated, truncated, info = env.step(action). Convert with done = terminated or truncated.
  • Once you also account for reset() returning (obs, info), most of the exercise code can be used as is.
  • For image observations, create the environment with render_mode="rgb_array", convert the RGB image returned by env.render() to grayscale (1 channel), and resize it to 64×64. The models in the text take 1-channel input.

Frequently asked questions

Can I follow along to the end without a GPU?
Up to chapter 10, and chapter 13, yes. The code is kept small enough to run on a CPU, and each chapter lists the actual time it took (about 1 – 20 minutes on a CPU for chapters 06, 07, and 13). The settings in the text for chapters 11 and 12 (PlaNet and Dreamer, which see images) take more than a day on a CPU, which is impractical. First confirm that the code runs with --quick, then either run the full training on a GPU (a free cloud notebook works) or reduce the number of training steps and look only at the trends.
How does a World Model differ from a simulator (physics engine)?
A physics engine contains rules (equations) written in by people; a World Model learns those rules from data. Chapter 6 learns the pendulum with a neural network even though its equations are known, because in the real world (robots, games, driving) the equations are usually unknown or too complex. A learned model can be wrong, so always keep uncertainty (chapter 7) and model exploitation (chapter 10) in mind.
Is model-based reinforcement learning always better than model-free reinforcement learning?
No. When interacting with the environment is expensive (real robots, slow simulators), sample efficiency is a big advantage, but it comes at a price: the compute needed to learn the model, and model error. When the simulator is very fast and cheap, model-free methods are often simpler and reach better final performance. That said, methods such as DreamerV3 that aim for the advantages of both are becoming more common.
Why use only the pendulum from start to finish?
Keeping the world the same lets you compare only the effect of changing the model. The pendulum's physics fits in three lines, so results can be checked against the ground truth; seen as images, it becomes a partially observable problem in which velocity cannot be seen; and, like a swing, it requires building up energy, so a simple greedy policy cannot solve it. It is small, but it contains all the core difficulties of World Models.
What to change when making images larger or switching to color
Change the number of convolutional layers and the input channels (grayscale 1 → color 3) in the encoder and decoder, and recompute the sizes layer by layer following the output-size calculations in chapters 8 and 11. More pixels increase the magnitude of the reconstruction loss, so rebalance it against the KL (β, free nats) as well.
When Dreamer does not perform as well as in the paper
First compare the number of training steps. The papers use hundreds of thousands to millions of steps, while the tutorial uses far fewer. Then check "the gap between imagined and real returns", "open-loop imagined images", and "KL magnitude" in the symptom tables. If reproduction is the goal, comparing against a public implementation with the same settings is the fastest route.
Are large language models (LLMs) also World Models?
LLMs clearly hold a great deal of knowledge about the world, but they do not exactly fit the tutorial's narrow definition ("given an action, predict the next state, and use that to plan or learn"). Research that combines language with World Models is active, such as using an LLM as an agent's planner (imagining the outcomes of actions in text) or video World Models that create and modify worlds from text, like Genie 3 and 4 in chapter 13. Researchers have not agreed on how far the term World Model should extend.
When the math gets stuck
Go back to the chapter that the derivation of the equation points to (usually chapters 2 and 3). Then plug small numbers into the equation and compute it yourself. Every key equation in the tutorial comes with a numerical example and code that checks those numbers.

Next steps: projects to try

Projects worth trying, in order of difficulty, using the tutorial code as a starting point.

Difficulty

Project

What you learn

★☆☆

Change the pendulum's reward so the goal is to “spin fast while hanging upside down”, and run the chapter 7 MPC on it

The advantage of model-based methods: solving a new task by changing only the reward and keeping the World Model as is

★☆☆

Change the chapter 6 dynamics model to predict the “next observation” directly instead of the “change” (turning off trick 2), and compare one-step error and compounding error

How the design of the output affects multi-step prediction

★★☆

Port the chapter 6 and 7 exercises to Gymnasium's MountainCarContinuous-v0 (chapter 7 uses a known reward function, so you must write the new environment's reward function yourself; discrete-action environments such as CartPole need a categorical CEM instead of a Gaussian one)

Environment interfaces, and the exploration problem in sparse-reward tasks

★★☆

Add DreamerV3's symlog, two-hot, free bits, and return normalization to the chapter 12 Dreamer one at a time and compare their effects

Confirming the role of each technique by experiment (ablation study)

★★☆

Add the disagreement of the chapter 7 ensemble as a penalty on the chapter 12 imagined rewards to reduce model exploitation

Uncertainty-aware learning in imagination

★★★

Scale the chapter 13 diffusion exercise up to “next-frame prediction for 64×64 pendulum images”

Conditional diffusion World Models, drift, and noise augmentation

★★★

Remove the decoder from the chapter 11 RSSM and replace it with a TD-MPC-style latent consistency + TD loss

Reconstruction-free World Models and representation collapse

★★★

Train a Genie-style latent action model on pendulum videos (without action logs) and check whether the codes correspond to the direction of the torque

Learning World Models without action labels, information bottleneck

Summary

  • Start debugging by checking the data, checking shapes, and overfitting a small dataset, in that order; change hyperparameters last, one at a time.
  • Log losses term by term and gradients by their norm. A norm of 0 means a cut-off gradient; values jumping into the thousands mean an explosion.
  • NaN mostly comes from \(\log 0\), exponential overflow, and zero standard deviation. Prevent it with a stable softplus, logit BCE (\(\operatorname{softplus}(x) - yx\)), logsumexp, a standard deviation floor, and gradient clipping.
  • Hyperparameters from the original papers are starting points and differ by version and official code. If things do not work, adjust the amount of data, the learning rate, model size, KL-related values, and the imagination and planning horizons, in that order.
  • The tutorial as a whole starts from a one-sentence definition, "a learned simulator that predicts what comes next given a state and an action", and goes from an MLP that predicts three numbers to Dreamer, which learns to swing the pendulum upright in imagination from images alone.
  • Today's large World Models are also different answers to the same three questions: what to predict, how to represent an uncertain future, and how to turn imagination into action.

Review questions

1. The result of clipping the gradient \(g = (3, 4)\) to a maximum length of \(c = 1\), and the result when \(c = 10\)
\(\lVert g\rVert = 5\). With \(c = 1\), \(g' = (3, 4)/5 = (0.6, 0.8)\). With \(c = 10\), \(5 \le 10\), so it stays \((3, 4)\).
2. For logit \(x = 0\) and label \(y = 1\), compare the BCE computed as \(\operatorname{softplus}(x) - yx\) with the definition \(-\log\sigma(0)\)
\(\operatorname{softplus}(0) - 0 = \log 2 \approx 0.693\). By the definition, \(\sigma(0) = 0.5\), so \(-\log 0.5 = \log 2\), the same value.
3. Diagnosis when the VAE's KL term is nearly 0 and the reconstruction is nearly the same blurry image no matter which image goes in
This is posterior collapse. The encoder's \(q(z\mid x)\) has become equal to the prior \(\mathcal{N}(0, I)\), so the latent carries no information and the decoder draws only the average image. Lower β or ramp the KL weight up slowly from 0 (warm-up), use free bits, and if the decoder is too powerful (e.g., an autoregressive decoder), weaken it (chapters 8, 11, and 12).
4. The range that \(\gamma = 0.9\) covers in the pendulum environment with action repeat 2 (100 decisions per episode)
\(\frac{1}{1 - 0.9} = 10\) steps, or about \(10 \times 2 \times 0.05 = 1\) second. Swinging the pendulum up takes a few seconds, so this may be too short. With \(\gamma = 0.99\), it is about 100 steps (10 seconds).

See also

Favorite site

author MCPBot, Zer0 sitemap