Microgpt
lastmod 2026-02-18created 2026-02-181 edits
GPT training and inference implemented in 200 lines of pure Python
About
- An art project released by karpathy. Implements the entire GPT algorithm in a single 200-line file with no external dependencies
- The difference from production LLMs is only scale and efficiency; the core is the same, so understanding this code means understanding the algorithmic essence of GPT
- Includes the dataset, tokenizer, autograd engine, GPT-2-like Transformer architecture, Adam optimizer, and training and inference loops
- The culmination of earlier projects such as micrograd, makemore, and nanogpt and of 10 years of work simplifying LLMs; captures the essence of GPT in a minimal form that cannot be simplified any further
- Trains on 32,000 names to generate plausible new names, performing every computation directly with scalar-level autograd
- The training process consists of loss computation → backpropagation → Adam update, and runs in about 1 minute
microgpt overview
- microgpt is a 200-line Python script that fully implements the training and inference process of a GPT model
- Includes the dataset, tokenizer, autograd, model, optimizer, and training loop, all without external libraries
Dataset
- The fuel of large language models is a stream of text data; production uses web pages from the internet, but microgpt uses a simple example containing 32,000 names, one per line
- Each name is treated as one "document", and the goal is for the model to learn the statistical patterns in the data and generate similar new documents
- After training, the model "hallucinates" plausible new names such as "kamon", "karai", and "vialan"
- From ChatGPT's perspective, a conversation with the user is just "a strange-looking document"; when you initialize a document with a prompt, the model's response amounts to statistical document completion
Tokenizer
- Neural networks work on numbers, not characters, so a way is needed to convert text into a sequence of integer token IDs and back
- Production tokenizers such as tiktoken (used by GPT-4) work on chunks of characters for efficiency, but the simplest tokenizer assigns one integer to each unique character in the dataset
- The lowercase letters a-z are sorted and each character gets its index as its ID; the integer values themselves carry no meaning, and each token is a separate discrete symbol
- A special BOS (Beginning of Sequence) token is added to signal "a new document starts/ends"; "emma" is wrapped as [BOS, e, m, m, a, BOS]
- The final vocabulary size is 27 (26 lowercase letters + 1 BOS)
Automatic differentiation (Autograd)
- Training a neural network requires gradients: for each parameter you need to know "if I nudge this value up slightly, does the loss go up or down, and by how much?"
- The computation graph has many inputs (model parameters and input tokens) but converges to a single scalar output, the loss
- Backpropagation starts at the output and walks the graph backward, relying on the chain rule of calculus to compute the gradient of the loss with respect to every input
- Implemented as the Value class: each Value wraps a single scalar (.data) and tracks how it was computed
- On operations such as addition and multiplication, the new Value remembers its inputs (_children) and the local derivatives of that operation (_local_grads)
- Example: mul records ∂(a·b)/∂a=b and ∂(a·b)/∂b=a
- Starts with self.grad = 1 at the loss node (∂L/∂L=1)
- Multiplies local gradients along the path and propagates them down to the parameters
Parameter initialization
- Parameters are the model's knowledge: a large set of floating-point numbers that start random and are iteratively optimized during training
- Initialized with small random values from a Gaussian distribution
- Organized as named matrices in state_dict: embedding tables, attention weights, MLP weights, and the final output projection
- Hyperparameter settings:
- n_embd = 16: embedding dimension
- n_head = 4: number of attention heads
- n_layer = 1: number of layers
- block_size = 16: maximum sequence length
Architecture
- The model architecture is a stateless function: it takes a token, a position, the parameters, and the cached keys/values from previous positions, and returns logits (scores) for the next token
- Follows GPT-2 with slight simplifications: RMSNorm (instead of LayerNorm), no biases, ReLU (instead of GeLU)
- Helper functions
- linear: matrix-vector multiplication, computing one dot product for each row of the weight matrix; a learned linear transformation, the basic building block of neural networks
- softmax: converts raw scores (logits) into a probability distribution where every value falls in the [0,1] range and they sum to 1; subtracts the maximum first for numerical stability
- rmsnorm: rescales a vector to have unit root mean square, preventing activations from growing or shrinking as they pass through the network and stabilizing training
- Embedding: the token ID and position ID each look up a row in their own embedding table (wte, wpe), and the two vectors are added to encode both what the token is and where it is in the sequence
- Modern LLMs skip position embeddings and use relative positioning techniques such as RoPE
- Attention block: projects the current token into three vectors, Q (query), K (key), and V (value)
- Query: "What am I looking for?", key: "What do I contain?", value: "What do I offer if selected?"
- Example: in "emma", when the second "m" predicts what comes next, it may learn a query like "which vowel appeared recently?", and the earlier "e" matches this query well and gets a high attention weight
- Keys and values are appended to the KV cache so earlier positions can be referenced
- Each attention head computes the dot product (scaled by √d_head) between the query and all cached keys, gets attention weights via softmax, and computes the weighted sum of the cached values
- All head outputs are concatenated and projected with attn_wo
- The attention block is the only place where the token at position t can "see" past tokens 0..t-1; attention is the token communication mechanism
- MLP block: a two-layer feed-forward network: expand to 4 times the embedding dimension → apply ReLU → project back down
- Where most of the per-position "thinking" happens
- Unlike attention, a computation that is fully local to time t
- The Transformer interleaves communication (attention) and computation (MLP)
- Residual connections: both the attention and MLP blocks add their output back to their input
- Lets gradients flow directly through the network, making deep models trainable
- Output: the final hidden state is projected to the vocabulary size with lm_head, producing one logit per token (27 numbers here); a higher logit = that token is more likely to come next
- KV cache quirk: using a KV cache even during training is unusual, but microgpt builds it explicitly because it processes only one token at a time; the cached keys and values are live Value nodes in the computation graph and are subject to backpropagation
Training loop
- The training loop repeatedly: (1) picks a document → (2) runs the model forward over its tokens → (3) computes the loss → (4) gets gradients via backpropagation → (5) updates the parameters
- Tokenization
- At each training step, one document is picked and wrapped with BOS on both sides: "emma" → [BOS, e, m, m, a, BOS]
- The model's goal is to predict each next token given the preceding tokens
- Feeds tokens into the model one at a time, building the KV cache
- At each position the model outputs 27 logits, converted to probabilities with softmax
- The loss at each position is the negative log probability of the correct next token: −log p(target), called the cross-entropy loss
- The loss measures how surprised the model is by what actually comes: assigning probability 1.0 gives loss 0, probability near 0 gives loss +∞
- Averages the per-position losses over the whole document to get a single scalar loss
- A single call to loss.backward() runs backpropagation over the entire computation graph
- Afterward, each parameter's .grad tells how it should change to reduce the loss
- Uses Adam instead of plain gradient descent (p.data -= lr * p.grad)
- Keeps two moving averages per parameter:
- m: average of recent gradients (momentum)
- v: average of recent squared gradients (per-parameter learning rate adaptation)
- m_hat and v_hat are bias corrections for m and v, which are initialized to 0
- The learning rate decays linearly during training
- Resets .grad = 0 after the update
- Over 1,000 steps the loss drops from about 3.3 (random guessing among 27 tokens: −log(1/27)≈3.3) to about 2.37
- Lower is better and the minimum is 0 (perfect prediction), so there is room for improvement, but the model is clearly learning the statistical patterns of names
Inference
- After training, new names can be sampled from the model: with the parameters frozen, the forward pass runs in a loop, feeding each generated token back as the next input
- Sampling process
- Each sample starts with the BOS token ("start a new name")
- The model produces 27 logits → converts them to probabilities → randomly samples one token according to those probabilities
- That token is fed back as the next input, repeating until the model produces BOS again ("done") or the maximum sequence length is reached
- Divides the logits by the temperature before softmax
- Temperature 1.0: samples directly from the distribution the model learned
- Low temperature (e.g. 0.5): sharpens the distribution, making the model more conservative and more likely to pick top choices
- Temperature near 0: always picks the single most probable token (greedy decoding)
- High temperature: flattens the distribution, giving more diverse but less coherent output
How to run
- Only Python is needed (no pip install, no dependencies): python train.py
- Takes about 1 minute on a MacBook
- Prints the loss at each step: drops from ~3.3 (random) to ~2.37
- After training, generates new hallucinated names: "kamon", "ann", "karai", and so on
- Can also run in a Google Colab notebook, where you can ask Gemini questions
- Better results are possible by trying other datasets, training longer by increasing num_steps, or increasing the model size
Code progression
| File | What it adds |
| train0.py | Bigram count table — no neural network, no gradients |
| train1.py | MLP + manual gradients (numerical & analytic) + SGD |
| train2.py | Autograd (Value class) — replaces manual gradients |
| train3.py | Position embeddings + single-head attention + rmsnorm + residuals |
| train4.py | Multi-head attention + layer loop — the full GPT architecture |
| train5.py | Adam optimizer — this is train.py |
- All versions and the diffs between each step can be viewed in the Revisions of the build_microgpt.py Gist
Differences from production LLMs
- microgpt contains the complete algorithmic essence of training and running a GPT; what separates it from production LLMs such as ChatGPT does not change the core algorithm but consists of the things that make it work at scale
- Data
- Trained on trillions of tokens of internet text (web pages, books, code, etc.) instead of 32K short names
- Data deduplication, quality filtering, careful mixing across domains
- Uses subword tokenizers such as BPE (Byte Pair Encoding) instead of single characters
- Merges character sequences that often appear together into single tokens; common words like "the" become a single token, rare words are split into pieces
- A vocabulary of ~100K tokens; much more efficient because the model sees more content per position
- Uses tensors (large multidimensional arrays of numbers) instead of scalar Value objects in pure Python, running on GPUs/TPUs that perform billions of floating-point operations per second
- PyTorch handles autograd over tensors, and CUDA kernels such as FlashAttention fuse multiple operations
- The math is the same; many scalars are processed in parallel
- microgpt: 4,192 parameters; GPT-4-class models: hundreds of billions
- Overall a very similar Transformer neural network, but much wider (embedding dimension 10,000+) and much deeper (100+ layers)
- Additional Lego block types and reordering:
- RoPE (rotary position embedding) — instead of learned position embeddings
- GQA (grouped-query attention) — reduces KV cache size
- Gated linear activations — instead of ReLU
- MoE (mixture of experts) layers
- The core structure of attention (communication) and MLP (computation) alternating on top of a residual stream is well preserved
- Large batches (millions of tokens per step) instead of one document per step, gradient accumulation, mixed precision (float16/bfloat16), careful hyperparameter tuning
- Training a frontier model runs thousands of GPUs for months
- microgpt: Adam + simple linear learning rate decay
- At scale, optimization is a field of its own: reduced precision (bfloat16, fp8), training on large GPU clusters
- Optimizer settings (learning rate, weight decay, beta parameters, warmup/decay schedules) need precise tuning, and the right values depend on model size, batch size, and dataset composition
- Scaling laws (e.g. Chinchilla) guide how to split a fixed compute budget between model size and number of training tokens
- Getting these details wrong at scale can waste millions of dollars of compute, so teams run extensive small-scale experiments before a full training run
- The base model that comes out of training (the "pretrained" model) is a document completer, not a chatbot
- Turning it into ChatGPT takes two stages:
- SFT (supervised fine-tuning): replaces the documents with curated conversations and continues training; algorithmically nothing changes
- RL (reinforcement learning): the model generates responses → they are scored (by humans, a "judge" model, or an algorithm) → it learns from the feedback
- Fundamentally it still trains on documents, but now the documents consist of tokens that came from the model itself
- Serving a model to millions of users needs its own engineering stack: request batching, KV cache management and paging (vLLM, etc.), speculative decoding for speed, quantization to reduce memory (running in int8/int4), distributing the model across multiple GPUs
- Fundamentally it still predicts the next token in a sequence, but a lot of effort goes into the engineering that makes it faster
FAQ
- Does the model "understand" anything?
- A philosophical question, but mechanically: no magic happens
- The model is a big mathematical function that maps input tokens to a probability distribution over the next token
- During training, the parameters are adjusted to make the correct next token more probable
- Whether this constitutes "understanding" is up to the individual, but the mechanism is fully contained in the 200 lines above
- The model has thousands of adjustable parameters, and the optimizer nudges them a little at each step to lower the loss
- Over many steps, the parameters settle on values that capture the statistical regularities of the data
- For names: they often start with a consonant, "qu" tends to appear together, three consonants in a row are rare, and so on
- The model learns a probability distribution that reflects these, not explicit rules
- ChatGPT is this same core loop (predict the next token, sample, repeat) scaled up enormously, with post-training added to make it conversational
- When chatting, the system prompt, user messages, and responses are all just tokens in a sequence
- The model completes the document one token at a time, the same way microgpt completes a name
- The model generates tokens by sampling from a probability distribution
- It has no concept of truth; it only knows which sequences are statistically plausible given the training data
- microgpt "hallucinating" a name like "karia" is the same phenomenon as ChatGPT confidently stating a false fact
- Both are plausible-sounding completions that are not real
- microgpt processes one scalar at a time in pure Python, so a single training step takes several seconds
- On a GPU, the same math processes millions of scalars in parallel and runs orders of magnitude faster
- Yes: train longer (increase num_steps), increase the model size (n_embd, n_layer, n_head), use a larger dataset
- These are the same knobs that matter at scale
- The model learns whatever patterns are in the data
- Swap in a file of city names, Pokémon names, English words, or short poems, and it learns to generate those instead
- The rest of the code needs no changes
Source code
"""
The most atomic way to train and run inference for a GPT in pure, dependency-free Python.
This file is the complete algorithm.
Everything else is just efficiency.
@karpathy
"""
import os # os.path.exists
import math # math.log, math.exp
import random # random.seed, random.choices, random.gauss, random.shuffle
random.seed(42) # Let there be order among chaos
# Let there be a Dataset `docs`: list[str] of documents (e.g. a list of names)
if not os.path.exists('input.txt'):
import urllib.request
names_url = 'https://raw.githubusercontent.com/karpathy/makemore/988aa59/names.txt'
urllib.request.urlretrieve(names_url, 'input.txt')
docs = [line.strip() for line in open('input.txt') if line.strip()]
random.shuffle(docs)
print(f"num docs: {len(docs)}")
# Let there be a Tokenizer to translate strings to sequences of integers ("tokens") and back
uchars = sorted(set(''.join(docs))) # unique characters in the dataset become token ids 0..n-1
BOS = len(uchars) # token id for a special Beginning of Sequence (BOS) token
vocab_size = len(uchars) + 1 # total number of unique tokens, +1 is for BOS
print(f"vocab size: {vocab_size}")
# Let there be Autograd to recursively apply the chain rule through a computation graph
class Value:
__slots__ = ('data', 'grad', '_children', '_local_grads') # Python optimization for memory usage
def __init__(self, data, children=(), local_grads=()):
self.data = data # scalar value of this node calculated during forward pass
self.grad = 0 # derivative of the loss w.r.t. this node, calculated in backward pass
self._children = children # children of this node in the computation graph
self._local_grads = local_grads # local derivative of this node w.r.t. its children
def __add__(self, other):
other = other if isinstance(other, Value) else Value(other)
return Value(self.data + other.data, (self, other), (1, 1))
def __mul__(self, other):
other = other if isinstance(other, Value) else Value(other)
return Value(self.data * other.data, (self, other), (other.data, self.data))
def __pow__(self, other): return Value(self.data**other, (self,), (other * self.data**(other-1),))
def log(self): return Value(math.log(self.data), (self,), (1/self.data,))
def exp(self): return Value(math.exp(self.data), (self,), (math.exp(self.data),))
def relu(self): return Value(max(0, self.data), (self,), (float(self.data > 0),))
def __neg__(self): return self * -1
def __radd__(self, other): return self + other
def __sub__(self, other): return self + (-other)
def __rsub__(self, other): return other + (-self)
def __rmul__(self, other): return self * other
def __truediv__(self, other): return self * other**-1
def __rtruediv__(self, other): return other * self**-1
def backward(self):
topo = []
visited = set()
def build_topo(v):
if v not in visited:
visited.add(v)
for child in v._children:
build_topo(child)
topo.append(v)
build_topo(self)
self.grad = 1
for v in reversed(topo):
for child, local_grad in zip(v._children, v._local_grads):
child.grad += local_grad * v.grad
# Initialize the parameters, to store the knowledge of the model
n_layer = 1 # depth of the transformer neural network (number of layers)
n_embd = 16 # width of the network (embedding dimension)
block_size = 16 # maximum context length of the attention window (note: the longest name is 15 characters)
n_head = 4 # number of attention heads
head_dim = n_embd // n_head # derived dimension of each head
matrix = lambda nout, nin, std=0.08: [[Value(random.gauss(0, std)) for _ in range(nin)] for _ in range(nout)]
state_dict = {'wte': matrix(vocab_size, n_embd), 'wpe': matrix(block_size, n_embd), 'lm_head': matrix(vocab_size, n_embd)}
for i in range(n_layer):
state_dict[f'layer{i}.attn_wq'] = matrix(n_embd, n_embd)
state_dict[f'layer{i}.attn_wk'] = matrix(n_embd, n_embd)
state_dict[f'layer{i}.attn_wv'] = matrix(n_embd, n_embd)
state_dict[f'layer{i}.attn_wo'] = matrix(n_embd, n_embd)
state_dict[f'layer{i}.mlp_fc1'] = matrix(4 * n_embd, n_embd)
state_dict[f'layer{i}.mlp_fc2'] = matrix(n_embd, 4 * n_embd)
params = [p for mat in state_dict.values() for row in mat for p in row] # flatten params into a single list[Value]
print(f"num params: {len(params)}")
# Define the model architecture: a function mapping tokens and parameters to logits over what comes next
# Follow GPT-2, blessed among the GPTs, with minor differences: layernorm -> rmsnorm, no biases, GeLU -> ReLU
def linear(x, w):
return [sum(wi * xi for wi, xi in zip(wo, x)) for wo in w]
def softmax(logits):
max_val = max(val.data for val in logits)
exps = [(val - max_val).exp() for val in logits]
total = sum(exps)
return [e / total for e in exps]
def rmsnorm(x):
ms = sum(xi * xi for xi in x) / len(x)
scale = (ms + 1e-5) ** -0.5
return [xi * scale for xi in x]
def gpt(token_id, pos_id, keys, values):
tok_emb = state_dict['wte'][token_id] # token embedding
pos_emb = state_dict['wpe'][pos_id] # position embedding
x = [t + p for t, p in zip(tok_emb, pos_emb)] # joint token and position embedding
x = rmsnorm(x) # note: not redundant due to backward pass via the residual connection
for li in range(n_layer):
# 1) Multi-head Attention block
x_residual = x
x = rmsnorm(x)
q = linear(x, state_dict[f'layer{li}.attn_wq'])
k = linear(x, state_dict[f'layer{li}.attn_wk'])
v = linear(x, state_dict[f'layer{li}.attn_wv'])
keys[li].append(k)
values[li].append(v)
x_attn = []
for h in range(n_head):
hs = h * head_dim
q_h = q[hs:hs+head_dim]
k_h = [ki[hs:hs+head_dim] for ki in keys[li]]
v_h = [vi[hs:hs+head_dim] for vi in values[li]]
attn_logits = [sum(q_h[j] * k_h[t][j] for j in range(head_dim)) / head_dim**0.5 for t in range(len(k_h))]
attn_weights = softmax(attn_logits)
head_out = [sum(attn_weights[t] * v_h[t][j] for t in range(len(v_h))) for j in range(head_dim)]
x_attn.extend(head_out)
x = linear(x_attn, state_dict[f'layer{li}.attn_wo'])
x = [a + b for a, b in zip(x, x_residual)]
# 2) MLP block
x_residual = x
x = rmsnorm(x)
x = linear(x, state_dict[f'layer{li}.mlp_fc1'])
x = [xi.relu() for xi in x]
x = linear(x, state_dict[f'layer{li}.mlp_fc2'])
x = [a + b for a, b in zip(x, x_residual)]
logits = linear(x, state_dict['lm_head'])
return logits
# Let there be Adam, the blessed optimizer and its buffers
learning_rate, beta1, beta2, eps_adam = 0.01, 0.85, 0.99, 1e-8
m = [0.0] * len(params) # first moment buffer
v = [0.0] * len(params) # second moment buffer
# Repeat in sequence
num_steps = 1000 # number of training steps
for step in range(num_steps):
# Take single document, tokenize it, surround it with BOS special token on both sides
doc = docs[step % len(docs)]
tokens = [BOS] + [uchars.index(ch) for ch in doc] + [BOS]
n = min(block_size, len(tokens) - 1)
# Forward the token sequence through the model, building up the computation graph all the way to the loss
keys, values = [[] for _ in range(n_layer)], [[] for _ in range(n_layer)]
losses = []
for pos_id in range(n):
token_id, target_id = tokens[pos_id], tokens[pos_id + 1]
logits = gpt(token_id, pos_id, keys, values)
probs = softmax(logits)
loss_t = -probs[target_id].log()
losses.append(loss_t)
loss = (1 / n) * sum(losses) # final average loss over the document sequence. May yours be low.
# Backward the loss, calculating the gradients with respect to all model parameters
loss.backward()
# Adam optimizer update: update the model parameters based on the corresponding gradients
lr_t = learning_rate * (1 - step / num_steps) # linear learning rate decay
for i, p in enumerate(params):
m[i] = beta1 * m[i] + (1 - beta1) * p.grad
v[i] = beta2 * v[i] + (1 - beta2) * p.grad ** 2
m_hat = m[i] / (1 - beta1 ** (step + 1))
v_hat = v[i] / (1 - beta2 ** (step + 1))
p.data -= lr_t * m_hat / (v_hat ** 0.5 + eps_adam)
p.grad = 0
print(f"step {step+1:4d} / {num_steps:4d} | loss {loss.data:.4f}", end='\r')
# Inference: may the model babble back to us
temperature = 0.5 # in (0, 1], control the "creativity" of generated text, low to high
print("\n--- inference (new, hallucinated names) ---")
for sample_idx in range(20):
keys, values = [[] for _ in range(n_layer)], [[] for _ in range(n_layer)]
token_id = BOS
sample = []
for pos_id in range(block_size):
logits = gpt(token_id, pos_id, keys, values)
probs = softmax([l / temperature for l in logits])
token_id = random.choices(range(vocab_size), weights=[p.data for p in probs])[0]
if token_id == BOS:
break
sample.append(uchars[token_id])
print(f"sample {sample_idx+1:2d}: {''.join(sample)}")