SmolGPT

lastmod 2025-02-02created 2025-02-021 edits

Minimal PyTorch implementation for training a small LLM from scratch

Simple PyTorch-based LLM training project designed for educational purposes, applying efficient training and modern sampling techniques

Features

• Minimal codebase: implemented in PyTorch with no complex abstractions
• Modern architecture: a GPT model including flash attention, RMSNorm, SwiGLU, and efficient sampling techniques
• Training features:
  • Mixed precision (bfloat16/float16)
  • Gradient accumulation
  • Learning rate decay with warmup
  • Weight decay and gradient clipping
• Dataset support: built-in TinyStories dataset processing
• Custom tokenizer: integrated SentencePiece tokenizer training

See also

• LLM

Favorite site

• Github - Om-Alve/smolGPT
• SmolGPT: Minimal PyTorch implementation for training a small LLM from scratch | GeekNews