NanoGPT

lastmod 2024-08-16created 2023-01-262 edits

Open source for the fastest training/fine-tuning of medium-sized GPTs

Features

• A rewrite of minGPT by Andrej Karpathy, former head of AI at Tesla
• Aims to be readable, plain code
  • train.py of about 300 lines
  • model.py of about 300 lines
• Work in progress on reproducing GPT-2 on the OpenWebText dataset

See also

• GPT
• nano-llama31 - a Llama 3.1 version in the style of nanoGPT

Favorite site

• Github - nanoGPT