Open-LLaMa

lastmod 2023-05-17created 2023-05-171 edits

A complete training pipeline for building an LLM

Features

• Open source that covers the entire pipeline from dataset creation to tokenization, prompt tuning, LoRA, and RLHF
• The pretrained Open-LLama-V2-pretrain model is also released on Hugging Face
• Reportedly reaches about 89% of GPT-3.5's performance by the FastChat evaluation method (on questions in Chinese)
• Training speed is 3620 tokens/s, slightly faster than the original LLama's 3370 (7B model)
• Training on 500B tokens requires 38300 GPU hours
  • Using 8 A100-80G Spot GPUs on Google Cloud for one hour costs 12.6 dollars
  • Total cost is 60300 dollars

See also

• LLM
• LLaMA

Favorite site

• Github - Open-Llama