Lit-LLaMA
lastmod 2023-04-03created 2023-04-031 edits
Open-source LLaMA implementation based on nanoGPT
Features
- Instead of the GPL-licensed LLaMA code, it is open source developed entirely separately on top of nanoGPT under the Apache 2.0 license
- Simple, implemented in a single file without bolierplate, and numerically equivalent to the original model
- Supports FlashAttention, quantization, LoRA fine-tuning, pre-training, etc.
- Optimized to run on consumer hardware