MiniLLM
lastmod 2023-03-30created 2023-03-301 edits
Running LLMs on personal GPUs
Features
- A minimal system for running modern LLMs on consumer NVIDIA GPUs
- Supports multiple LLMs (LLAMA, BLOOM, OPT) up to 170B in size
- Supports many kinds of NVIDIA GPUs
- Small, easy-to-use code written in Python
- Internally uses the GPTQ algorithm for 3-bit compression to minimize GPU memory usage