Llama.cpp

lastmod 2025-11-04created 2023-03-147 edits

Inference of Facebook's LLaMA model in pure C/C++

Features

• Aims to run LLaMA models with 4-bit quantization on a MacBook
• Pure C/C++ implementation without dependencies
• Optimized for the Arm Neon/Accelerate frameworks (Apple Silicon)
• AVX2 support for x86
• Mixed F16/F32 precision
• 4-bit quntization support
• Runs on the CPU
• Currently supports only Mac/Linux. Windows support planned

See also

• LLaMA
• Alpaca.cpp - Run a ChatGPT-like model on your own laptop
• LocalAI - Self-hostable OpenAI-compatible API
• Sidekick - Open-source local-first LLM chatbot application for Mac

Favorite site

• Github - llama.cpp

Articles

• Llama.cpp now supports vision (multimodal input) | GeekNews
• Do you use open LLMs and coding assistants locally? Share your setup | GeekNews