Skip to content
Search

BitNet

lastmod 2026-03-13created 2025-04-212 edits

Microsoft's hyper-efficient AI model that can run on CPUs

About

  • Microsoft researchers developed a hyper-efficient AI model called BitNet b1.58 2B4T
  • Achieves high speed and low memory usage through 1-bit quantization, so it can run even on CPUs, and is released under the MIT license
  • Runs even on CPUs like the Apple M2 and works without a GPU
  • With 2 billion parameters, BitNet b1.58 2B4T outperforms models from Meta, Google, and Alibaba
  • However, it requires Microsoft's bitnet.cpp framework, and compatibility issues with GPUs still remain
  • Microsoft's bitnet.cpp is the official inference framework for 1-bit large language models (LLMs), supporting fast, lossless inference on CPUs and GPUs
  • Achieves 1.37–5.07x speedups and 55.4–70% energy savings on ARM CPUs, and 2.37–6.17x speedups and 71.9–82.2% energy savings on x86 CPUs
  • Adds parallel kernel implementations and embedding quantization for a further 1.15–2.1x performance gain over before
  • Can run BitNet b1.58 models on a single CPU at 5–7 tokens per second, expanding the possibility of running very large models locally
  • Built on llama.cpp and T-MAC, contributing to the growth of the open-source ecosystem for efficient low-bit LLM inference

bitnet.cpp overview

  • bitnet.cpp is the official inference framework for 1-bit LLMs (e.g. BitNet b1.58) , providing optimized kernels for CPUs and GPUs
    • NPU support to be added in the future
  • The first release supports CPU inference and demonstrates speed and energy efficiency improvements on both ARM and x86 architectures
    • ARM CPU: 1.37–5.07x speedup, 55.4–70% energy savings
    • x86 CPU: 2.37–6.17x speedup, 71.9–82.2% energy savings
  • Can run a 100B-parameter BitNet b1.58 model on a single CPU at 5–7 tokens per second

Latest optimizations

  • Parallel kernel implementations, tiling configuration, and embedding quantization were added, achieving a further 1.15–2.1x speedup over before
  • Performance improvements confirmed across various hardware platforms and workloads
  • Technical details are provided in the optimization guide document

Demo and official models

  • Provides a demo running the BitNet b1.58 3B model on Apple M2
  • BitNet-b1.58-2B-4T is released on Hugging Face as the official model, supported on x86 and ARM CPUs
  • Supported models include bitnet_b1_58-large(0.7B), bitnet_b1_58-3B(3.3B), Llama3-8B-1.58, and the Falcon3 and Falcon-E series

Installation and build

  • Requirements: Python 3.9 or later, CMake 3.22 or later, Clang 18 or later
    • Windows users need to install Visual Studio 2022
    • An automatic installation script is provided for Debian/Ubuntu users
  • A Conda environment is recommended, and setup_env.py can be used to set up the model environment
  • Models can be downloaded from Hugging Face and run from a local path

Usage and benchmarks

  • Run inference on quantized models with run_inference.py
    • Main arguments: model path(-m), prompt(-p), number of threads(-t), context size(-c), temperature(-temp)
  • The e2e_benchmark.py script can measure performance with a specified number of tokens, prompt length, and number of threads
  • If no public model is available, a dummy model can be generated with generate-dummy-bitnet-model.py for benchmarking

Model conversion and FAQ

  • Provides a script to convert .safetensors checkpoints to the gguf format
  • The FAQ explains how to fix llama.cpp build errors (related to std::chrono) and how to set up the clang environment on Windows
    • The environment must be initialized in the Visual Studio Developer Command Prompt or PowerShell

See also

Favorite site

author Zer0 sitemap