Ntransformer

lastmod 2026-02-23created 2026-02-232 edits

An NVMe-to-GPU inference engine that runs Llama 3.1 70B on a single RTX 3090

About

A C++/CUDA-based LLM inference engine that can run the Llama 70B model on an RTX 3090 (24GB VRAM) through GPU memory streaming and direct NVMe I/O
Uses a 3-tier adaptive caching structure that automatically splits across VRAM, pinned RAM, and NVMe/mmap, achieving up to 83x speedup over mmap
The gpu-nvme-direct backend bypasses the CPU entirely and transfers data directly from NVMe to the GPU, making full use of PCIe bandwidth
Layer skip and self-speculative decoding reduce unnecessary computation and increase throughput without quality loss
Enables very large models to run efficiently on consumer hardware, suggesting the potential for wider access to high-performance LLM inference

A high-efficiency C++/CUDA LLM inference engine that runs the Llama 70B model on an RTX 3090 (24GB VRAM)
- Streams model layers through GPU memory, and can optionally use direct NVMe I/O to bypass the CPU entirely
No external dependencies besides the CUDA Toolkit; PyTorch and cuBLAS not required
Supports the GGUF model format, with Q4_0, Q8_0, Q4_K_M, Q5_K, Q6_K, F16, and F32 quantization formats

3-Tier Adaptive Caching
- VRAM-resident layers (0 I/O)
- Pinned RAM (H2D transfer only)
- NVMe/mmap fallback
83x speedup over mmap on an RTX 3090 + 48GB RAM setup
PCIe Gen3 x8 bandwidth (about 6.5 GB/s) is the bottleneck
Q4_K_M quantization fits 10 more layers in VRAM (36 vs 26), reducing transfer volume
Layer skip (based on cosine similarity) skips 20 of 80 layers with minimal quality loss

SLEP streaming: overlaps NVMe reads, PCIe DMA, and GPU compute with double buffering
gpu-nvme-direct backend: reads NVMe data directly into GPU-accessible pinned memory
Self-speculative decoding: uses VRAM-resident layers as the draft model, no extra model needed
Automatic data path selection: VRAM-resident > pinned RAM H2D > mmap pinned > CPU memcpy
Llama architecture support: includes RoPE, GQA, SwiGLU, RMSNorm, KV cache

Linux (Ubuntu, kernel 6.17+), CUDA Toolkit 13.1, gcc/g++ 14, CMake 3.24+
Compute Capability 8.0+ GPU (tested on RTX 3090)
Direct NVMe I/O requires an NVMe SSD in a separate PCIe slot and the gpu-nvme-direct library

When the model does not fit in VRAM, the direct NVMe → GPU path removes the CPU entirely
- Data flow: NVMe SSD → DMA → pinned staging memory → PCIe H2D → GPU buffer → compute
Binds the NVMe to VFIO for direct access from user space
Each layer (about 670MB for 70B Q6_K) is read with 670 NVMe commands in about 202ms
NVMe reads, H2D DMA, and GPU compute run in parallel in a double-buffered pipeline

The automatic setup script (setup_system.sh) configures GRUB, NVIDIA DKMS, CUDA headers, VFIO, and NVMe binding in sequence
Includes high-risk operations such as disabling the IOMMU, patching kernel modules, and binding NVMe to VFIO
Misconfiguration can cause boot failure, NVMe data loss, or system instability
Never use the boot drive; a separate dedicated NVMe device is required
Backup and restore scripts are provided for all changes

Phase 1: Llama 8B Q8_0, custom CUDA kernels, 48.9 tok/s (done)
Phase 2: SLEP streaming, 70B on a single GPU, 33x speedup (done)
Phase 3: Q4_K_M/Q5_K support, Layer skip, self-speculative decoding, F16 KV cache (done)
Phase 4: NVMe Direct backend, GPU-initiated NVMe reads at 3.35 GB/s (done)
Phase 5: inference optimization and public C API (planned)

author Zer0 sitemap