Hypura
lastmod 2026-03-25created 2026-03-251 edits
Storage-tier-aware LLM inference scheduler for Apple Silicon
About
- A storage-tier-aware inference scheduler that runs large language models by optimizing tensor placement across GPU, RAM, and NVMe
- On a 32GB Mac mini, it can run the Mixtral 8x7B (31GB) model at 2.2 tok/s and the Llama 70B (40GB) model at 0.3 tok/s
- Analyzes access patterns and hardware bandwidth to reliably run models that exceed physical memory, including models on which llama.cpp previously failed with OOM
- Through MoE expert routing, a neuron cache, and prefetching, it cuts I/O by up to 75% and achieves a 99.5% cache hit rate
- Automatically selects Full-resident, Expert-streaming, or Dense FFN-streaming mode depending on model size and hardware to maintain optimal performance
- Provides an Ollama-compatible HTTP API so it integrates with OpenClaw and others; uses the SSD read-only, supporting NVMe-based inference without reducing its lifespan