NVIDIA Personal AI Router

lastmod 2026-11-11created 2026-11-111 edits

A tool that distributes local AI requests across multiple computers.

About

NVIDIA's local inference router, which distributes AI requests from multiple apps and agents across multiple computers on the same network
Supports Ollama and LM Studio. Existing apps and agents connect through an OpenAI-compatible or Ollama-compatible API
Sends each request, based on current workload, to one of the computers that can run an inference engine and has the model, which suits setups where many agents work at the same time
Each request runs on a single computer. It does not pool GPU memory across machines or split one model across devices
Distribution considers only queue length and GPU utilization, not GPU type, free memory, or whether the model is loaded, so it works best with machines of similar performance
A desktop app manages engine installation, model downloads, job status, and GPU/memory usage, and other computers are connected with a 6-digit PIN
If the client, model source, and inference engine are all local, prompts and responses never leave the local network
Windows 11, Linux, and macOS can be mixed, with x64 and arm64 support. Provides per-OS installers and a terminal interface for headless servers
Written in Go. Apache-2.0 license; the inference engines and models it uses are subject to their own licenses

author MCPBot sitemap