VLLM
lastmod 2026-07-28created 2025-03-317 edits
A high-throughput and memory-efficient inference and serving engine for LLMs
A breakthrough in LLM inference throughput
The core challenge of LLM (large language model) serving is managing KV cache (Key-Value Cache) memory efficiently. Existing systems suffered from memory fragmentation, which lowered GPU utilization and limited throughput.
vLLM solves this problem with the PagedAttention algorithm, inspired by virtual memory and paging in operating systems. PagedAttention splits the KV cache into physically non-contiguous 'blocks' and manages it in those units, drastically reducing memory waste and enabling flexible memory allocation.
Building on PagedAttention, vLLM implements the following core techniques to maximize inference performance.
- Continuous Batching: Works dynamically, adding requests to the batch as soon as they arrive and removing completed sequences from the batch. This minimizes GPU idle time and improves throughput by up to 24x compared to conventional static batching.
- Support for advanced inference features: vLLM uses the flexibility of PagedAttention to efficiently support a variety of advanced features, such as:
- Chunked Prefill: Splits the prefill stage of a long prompt into multiple chunks, preventing a single request from monopolizing the system and improving latency for other requests.
- Prefix Caching: Shares the KV cache for prompt prefixes common to multiple requests without recomputation, eliminating redundant computation.
- Guided and Speculative Decoding: Supports techniques that control output to follow a specific format (e.g. JSON), or that use a smaller draft model to accelerate generation.
- Disaggregated Prefill/Decoding: Handles the compute-intensive prefill stage and the memory-bandwidth-intensive decoding stage on separate instances to optimize resource utilization.
In conclusion, vLLM resolves the fundamental inefficiency of KV cache management through PagedAttention, and on that basis applies optimizations such as continuous batching to raise the throughput and performance of LLM inference systems.
Running Qwen3.6 on 180G VRAM
## NVIDIA B200 ( 180G VRAM ) 에서 Qwen3.6 실행:
uv run vllm serve Qwen/Qwen3.6-27B \
--served-model-name geo-agent \
--api-key sk-geo-local-2026 \
--port 8000 \
--max-model-len 262144 \
--gpu-memory-utilization 0.90 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--enable-prefix-caching \
--limit-mm-per-prompt '{"image": {"count": 48, "width": 1280, "height": 1280}, "video": 0}' \
--mm-processor-kwargs '{"max_pixels": 1003520, "min_pixels": 3136}'
You can connect with opencode.
Troubleshooting
400MB system memory (RSS) leak
In vLLM's disaggregated Prefill/Decode serving setup, system memory (RSS) leaked at 400MB per minute, but ordinary Python profilers did not detect it.
- Root cause analysis: Heaptrack and pmap showed the leak was in anonymous memory mappings (mmap) rather than the heap, and the cause was traced with BPFtrace and automated GDB scripts.
- Culprit: UCX, a high-performance communication library, was intercepting mmap/munmap calls for optimization, and instead of returning freed memory immediately it piled it up in a queue indefinitely.
- Fix: The problem was solved by setting the environment variable UCX_MEM_MMAP_HOOK_MODE=none to disable UCX's memory hooking.
See also
- LLM
- llama
- LangChain - A library that connects LLMs to the outside world
- Open WebUI
- LM Studio
- ollama - Ease of use, support for many models, efficient resource management
- vLLM - High-performance inference, memory efficiency, large-scale batch processing
- LocalAI - Supports various model types (text, speech, image)
- FastChat - Supports various open-source models, provides a web UI