Anthropic

lastmod 2026-03-02created 2023-05-256 edits

Two different fast LLM inference techniques from Anthropic and OpenAI

Structure of Anthropic's Fast Mode:

Anthropic's approach minimizes latency by reducing the batch size
- The main GPU bottleneck is memory transfer; batching that groups requests from multiple users raises throughput but increases wait time
- Fast mode is like a 'bus that departs immediately', processing a request as soon as the user sends it
This approach offers 2.5x faster speed at 6x higher cost
- Batch wait time disappears, so latency to the first token drops
- However, overall GPU throughput decreases
Some users pointed out that wait time after the first token is not large, and the effect of small batches is closer to faster execution from reduced computation

Structure of OpenAI's Fast Mode:

OpenAI takes a completely different approach using Cerebras chips
- Uses a lightweight (distilled) model called GPT-5.3-Codex-Spark instead of the existing model
- Spark is less refined than the original but achieves over 15x faster inference speed
The Cerebras chip is a huge 70-square-inch wafer-scale chip with 44GB of built-in SRAM
- Loads the whole model into SRAM and performs in-memory inference without external memory access
- A big difference compared to the tens of MB of SRAM on GPUs
The Spark model is presumed to reside sharded across multiple Cerebras chips, and this setup is the key factor in the speedup

Technical comparison and evaluation:

Anthropic keeps the existing model as is and simply adjusts the batching policy
OpenAI combines a new hardware architecture with model design to achieve a more fundamental performance gain
Running a model on Cerebras chips is a complex technical challenge, and training the Spark model is also no easy task
The two companies reached the same goal of higher speed through different technical paths; OpenAI's approach is technically more impressive

author Zer0 sitemap