Flide
lastmod 2026-11-11created 2026-11-111 edits
Runs decision models like Jev up to 9.5x faster.
About
- Inference infrastructure for Jev-family decision models. Exact state forking makes inference up to 9.5x faster than vLLM and SGLang
- Shared input is computed only once, and each question forks from that model state and is answered in a single forward pass. Each answer is identical to asking that question alone on the same input
- The
/v1/systemoneAPI takes a sharedstateand multiplequestionsin one request, with up to 256 choices per question.flide verifychecks that state forking is exact for your own model and requests - In an email classification benchmark, one H200 with OneJev 0.8B processed 10,000 emails with 8 questions each in 28 seconds, 4.1x faster end to end than the fastest engine compared
- The speedup varies widely by model and task. By median per-request latency on one H200, OneJev is 4.7-7.3x faster than qev, Qwen3.5 1.3-4.8x, and Gemma 4 1.0-1.7x
- Supports both chat models and decision models, including Qwen3.5, Gemma 4, OneJev, Open-Jev, and pplx-decider, and serves LoRA adapters directly. Some models accept image input, and OneJev and Xor support text, images, and video
- A session cache keeps the state of previous requests, so an agent's next step reads only the newly added part.
--batch-invarianttrades some speed for bitwise-identical results regardless of batch composition - Supports FP8 quantization of Qwen3.5 on Ada, Hopper, and Blackwell GPUs. HTTP handling, rendering, and tokenization run in a separate frontend process, so it can serve many clients
- Written in Python. Apache-2.0 license; some kernels are derived from flash-linear-attention, which is MIT-licensed