VoxCPM2
lastmod 2026-04-15created 2026-04-151 edits
VoxCPM2: Tokenizer-Free TTS for Multilingual Speech Generation, Creative Voice Design, and True-to-Life Cloning
Open-source multilingual speech synthesis capable of voice cloning close to real voices
About
- Supports 30 languages including Korean; synthesize right away by just entering text, without specifying language tags
- Without voice actor samples, just prepend a natural-language description (gender, age, tone, emotion, speed) such as (Young female voice, warm and gentle) to the text to generate the desired voice character instantly
- A diffusion autoregressive TTS system that directly generates continuous speech representations without discrete tokenization; the 2B parameter model was trained on over 2 million hours of multilingual data
- Supports Controllable Voice Cloning, which clones timbre from a short reference clip while separately controlling emotion, speed, and style (slightly faster, cheerful tone)
- Ultimate Cloning mode reproduces timbre, rhythm, emotion, and style entirely when reference audio + transcript are provided together; no separate post-processing needed
- Even with 16kHz reference input, directly generates 48kHz studio-grade output via AudioVAE V2 asymmetric encode/decode. No need to build an external upsampler or post-processing pipeline
- RTF (Real-Time Factor) of about 0.3 on an RTX 4090, about 0.13 with Nano-vLLM acceleration, usable for real-time streaming
- Runs with about 8GB of VRAM, so it can operate on RTX 3070–4060-class GPUs
- Top-tier speaker similarity (SIM) among open-source models on Seed-TTS-eval; average error rate of 1.68% on an internal ASR benchmark across 30 languages
- With just 5–10 minutes of audio, adapt to a specific speaker or domain via LoRA fine-tuning; lora_ft_webui.py also supports WebUI-based training and inference
- Based on the MiniCPM-4 backbone, composed of a 4-stage pipeline LocEnc → TSLM → RALM → LocDiT
- Many options for inference without a GPU, such as GGML/GGUF CPU inference (VoxCPM.cpp), ONNX conversion, Apple Neural Engine backend, Rust reimplementation, ComfyUI nodes, etc. etc.
- Apache-2.0 license with no restrictions on commercial use; implemented in Python