Hibiki
lastmod 2025-02-11created 2025-02-111 edits
Hibiki is a model for streaming speech translation (also known as simultaneous translation). Unlike offline translation—where one waits for the end of the source utterance to start translating--- Hibiki adapts its flow to accumulate just enough context to produce a correct translation in real-time, chunk by chunk.
High-quality simultaneous speech-to-speech translation model
About
- A High Fidelity real-time streaming Speech-to-Speech Translation model
- Unlike typical offline translation, it "generates translated speech in real time" while the user is speaking
- Also provides text translation, and includes a voice conversion feature that preserves the style of the source speech
Architecture
- A "Decoder-only model" designed for simultaneous speech translation
- Uses Moshi's multistream architecture to model source and translated speech simultaneously
- Produces a continuous output stream at a constant frame rate of 12.5Hz per second, and also provides timestamped text translation
Training method
- Requires data in which source and translated speech and text are aligned, but such data is scarce in practice
- So training is done by generating synthetic data
- Uses the MADLAD machine translation system to align source and translated text in a weakly supervised manner
- Applies alignment rules so that words appear only once the translation becomes predictable from the source
- Method 1: silence insertion
- Method 2: alignment-aware TTS
Inference
- Hibiki encodes the source speech and generates translated speech in real time
- Uses temperature sampling without complex inference schemes, making it compatible with batch processing
- Voice similarity can be controlled by adjusting the Classifier-Free Guidance coefficient
- A higher coefficient produces speech closer to the source, but too high a value can degrade translation quality
How to run the model
- Runs on PyTorch, Rust, MLX (macOS), and MLX-Swift (iOS)
- Hibiki's code is nearly identical to the Moshi project; the actual implementation can be found in the kyutai-labs/moshi repository
- Currently only two models are provided, both supporting French → English (FR → EN) translation
- Hibiki 2B: deeper Transformer architecture, 16 RVQ per stream
- Hibiki 1B: lightweight version, 8 RVQ per stream, can run on-device (In-device)