Pocket TTS

lastmod 2026-01-19created 2026-01-191 edits

A high-quality TTS that gives your CPU a voice

About

• With 100 million parameters, Pocket TTS is a lightweight text-to-speech model with voice cloning that can run in real time even on an ordinary laptop CPU
• Narrows the performance gap between existing large LLM-based TTS (1 billion+ parameters) and the small Kokoro TTS (82 million parameters), achieving both high quality and efficiency
• Accurately clones a speaker's timbre, emotion, intonation, acoustic conditions, etc. from only about 5 seconds of voice sample
• Uses a Continuous Audio Language Model-based architecture to directly predict continuous latent vectors instead of discrete tokens, shrinking model size without quality loss
• Released as open source under the MIT license, setting a new standard for lightweight TTS technology that delivers high-quality speech synthesis even on CPU

Pocket TTS Overview

• Pocket TTS is a 100-million-parameter text-to-speech model that supports voice cloning
  • Can run in real time even on a laptop CPU
  • Can be run locally with the command uvx pocket-tts serve or uvx pocket-tts generate
• Developed by Kyutai and released as open source under the MIT license
  • Training data uses only public English speech datasets
  • Mentions the possibility of extending to additional private data

Comparison with existing TTS models

• Current TTS technology falls into two categories
  • Large LLM-based models: e.g. Kyutai TTS 1.6B (about 1.6 billion parameters)
    • Can model diverse voices, emotions, and acoustic conditions but require a GPU
  • Small specialized models: e.g. Kokoro TTS (82 million parameters)
    • Efficient thanks to a fixed voice set and hand-crafted pipeline, but limited in flexibility
• Pocket TTS sits between these two approaches, enabling high-quality speech synthesis even on CPU

Performance evaluation

• Evaluated on the Librispeech test-clean set
  • Audio inputs were cleaned with Adobe Enhance Speech to ensure 24kHz quality
• Compared against: F5-TTS, DSM, Chatterbox Turbo, Kokoro TTS
• Evaluation metrics:
  • Word Error Rate (WER)
  • Audio quality (ELO)
  • Speaker similarity (ELO)
• Summary of results:
  • Pocket TTS has the lowest error rate with a WER of 1.84
  • Audio quality is better than F5-TTS and DSM
  • Speaker similarity is on par with the reference voice
  • The only model that can run faster than real time on CPU

When tested on Intel Core Ultra 7 165H and Apple M3 CPUs, only Pocket TTS and Kokoro could synthesize in real time

Architecture

• Pocket TTS is designed based on Continuous Audio Language Model research
  • Previous approaches predicted discrete audio tokens, but Pocket TTS directly predicts continuous latent vectors (latent)
  • This removes the RQ-transformer bottleneck and makes the model lighter

Neural Audio Codec

• Designed based on the Mimi codec
  • Mimi compresses into discrete tokens, but Pocket TTS uses continuous latent representations
  • Applies VAE training normalized to a normal distribution
  • Distills (distillation) WavLM into the inner representation with a cosine similarity loss
  • Removes the RVQ stage and applies the distillation loss to the entire latent representation

Generative Model

• Based on the Masked Autoregressive (MAR) framework
  • Consists of a Causal Transformer backbone and an MLP sampler
  • Implements 1-step sampling using a Lagrangian Self-Distillation (LSD) loss
  • During inference, predicted latent vectors are fed back autoregressively

Voice and text conditioning

• Model input combines a voice prompt (a few seconds) and text
  • Voice is embedded with the codec encoder, text with the SentencePiece tokenizer

Model size breakdown

• Generative model (Transformer + MLP) : 90 million parameters
• Codec decoder: 10 million parameters
• Codec encoder: 18 million parameters (used only once, when encoding the voice sample)

Training data

• Composed entirely of public English speech datasets, 88,000 hours in total
  • AMI, EARNINGS22, GIGASpeech, SPGISpeech, TED-LIUM, VoxPopuli, LibriHeavy, Emilia

Key technical contributions

Head Batch Multiplier

• Reuses the z vector multiple times to ease the Transformer's compute bottleneck
  • Computes z once per input sequence and reuses it for 8 loss computations
  • Improves efficiency and stabilizes training

Gaussian Temperature Sampling

• Applies sampling temperature control even in continuous space
  • Improves quality by reducing Gaussian noise variance
  • Good results observed at temperature 0.7

Latent Classifier-Free Guidance (Latent CFG)

• Applies conventional CFG at the latent variable (z) level
  • Improves quality by linearly combining conditional/unconditional outputs
  • Uses α=1.5
  • A similar concept also appears in SoundReactor research

Distillation

• Uses the CFG model as a teacher and distills it into a lightweight student model
  • Freezes the teacher model's MLP head, and the student model learns z with an L2 loss
  • Can shrink from a 24-layer teacher model → 6-layer student model

Conclusion

• Pocket TTS is a lightweight TTS model capable of real-time, high-quality speech synthesis even on CPU
• Combines a continuous latent space-based architecture, efficient training techniques, and voice cloning
• Released as open source under the MIT license, offering reproducibility and extensibility to developers and researchers

See also

• TTS

Favorite site

• Pocket TTS: a high-quality TTS with voice cloning that runs on CPU
• Pocket TTS: A high quality TTS that gives your CPU a voice | Hacker News
• Pocket TTS: A high-quality TTS that gives your CPU a voice | GeekNews