Skip to content
Search

Smart-turn

lastmod 2025-03-10created 2025-03-101 edits

Open-source audio turn detection model

Features

  • Turn detection is one of the most important functions in the voice AI stack: the technique that decides when a voice agent should respond while a person is speaking
  • Most voice agents rely on VAD (Voice Activity Detection) to split audio into speech and non-speech segments, which does not reflect acoustic or linguistic meaning
  • Humans detect turns through complex cues such as grammar, intonation, and speaking pace
    • Goal: build a model closer to human expectations than VAD-based approaches
  • Open-source, community-driven project to develop an audio turn detection model
    • BSD 2-clause license → anyone can use, fork, and contribute
    • The project started in the Pipecat ecosystem
    • Pipecat: open-source, vendor-neutral voice and multimodal AI framework
  • Project goals
    • High-level goals
      • Easy for anyone to use
      • Easy to deploy in production
      • Easy to fine-tune for a specific application
    • Current model limitations
      • English only
      • Relatively slow inference: about 150ms on GPU, about 1500ms on CPU
      • Training data focuses mainly on incomplete speech (filler words) occurring at the end of segments
    • Medium-term goals
      • Support for many languages
      • Inference time: under 50ms on GPU, under 500ms on CPU
      • Training data covering a broader range of speech nuances
      • Build a fully synthetic data generation pipeline
      • Support text-based conditioning (e.g., entering credit card numbers, phone numbers, addresses)
  • Model architecture
    • Based on Meta AI's Wav2Vec2-BERT backbone (580M parameters)
      • Uses 4.5 million hours of unsupervised learning audio data in 143 languages
    • Current model structure:
      • Wav2Vec2-BERT → 2-layer classifier (classification head)
      • Uses Hugging Face's Wav2Vec2BertForSequenceClassification
    • Architectures under experimentation:
      • Testing whether a simple classifier stays effective as the dataset grows
      • Considering introducing more complex structures

Hacker News comments

  • Show HN: Open-source, native audio turn detection model | Hacker News
  • I've used pipecat and liked it. But I switched to sherpa-onnx, which compiles natively and can run on edge devices
    • When using the Google Translate app, I often pause or slow down while saying long sentences, so I avoid conversation mode
    • This problem needs low-latency turn detection and speech interruption detection, plus a very fast, low-latency LLM
    • Needs good recovery so the system can continue the last sentence without discarding earlier audio
    • Improving i/o latency requires low-latency audio APIs, very short audio buffers, and dedicated audio categories and modes
    • Not sure whether TTS can be used in streaming mode
    • A well-designed push-to-talk could be a good solution
  • There were some interesting updates today
    • 100ms inference using CoreML
    • An LSTM model trained on a subset of the data
  • Got most of my answers from the README. It's well written
  • Wondering if you could share the resources and amount needed to fine-tune Wav2Vec2-BERT
  • Wondering what turn detection is
  • Glad to see this technology advancing
    • From the worst voice systems like Siri to ChatGPT voice mode, computers don't do this well
    • It may be the biggest obstacle for 'agents' doing simple but useful tasks
    • There are many situations where AI still struggles, and these errors can destroy conversational efficiency or cause serious functional failures
  • As someone diagnosed with HF autism, I'd like to apply this technology to an earpiece
  • After reviewing several turn-based models, the implementations are very consistent. Looking forward to seeing how this technology develops
  • I hope Vedal integrates this into Neuro-sama's model. A case of going from an osu bot to an AI Vtuber
  • Wondering whether it supports multiple speakers

See also

Favorite site

author Zer0 sitemap