Skip to content
Search

RealtimeVoiceChat

lastmod 2025-05-08created 2025-05-081 edits

RealtimeVoiceChat - real-time AI voice chat at about 500ms latency

About

  • A system for natural voice conversations with AI in real time using the browser microphone
  • Through an STT → LLM → TTS flow, converts the user's speech to text and turns the AI response back into speech for playback
  • Core components are a FastAPI server, WebSocket streaming, Pod-based speech processing modules, and various LLM backends
  • A Docker-based deployment environment is provided; with a GPU, even lower latency and better performance can be expected
  • Provides a highly customizable environment with advanced settings such as user interrupt detection, model swapping, and voice selection

Real-time AI voice chat

  • This project is a client-server architecture designed for real-time two-way voice conversation
  • The user speaks in the browser, and the AI returns its response as speech
  • Supports interrupt handling, display of partial text responses, and a choice of TTS engines

Main flow

  • Voice input: captures the user's speech in the browser
  • Streaming: sends audio chunks to the Python backend over WebSocket
  • Speech recognition: RealtimeSTT converts speech to text
  • LLM processing: passes the text to the LLM to generate a response
  • Speech synthesis: RealtimeTTS converts the response text to speech
  • Response playback: streams the generated speech back to the browser
  • Interrupt detection: automatically detects and handles the user cutting in

Key features

  • Real-time voice conversation with partial transcription/response preview
  • Audio chunk-based streaming for low latency
  • Supports static/dynamic silence detection (turn detection)
  • Various LLM backends: Ollama by default, OpenAI optional
  • Supports multiple TTS engines: Kokoro, Coqui, Orpheus
  • Web interface: Vanilla JS UI based on the Web Audio API
  • Docker Compose-based deployment

Tech stack

  • Backend: Python 3.x, FastAPI
  • Frontend: HTML, CSS, JavaScript (Web Audio API)
  • Communication: WebSockets
  • Containerization: Docker, Docker Compose
  • AI/ML libraries:
    • RealtimeSTT, RealtimeTTS, transformers, torch, torchaudio
    • ollama, openai
  • Audio processing: numpy, scipy

System requirements and recommendations

  • OS: Docker on Linux recommended (better GPU integration)
  • Python 3.9+, NVIDIA GPU with CUDA 12.1 or later recommended
  • NVIDIA Container Toolkit required when using Docker
  • Configure Ollama or an OpenAI API Key as needed

Installation

  • Clone the repository, then docker compose build
  • Start the app and Ollama with docker compose up -d
  • Download Ollama models separately (e.g. docker compose exec ollama ollama pull ...)
  • Stop the services: docker compose down
  • Restart: docker compose up -d

Option B: Manual installation

  • Set up a Python venv, then install dependencies
  • Manually install PyTorch matching your CUDA version
  • Run server.py to start the FastAPI server

Running

  • Open http://localhost:8000 in the browser
  • Allow microphone permission, then click "Start"
  • "Stop" to end, "Reset" to reset the conversation

Changing settings

  • Change TTS engine/voice: edit in server.py, audio_module.py
  • Change LLM model/backend: configure in server.py, llm_module.py
  • Change STT model/silence thresholds: transcribe.py, turndetect.py
  • SSL can be configured: whether to use HTTPS and certificate settings in server.py

License

  • Released under the MIT license
  • External engines such as Coqui are subject to their own licenses

See also

Favorite site

author Zer0 sitemap