Cbp-translate

lastmod 2023-03-28created 2022-12-291 edits

Implementing the real-time translation from Cyberpunk 2077

Features

• In the game, when a character speaks a foreign language, subtitles in the original language appear above the character's head and are then translated into your own language in real time
• Code implementing this feature with DL speech recognition/face recognition and more, plus a detailed explanation of how it works
• OpenAI Whisper + NVDIA NeMo + DeepL + RetinaFace + DeepFace + Sckikit-learn + Gradio + Modal + PIL/OpenCV + ffpmpeg

How it works

• Download the YouTube video with yt-dlp
• Split video/audio with ffmpeg
• Audio:
  • Feed into Whisper to generate transcripts, and translate with DeepL
  • Feed into NeMo to extract speaker IDs (who is speaking)
  • Match the two by timestamp to produce text labeled with speaker IDs
• Video:
  • Extract face locations with MTCNN/RetinaFace. Parallelized with Modal for speed.
  • Extract an embedding for each face with DeepFace/Scikit-Learn and assign unique IDs
• Match FaceIDs and SpeakerIDs with the Jaccard Index
• Annotate frames with PIL/OpenCV using the generated frame metadata
• Generate the video with the ffmpeg Writer

See also

• Deep learning

Favorite site

• Github - cbp-translate