VoiceBridge: An AI-powered framework for low-cost multilingual video dubbing into Indian regional languages
VoiceBridge is an AI-powered, low-cost multilingual dubbing framework specifically built to make English video content available in various Indian regional languages, specifically South Indian languages: Kannada, Tamil, Telugu, and Malayalam. VoiceBridge combines bleeding- edge open-source technologies such as Whisper for Automatic Speech Recognition (ASR), IndicTrans2 for Text Translation (TT), and Coqui or Indic-TTS for Text-to-Speech (TTS), to create an end-to-end pipeline of transcription, translation, speech synthesis, and video dubbing that is affordable, culturally relevant, and easily scalable. The framework features a simplified interface that allows users to upload videos, translate speech, and produce dubbed outputs without having to have any background knowledge of the processes or technologies being used. Evaluation of performance gave promising results, arriving at a Word Error Rate (WER) of 11.9% and Character Error Rate (CER) of 11.09%, showing significant levels of recognition and translation accuracy despite minor differences in pronunciation and a 90millisecond audio-video delay. VoiceBridge utilizes open-source models and adapts those for low-resource languages to serve as abridge towards mitigating the digital languages gap, and as a means to provide access to educational and informational content in video format to various linguistic communities.