Microsoft patents real-time video dubbing that runs directly on your device
Microsoft has been granted a patent for a system that translates video audio into a new language while you watch, without sending data to a server. It works by splitting the audio into small chunks and processing them in parallel on your phone or computer to keep the voiceover in sync with the video.
In the filing’s own words
This disclosure describes a framework for generating real-time audio translations of videos on a client device. Specifically, this disclosure describes a video dubbing system that utilizes a concurrent batch-processing architecture to provide real-time audio translations of videos on a client device. Additionally, in one or more implementations, the video dubbing system utilizes time-aware segmentation to prevent audio misalignment of the translated audio. As described below, the video dubbing system efficiently provides high-quality audio translations of videos that accurately align with the video content for the entire video, regardless of the video's length.
What exactly they claim (claim 1)
1. A computer-implemented method for generating real-time audio translations in one or more videos, comprising: segmenting audio in a first language from a video in a first language into a first audio segment and a second audio segment using an audio segmentation model on a first processing thread of a client device; generating a first translated text string in a second language from the first audio segment using a speech translation model on a second processing thread of the client device; generating a first translated audio segment in the second language from the first translated text string using a text-to-speech model corresponding to the second language on a third processing thread of the client device; concurrent with generating the first translated audio segment on the third processing thread, generating a second translated text string in the second language from the second audio segment using the speech translation model on the second processing thread of the client device; generating a second translated audio segment in the second language from the second translated text string using the text-to-speech model on the third processing thread of the client device; and providing the first translated audio segment with the video.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- Generating real-time audio dubbing for a video using a concurrent batch framework
- Patent number
- US12744031B2
- Application number
- 18917755
- Filed
- Oct 16, 2024
- Granted
- Sep 22, 2026
- Inventors
- 4
- Claims
- 20
Classified under
G10L 15/005 Physics & computingG10L 13/02 Physics & computingG10L 15/26 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.