Disney patents tech to sync translated speech with body gestures
Disney filed a patent for a system that translates video content by separating audio and video tracks. It identifies hand gestures and lip movements in the original video, then adjusts the timing of the translated audio or the gestures so they match up correctly in the new language.
In the filing’s own words
Multilingual multimedia translation by generating a synchronized multimedia asset based on a multimedia asset that is translated into a different language. The multilingual multimedia translation segments the multimedia asset into an audio segment and a video segment. The multilingual translation uses the audio segment to generate a translated audio segment. The multilingual translation may use the video segment identify body gestures of individuals in the video segment, generate translated body gestures, and generate translated lip movements. Using the translated audio segment and any translated lip movements or translated body gestures, the multilingual translation generates a synchronized multimedia asset.
What exactly they claim (claim 1)
1. A method, comprising: accessing a multimedia asset depicting an individual speaking in a first language of a plurality of languages; extracting an audio segment of the individual speaking in the multimedia asset; identifying a first gesture depicted in the multimedia asset during a first portion of the audio segment; determining a first temporal alignment between an utterance made in the first portion of the audio segment and the first gesture; and generating a synchronized multimedia asset based on at least one of: modifying a temporal location of the first gesture in the multimedia asset or modifying a timing of a translated audio segment to set a second temporal alignment between the first gesture and a translation of the utterance in the translated audio segment.
Facts
- Company
- Disney (Disney Enterprises, Inc.)
- Official title
- MULTILINGUAL MULTIMEDIA TRANSLATION WITH SYNCHRONIZED AUDIO AND BODY GESTURES
- Publication number
- US20260281513A1
- Application number
- 19075907
- Filed
- Mar 11, 2025
- Published
- Sep 17, 2026
- Inventors
- 1
- Claims
- 20
Classified under
H04N 21/8146 Electricity & electronicsG06V 40/20 Physics & computingH04N 21/8106 Electricity & electronicsMore from Disney in AI & software
- Disney patents AI character that creates art from your shared memories Sep 8, 2026
- Disney patents video compression using generative models to cut file sizes Sep 3, 2026
- Disney patents AI characters that create art from shared memories Oct 22, 2026
- Disney patents AI characters that pause and react when you interrupt them Sep 24, 2026
- Disney patents video system to detect when guests need help Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.