Nvidia patents method to animate characters from audio using diffusion models
Nvidia describes a system that uses a diffusion model to create realistic facial animations for digital characters based on audio input. The model processes specific facial parts like the tongue and eyes separately and uses a limited number of processing steps to generate frames quickly.
In the filing’s own words
In various examples, audio-driven character animation using diffusion models is described herein. Systems and methods described herein may use a diffusion model that is trained to perform improved audio-driven character animation for real-time and/or near real-time applications. For instance, the diffusion model may be trained to encode and/or decode components of the face—such as the tongue, the eyes, the gums, and/or the like—separately when animating the character to provide realistic character animation. Additionally, such as during inference, the diffusion model may be configured to process segments of the audio during different processing stages by at least generating and storing state information that relates the processing stages. Furthermore, such as to further reduce the latency, the diffusion model may process data using a specific number of diffusion steps—such as ten or less diffusion steps—when generating denoised frames representing the animated character.
What exactly they claim (claim 1)
1. A method comprising: generating a first representation using a diffusion model and based at least on input data representative of one or more frames associated with a character and audio data associated with a portion of speech; generating, using one or more layers of the diffusion model and based at least on the first representation, second representations of one or more components of a face of the character; generating, using the diffusion model and based at least on concatenating the second representations, one or more denoised frames depicting an animation of a pronunciation by the character of the portion of speech; and causing output of the one or more denoised frames.
Facts
- Company
- Nvidia (NVIDIA Corporation)
- Official title
- AUDIO-DRIVEN CHARACTER ANIMATION USING DIFFUSION MODELS
- Publication number
- US20260278899A1
- Application number
- 19272665
- Filed
- Jul 17, 2025
- Published
- Sep 17, 2026
- Inventors
- 5
- Claims
- 20
Classified under
G06T 13/205 Physics & computingG06T 5/60 Physics & computingG06T 5/70 Physics & computingG06T 13/40 Physics & computingG10L 15/04 Physics & computingMore from Nvidia in AI & software
- Nvidia patents AI coach that analyzes your gameplay to suggest strategies Oct 13, 2026
- Nvidia patents AI system that writes and voices character conversations Sep 10, 2026
- Nvidia patents system that uses AI to find bugs in game videos Sep 17, 2026
- Nvidia patents AI method to make digital characters move realistically Sep 8, 2026
- Nvidia patents AI system that builds image generators from text prompts Sep 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.