Samsung patents 3D avatars that match facial and body moves to speech
Samsung describes a system that creates a 3D virtual character that moves its face and body in sync with spoken audio. It uses AI to read the speaker's emotions from text and predict specific mouth shapes, eye movements, and body gestures to make the avatar look natural.
In the filing’s own words
Multi-modal artificial intelligence (AI) generation of synchronized face and full-body animation for a three-dimensional (3D) avatar includes obtaining input audio and text data, extracting audio features, and extracting emotion features using a multilingual language model trained to classify emotion type and corresponding intensity from text. A facial expression generation model predicts phonemes, phoneme/word boundaries for lip-sync activity, facial blendshape weights for facial expressions, iris movements and eye blink activity for eye movements, and/or orientation of a neck joint for avatar head movements. Final facial blendshape weights for a specified facial expression are determined based on the predicted phonemes, facial blendshape weights and/or the emotion features. A gesture generation model predicts a 3D position of a body center and orientation of body joints. Final 3D full-body animation keyframes of avatar body gestures, synchronizing body motion intensity with lip-sync activity, and final facial blendshape weights are presented synchronously with corresponding audio.
What exactly they claim (claim 1)
1. A method comprising: obtaining, using at least one processing device of an electronic device, input audio data and input text data; extracting, using the at least one processing device, audio features from the input audio data using an audio encoder; extracting, using the at least one processing device, emotion features for the input text data using a multilingual language model trained to classify emotion type and emotion intensity from text; providing, using the at least one processing device, the audio features to a facial expression generation model trained to predict at least one of phonemes, phoneme/word boundaries for lip-sync activity, facial blendshape weights for facial expressions, iris movements and eye blink activity for eye movements, and orientation of a neck joint for head movements of an avatar; determining, using the at least one processing device, final facial blendshape weights for a specified facial expression of the avatar based on at least one of the predicted phonemes, the predicted facial blendshape weights, and the emotion features; providing, using the at least one processing device, the audio features and the emotion features to a gesture generation model trained to predict at least one of a three-dimensional (3D) position of a body center and orientation of body joints; generating, using the at least one processing device, final 3D full-body animation keyframes of body gestures of the avatar based on at least one of the predicted phoneme/word boundaries, the predicted 3D position of the body center, and the predicted orientation of the body joints; and playing, using the at least one processing device, the input audio data and synchronously presenting the final facial blendshape weights and the final 3D full-body animation keyframes.
Facts
- Company
- Samsung (Samsung Electronics Co., Ltd.)
- Official title
- ON-DEVICE MULTI-MODAL AI WITH SYNCHRONIZED FACE AND FULL-BODY ANIMATION FOR 3D VIRTUAL ASSISTANTS
- Publication number
- US20260311411A1
- Application number
- 19305538
- Filed
- Aug 20, 2025
- Published
- Oct 8, 2026
- Inventors
- 4
- Claims
- 20
Classified under
G06T 13/205 Physics & computingG06F 40/242 Physics & computingG06F 40/40 Physics & computingG06T 13/40 Physics & computingG10L 13/02 Physics & computingG10L 13/08 Physics & computingMore from Samsung in AI & software
- Samsung patents tool to erase objects from photos using AI Sep 24, 2026
- Samsung patents drawing objects into photos with AI Oct 15, 2026
- Samsung patents AI that creates personalized exercise videos Sep 10, 2026
- Samsung patents AI summaries for paused TV shows based on your viewing habits Sep 24, 2026
- Samsung patents turning chat conversations into image prompts Oct 1, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.