Sony patents system that adds emotion to generated speech using video cues
Sony describes a device that creates artificial speech by analyzing both audio and video from a person. It extracts emotional cues like facial expressions and gestures, then uses these to make the generated voice sound more natural and synchronized with the original performance.
In the filing’s own words
An information processing device for generating artificial speech data has circuitry which is configured to obtain, based on speech data, speech emotional indicators and associated timing data of the emotional indicators; and to obtain, based on the speech data, text data; and to generate artificial speech data based on the text data, the speech emotional indicators and the associated timing data.
What exactly they claim (claim 1)
1. An information processing device for generating artificial speech data, comprising circuitry configured to: obtain, based on speech data, speech emotional indicators and associated timing data of the emotional indicators; obtain, based on the speech data, text data; obtain, based on video data associated with the speech data, video emotional indicators comprising at least one of facial expressions, eye direction, gestures, and body pose; associate the speech emotional indicators and the video emotional indicators with the text data based on the associated timing data; encode the speech emotional indicators and the video emotional indicators into a synchronized latent feature space matched to timestamps of the text data: generate artificial speech data based on the synchronized latent feature space, the text data, the speech emotional indicators associated with the text data, the video emotional indicators associated with the text data, and the associated timing data.
Facts
- Company
- Sony (Sony Group Corporation)
- Official title
- Information processing device and information processing method for artificial speech generation
- Patent number
- US12763063B2
- Application number
- 18404943
- Filed
- Jan 5, 2024
- Granted
- Oct 13, 2026
- Inventors
- 1
- Claims
- 18
Classified under
G10L 13/027 Physics & computingG10L 13/033 Physics & computingG10L 25/30 Physics & computingG10L 25/63 Physics & computingMore from Sony in AI & software
- Sony patents AI that plays video games in your style Oct 6, 2026
- Sony patents AI that swaps game visuals for accessibility formats Sep 17, 2026
- Sony patents AI that changes game characters based on your play Oct 1, 2026
- Sony patents method to turn 2D videos into high-detail 3D animations Oct 15, 2026
- Sony patents AI bots that mimic your friends' gaming styles Sep 8, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.