Sony patents AI that adds game sounds to videos based on actions
Sony describes a system that uses two different machine learning models to analyze video footage. One model looks at specific parts of the image, while the other tracks how things move over time to identify specific actions. The system then automatically finds and inserts matching sound effects from a video game library into the video.
In the filing’s own words
A domain adaptation module is used to optimize a first domain derived from a second domain using respective outputs from respective parallel hidden layers of the domains.
What exactly they claim (claim 1)
1. A method, comprising: generating, using a first machine learning model, one or more feature vectors that represent a set of spatial regions depicted in input video; identifying, based on generating the one or more feature vectors, (i) first output indicating region features corresponding to the set of spatial regions and (ii) second output indicating scene features corresponding to elements outside the set of spatial regions; concatenating the first and second outputs into a set of frame-level feature vectors; providing the set of frame-level feature vectors to a second machine learning model that is different than the first machine learning model and that is trained to model temporal dynamic information; generating, based on output of the second machine learning model processing the set of frame-level feature vectors, a classification of an action depicted in the input video; accessing, from a sound effect database of a video game, a sound effect that matches the classification of the action for inserting into the video game; and modifying the video game by inserting the sound effect that matches the classification of the action into the video game.
Facts
- Company
- Sony (Sony Interactive Entertainment Inc.)
- Official title
- Systems and methods for domain adaptation in neural networks using cross-domain batch normalization
- Patent number
- US12801894B2
- Application number
- 18296926
- Filed
- Apr 6, 2023
- Granted
- Nov 17, 2026
- Inventors
- 4
- Claims
- 20
Classified under
G06N 3/045 Physics & computingG06N 3/08 Physics & computingG06N 3/084 Physics & computingMore from Sony in AI & software
- Sony patents AI that plays video games in your style Oct 6, 2026
- Sony patents stadium seats that turn crowd actions into on-screen graphics Nov 10, 2026
- Sony patents AI that swaps game visuals for accessibility formats Sep 17, 2026
- Sony patents AI that changes game characters based on your play Oct 1, 2026
- Sony patents method to turn 2D videos into high-detail 3D animations Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.
