SonyGranted patentAI & softwareInterest 6/10

Sony patents AI that adds game sounds to videos based on actions

Patent drawing
Patent drawing from the official filing.

Sony describes a system that uses two different machine learning models to analyze video footage. One model looks at specific parts of the image, while the other tracks how things move over time to identify specific actions. The system then automatically finds and inserts matching sound effects from a video game library into the video.

Why it mattersThis could help content creators or game developers automatically add realistic audio to video clips without manually selecting sound effects for every action.

In the filing’s own words

A domain adaptation module is used to optimize a first domain derived from a second domain using respective outputs from respective parallel hidden layers of the domains.

What exactly they claim (claim 1)

1. A method, comprising: generating, using a first machine learning model, one or more feature vectors that represent a set of spatial regions depicted in input video; identifying, based on generating the one or more feature vectors, (i) first output indicating region features corresponding to the set of spatial regions and (ii) second output indicating scene features corresponding to elements outside the set of spatial regions; concatenating the first and second outputs into a set of frame-level feature vectors; providing the set of frame-level feature vectors to a second machine learning model that is different than the first machine learning model and that is trained to model temporal dynamic information; generating, based on output of the second machine learning model processing the set of frame-level feature vectors, a classification of an action depicted in the input video; accessing, from a sound effect database of a video game, a sound effect that matches the classification of the action for inserting into the video game; and modifying the video game by inserting the sound effect that matches the classification of the action into the video game.

Facts

Company
Sony (Sony Interactive Entertainment Inc.)
Official title
Systems and methods for domain adaptation in neural networks using cross-domain batch normalization
Patent number
US12801894B2
Application number
18296926
Filed
Apr 6, 2023
Granted
Nov 17, 2026
Inventors
4
Claims
20

Classified under

G06N 3/045 Physics & computingG06N 3/08 Physics & computingG06N 3/084 Physics & computing

More from Sony in AI & software

Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.