/Patent Tracker
SonyGranted patentAI & softwareInterest 6/10

Sony patents system that describes video game actions using motion data

Sony has been granted a patent for a system that analyzes video game footage to describe what is happening on screen. The technology uses a model to detect specific movements of game objects and combines this with controller input data to generate captions. These captions can be provided as text, audio, or images to describe the action.

Why it mattersThis could improve accessibility for players who rely on audio or visual descriptions of game events, or help automated systems understand gameplay better.

In the filing’s own words

A data processing apparatus for determining description data for describing content includes: a video captioning model to receive an input comprising at least video images associated with the content, wherein the video captioning model is trained to detect one or more predetermined motions of one or more animated objects in the video images and determine one or more captions in dependence on one or more of the predetermined motions, one or more of the captions comprising respective caption data comprising one or more words for describing one or more of the predetermined motions, the respective caption data comprising one or more of audio data, text data and image data; and output circuitry to output description data in dependence on one or more of the captions.

What exactly they claim (claim 1)

1. An apparatus comprising: one or more processors; and one or more computer-readable media having stored thereon instructions that, when executed, cause the one or more processors to, at least: receive an input comprising a video image associated with video game content, the video image showing an object in motion in the video game content; receive controller data associated with the object in motion; detect, utilizing a model, a predetermined motion of the object in motion; map, utilizing the model, the predetermined motion to a plurality of captions based at least in part on the controller data associated with the object in motion, the model having been trained at least in part on a plurality of controller data associated with a plurality of objects in motion; detect an object type represented in the video image, the object type associated with the predetermined motion; and determine, utilizing the model, a caption of the plurality of captions based at least in part on the mapping and the object type, the caption describing the predetermined motion.

Facts

Company
Sony (Sony Interactive Entertainment Inc.)
Official title
Apparatus and methods for content description
Patent number
US12750560B2
Application number
18440024
Filed
Feb 13, 2024
Granted
Sep 29, 2026
Inventors
5
Claims
20

Classified under

H04N 21/4884 Electricity & electronicsG06T 7/20 Physics & computingG10L 13/08 Physics & computingH04N 5/278 Electricity & electronicsG06T 2207/20084 Physics & computing
Read the full filing (official PDF) ↗ Open in the Patent Tracker

More from Sony in AI & software

Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.