/Patent Tracker
GoogleGranted patentAI & softwareInterest 8/10

Google patents video search that answers questions using AI descriptions

This patent describes a system that picks important frames from a video and uses a vision model to write text descriptions of them. When a user asks a question while watching, the system combines the question with these descriptions to generate an answer.

Why it mattersIt allows users to ask specific questions about what is happening in a video in real-time, rather than just searching for the video itself.

In the filing’s own words

Implementations utilize a LLM to generate content responsive to a user query directed to a video and cause audio data for the generated content to be rendered as a response to the user query. Implementations extract a subset of frames from all frames of the video as key frame(s) for the video, and utilize a vision-language model in generating a natural language description for the key frame(s) of the video. A prompt can be generated based on a transcription of the user query and based on the natural language description for the key frame(s) of the video. The prompt is processed as input, using the LLM, to generate the content responsive to the user query directed to the video.

What exactly they claim (claim 1)

1. A method implemented by one or more processors, the method comprising: determining a plurality of key image frames from a video, wherein the plurality of key image frames includes a past key image frame that occurs earlier than a current image frame in the video and a future key image frame that occurs later than the current image frame in the video; processing the plurality of key image frames, including the past and future key image frames, using a vision-language model, wherein processing the plurality of key image frames using the vision-language model causes a natural language description of the past and future key image frames to be generated using the vision-language model; storing the natural language description for the plurality of key image frames in association with the video; receiving, from a computing device, a user query related to the video, wherein the user query is received when the current image frame of the video is being rendered; and in response to receiving the user query, generating a prompt based on the user query and based on the natural language description for the plurality of key image frames of the video, processing the prompt as input using a generative model, to generate a generative model output, wherein the generative model output is operable to cause a response responsive to the user query to be rendered by an output device, and providing the generative model output to the computing device.

Facts

Company
Google (GOOGLE LLC)
Official title
Converting video semantics into language for real-time query and information retrieval
Patent number
US12737414B2
Application number
18535554
Filed
Dec 11, 2023
Granted
Sep 15, 2026
Inventors
1
Claims
7

Classified under

G06F 16/583 Physics & computingG06F 40/30 Physics & computingG06V 10/764 Physics & computing
Read the full filing (official PDF) ↗ Open in the Patent Tracker

More from Google in AI & software

Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.