Google patents video search that answers questions from spoken words
This filing describes a system that turns video content into a structured text document by transcribing speech and reading text from images. It then uses a large language model to answer user questions based on this combined information.
In the filing’s own words
A method includes receiving a content feed that includes audio data corresponding to speech utterances and processing the content feed to generate a semantically-rich, structured document. The structured document includes a transcription of the speech utterances and includes a plurality of words each aligned with a corresponding audio segment of the audio data that indicates a time when the word was recognized in the audio data. During playback of the content feed, the method also includes receiving a query from a user requesting information contained in the content feed and processing, by a large language model, the query and the structured document to generate a response to the query. The response conveys the requested information contained in the content feed. The method also includes providing, for output from a user device associated with the user, the response to the query.
What exactly they claim (claim 1)
1. A computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations comprising: receiving a content feed comprising audio data and image data, the audio data corresponding to speech utterances; processing the content feed to generate a semantically-rich, structured document, wherein the structured document comprises: a transcription of the speech utterances, the transcription comprising a plurality of words each aligned with a corresponding audio segment of the audio data; and creator-provided text recognized in one or more image frames of the image data via optical character recognition; grouping the transcription and the creator-provided text into one or more sections of the structured document based on the corresponding audio segments to consolidate content corresponding to a respective time in the content feed; receiving a query from a user requesting information contained in the content feed; and processing, using a large language model, the natural language query and the semantically-rich, structured document as context to generate a response to the query.
Facts
- Company
- Google (Google LLC)
- Official title
- STRUCTURED VIDEO DOCUMENTS
- Publication number
- US20260267916A1
- Application number
- 19654023
- Filed
- Apr 21, 2026
- Published
- Sep 10, 2026
- Inventors
- 2
- Claims
- 20
Classified under
G06F 16/7844 Physics & computingG06F 16/739 Physics & computingG06F 40/169 Physics & computingG06F 40/30 Physics & computingMore from Google in AI & software
- Google patents video search that answers questions using AI descriptions Sep 15, 2026
- Google patents photo tool that removes people and their gear Sep 15, 2026
- Google patents AI that writes live meeting notes and summaries Sep 15, 2026
- Google patents AI tool that turns messy emergency notes into clear messages Sep 3, 2026
- Google patents AI that clicks through apps to finish tasks for you Sep 24, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.