Google patents system that uses AI to find answers in images
Google describes a system that takes an image and a text question, then uses a machine-learned model to figure out what extra information is needed to answer it. The model plans a series of steps, sending the data to different tools to gather more details before putting together a final answer.
In the filing’s own words
Systems and methods for generating a response to a visual information query can include obtaining input data, processing the input data with one or more machine-learned models, transmitting data to one or more data processing tools, and generating a response based on the input data and the one or more outputs of the one or more data processing tools. The one or more machine-learned models may iteratively process data to generate planning data that may include application programming interface calls to the one or more data processing tools. The one or more machine-learned models may be utilized for planning, reasoning, and response generation when obtaining and generating a response to a visual information query.
What exactly they claim (claim 1)
1. A computing system for visual information seeking, the system comprising: one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising: obtaining input data, wherein the input data comprises image data and text data, wherein the text data comprises a query associated with the image data; processing the input data with a machine-learned model to generate first planning data, wherein the first planning data is descriptive of instructions to provide the input data to a first data processing tool; transmitting, based on the first planning data, the input data to the first data processing tool to retrieve first output data; processing the input data and the first output data with the machine-learned model to generate second planning data, wherein the second planning data is descriptive of instructions to transmit data to a second data processing tool; transmitting, based on the second planning data, data to the second data processing tool to retrieve second output data; and processing the input data and the second output data with the machine-learned model to generate response data, wherein the response data is descriptive of a response to the query.
Facts
- Company
- Google (Google LLC)
- Official title
- Autonomous Visual Information Seeking with Machine-learned Language Models
- Publication number
- US20260310717A1
- Application number
- 19477700
- Filed
- Jun 4, 2024
- Published
- Oct 8, 2026
- Inventors
- 6
- Claims
- 30
Classified under
G06F 16/435 Physics & computingMore from Google in AI & software
- Google patents video search that answers questions using AI descriptions Sep 15, 2026
- Google patents photo tool that removes people and their gear Sep 15, 2026
- Google patents AI that writes live meeting notes and summaries Sep 15, 2026
- Google patents AI tool that turns messy emergency notes into clear messages Sep 3, 2026
- Google patents AI that clicks through apps to finish tasks for you Sep 24, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.