/Patent Tracker
GoogleGranted patentAI & softwareInterest 6/10

Google patents image search hints to help AI models see better

Patent drawing
Patent drawing from the official filing.

This patent describes a system that combines a visual search engine with an AI model that understands images and text. Before the AI model analyzes a photo, the search engine draws boxes around objects in the picture and adds numbers to them, giving the AI extra clues about what is in the image.

Why it mattersThis approach could make image-based AI assistants more accurate by providing them with structured information about the objects they are looking at.

In the filing’s own words

Provided are computer-implemented systems and methods for responding to visual queries using both a visual search engine and a vision language model (VLM). In particular, aspects of the present disclosure can improve the performance of a VLM at generating a response to a visual queries by supplementing the visual query with one or more annotations generated by or using the visual search engine.

What exactly they claim (claim 1)

1. A computer-implemented method for responding to visual queries, the method comprising: receiving, by a computing system comprising one or more computing devices, a query comprising one or more query images; processing, by the computing system, the one or more query images with a visual search engine to identify, by the visual search engine, one or more context items based on the one or more query images, wherein the one or more context items comprise bounding shapes for one or more objects depicted in the one or more query images; generating, by the computing system, one or more annotations based on the one or more context items; annotating, by the computing system, the query with the one or more annotations to generate an annotated query, wherein annotating the query with the one or more annotations comprises overlaying the one or more bounding shapes upon at least one of the one or more query images and adding reference numbers to the one or more bounding shapes and modifying the query to contain textual references for the reference numbers; and processing, by the computing system, the annotated query with a machine-learned vision language model to generate, as an output of the machine-learned vision language model, a response to the annotated query.

Facts

Company
Google (Google LLC)
Official title
Enhancing vision language model understanding via visual search service-derived annotations
Patent number
US12769802B2
Application number
19017157
Filed
Jan 10, 2025
Granted
Oct 20, 2026
Inventors
9
Claims
20

Classified under

G06F 16/9538 Physics & computingG06F 16/532 Physics & computing
Read the full filing (official PDF) ↗ Open in the Patent Tracker

More from Google in AI & software

Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.