Microsoft patents AI that answers location questions using photos and GPS
This filing describes a system that combines a live photo, GPS coordinates, and a user's question to create a specific data point called a multimodal embedding. This data point is sent to a generative AI model, which uses it to compare the current view with known visual features of that location to craft an answer.
In the filing’s own words
This disclosure describes a location-based response system that utilizes multimodal embedding and generative artificial intelligence (AI) models to generate location-based responses for user queries. For example, the location-based response system receives a real-time image, real-time location data, and a user query from a client device and determines a multimodal embedding in a multimodal vector embedding space based on the real-time image and the real-time location data. The multimodal embedding is then provided to a generative AI model, together with the data received from the client device, with instructions to generate a user query response. Upon receiving the user query response from the generative AI model, the system may provide it to the client device. Indeed, the location-based response system generates location-based responses for user queries by utilizing identified multimodal embedding and by instructing a generative AI model to respond to the user query based on the identified multimodal embedding.
What exactly they claim (claim 1)
1. A computer-implemented method for generating location-based responses from multimodal embeddings using one or more generative artificial intelligence (AI) models, the computer-implemented method comprising: receiving a real-time image, real-time location data, and a user query from a client device; determining a multimodal embedding in a multimodal vector embedding space based on the real-time image and the real-time location data, wherein the multimodal embedding represents a location associated with the real-time location data and includes visual features of the location generated from one or more previous images of the location; providing the real-time image, the real-time location data, the user query, and the multimodal embedding to a generative AI model with instructions to generate a user query response to the user query based on the multimodal embedding by comparing visual features of the real-time image with the visual features of the location represented by the multimodal embedding; and providing the user query response to the client device in response to receiving the user query response from the generative AI model.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- GENERATING LOCATION-BASED RESPONSES BASED ON MULTIMODAL EMBEDDINGS AND GENERATIVE ARTIFICIAL INTELLIGENCE (AI) MODELS
- Publication number
- US20260300300A1
- Application number
- 19096285
- Filed
- Mar 31, 2025
- Published
- Oct 1, 2026
- Inventors
- 8
- Claims
- 20
Classified under
G06F 16/24575 Physics & computingG06V 10/40 Physics & computingG06V 10/751 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.