/Patent Tracker
MicrosoftGranted patentAI & softwareInterest 7/10

Microsoft patents indoor navigation using photos and language models

Microsoft describes a system that helps people find their way inside large buildings like offices or hospitals. It works by comparing a photo taken by the user to a database of labeled images and floorplans, then using a language model to give spoken or written directions to a specific destination.

Why it mattersThis could make finding your way in complex indoor spaces easier, similar to how GPS works outside but adapted for buildings where GPS signals are weak.

In the filing’s own words

The techniques presented herein are directed to a system for automated navigation assistance in indoor environments. In general, navigating an unfamiliar indoor environment can be a daunting task. This is especially true for large environments such as corporate offices, university campuses, shopping centers, and hospitals. Accordingly, the present system utilizes a reference database containing information associated with the indoor environment such as images depicting various locations, annotations describing the images, and floorplans illustrating the layout of the indoor environment. The reference database is deployed in conjunction with a multimodal language model that utilizes the collected knowledge of the reference database to provide location identification and intuitive navigation instructions. Often referred to as retrieval-augmented generation (RAG), the reference database enables the multimodal language model to respond to user requests using domain-specific information and ensure accurate outputs that are relevant to an end user's context (e.g., the indoor environment).

What exactly they claim (claim 1)

1. A method for navigation assistance in an indoor environment comprising: receiving a reference image depicting a location within the indoor environment, wherein the reference image includes a natural language annotation describing the location depicted by the reference image; calculating a numerical representation of the reference image capturing a semantic content of the reference image and the natural language annotation; storing the numerical representation in a reference database that is configured to store a plurality of numerical representations of reference images and annotations associated with the indoor environment; receiving an input image from a user device, the input image depicting a current position of the user device within the indoor environment; in response to receiving the input image from the user device, identifying, by a multimodal language model, the current position of the user device based on a comparison of a numerical representation of the input image against the plurality of numerical representations stored by the reference database; receiving a natural language input from the user device, wherein the natural language input includes a user-defined objective; in response to receiving the natural language input from the user device: identifying, by the multimodal language model, a destination within the indoor environment from the user-defined objective; and outputting, by the multimodal language model, a sequence of natural language directions defining a path from the current position of the user device to the destination identified from the user-defined objective.

Facts

Company
Microsoft (MICROSOFT TECHNOLOGY LICENSING, LLC)
Official title
Location and navigation assistant
Patent number
US12747950B2
Application number
19059968
Filed
Feb 21, 2025
Granted
Sep 29, 2026
Inventors
2
Claims
20

Classified under

G01C 21/206 Physics & computingG06F 40/20 Physics & computingG06K 7/1417 Physics & computing
Read the full filing (official PDF) ↗ Open in the Patent Tracker

More from Microsoft in AI & software

Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.