MicrosoftPatent applicationAI & softwareInterest 6/10

Microsoft patents software that finds buttons on your screen automatically

Patent drawing
Patent drawing from the official filing.

Microsoft describes a system that turns on-screen buttons and menus into digital codes, or tokens, to help a computer understand what is visible. It uses these codes to figure out which button to click based on a specific task, such as opening a file or sending a message.

Why it mattersThis could make computer programs better at performing routine tasks by themselves, reducing the need for users to click through menus manually.

In the filing’s own words

According to implementations of the present disclosure, a solution for automated navigation of user interface (UI) is provided. According to the solution, for UI elements, tokens representing the UI elements are generated. These UI elements at least include one or more UI elements in a current user interface being presented. The tokens are transformed into respective feature representations of the UI elements using at least specific information corresponding to a current navigation task. Based on the feature representations, a target element is determined for the current navigation task from the UI elements. An operation associated with the target element is performed. In this way, it helps to improve the performance of various navigation tasks by using the specific information of the navigation task.

What exactly they claim (claim 1)

1 - 15. (canceled)

Facts

Company
Microsoft (Microsoft Technology Licensing, LLC)
Official title
USER INTERFACE AUTOMATED NAVIGATION
Publication number
US20260359075A1
Application number
19130831
Filed
Nov 7, 2023
Published
Nov 12, 2026
Inventors
5
Claims
21

Classified under

G06F 9/451 Physics & computingG06F 3/04842 Physics & computing

More from Microsoft in AI & software

Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.