Microsoft patents software that finds buttons on your screen automatically
Microsoft describes a system that turns on-screen buttons and menus into digital codes, or tokens, to help a computer understand what is visible. It uses these codes to figure out which button to click based on a specific task, such as opening a file or sending a message.
In the filing’s own words
According to implementations of the present disclosure, a solution for automated navigation of user interface (UI) is provided. According to the solution, for UI elements, tokens representing the UI elements are generated. These UI elements at least include one or more UI elements in a current user interface being presented. The tokens are transformed into respective feature representations of the UI elements using at least specific information corresponding to a current navigation task. Based on the feature representations, a target element is determined for the current navigation task from the UI elements. An operation associated with the target element is performed. In this way, it helps to improve the performance of various navigation tasks by using the specific information of the navigation task.
What exactly they claim (claim 1)
1 - 15. (canceled)
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- USER INTERFACE AUTOMATED NAVIGATION
- Publication number
- US20260359075A1
- Application number
- 19130831
- Filed
- Nov 7, 2023
- Published
- Nov 12, 2026
- Inventors
- 5
- Claims
- 21
Classified under
G06F 9/451 Physics & computingG06F 3/04842 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.
