Google patents AI that navigates app screens to finish tasks
This filing describes a system where a user asks a computer to perform a task using plain language. The software uses a machine-learning model to map out the different screens or states of an app, figuring out the specific screen needed to complete the request.
In the filing’s own words
Implementations herein relate to a process for performing, by a computing device, a task based on a natural language query from a user. The natural language query can be received from a user at an initial state of a user interface (UI) of an electronic device, and can relate to a target task performable at a terminal state of the UI. A multi-layer data structure related to a plurality of states of the UI can then be identified. This plurality of UI states can include at least the initial state of the UI, the terminal state of the UI, and one or more intermediate states. The natural language query and the multi-layer data structure can then be processed, via a multi-modal machine-learning model, to identify the terminal state of the UI at which the target task is performable. Other implementations can be described and/or claimed.
What exactly they claim (claim 1)
1. A method implemented by one or more processors, the method comprising: identifying a natural language query received from a user at an initial state of a user interface (UI) of an electronic device, wherein the natural language query relates to a target task performable at a terminal state of the UI; identifying a multi-layer data structure related to a plurality of states of the UI, wherein the plurality of states of the UI include at least the initial state of the UI, the terminal state of the UI, and one or more intermediate states of the UI that are between the initial state of the UI and the terminal state of the UI; processing, via a multi-modal machine-learning model, the natural language query and the multi-layer data structure to identify the terminal state of the UI at which the target task is performable; and causing, based on the identified terminal state of the UI, the target task to be performed.
Facts
- Company
- Google (GOOGLE LLC)
- Official title
- AGENT TRAVERSAL OF MULTI-LAYER DATA STRUCTURE
- Publication number
- US20260310472A1
- Application number
- 19172203
- Filed
- Apr 7, 2025
- Published
- Oct 8, 2026
- Inventors
- 1
- Claims
- 20
Classified under
G06F 9/451 Physics & computingG06F 16/243 Physics & computingG06F 16/986 Physics & computingG06F 3/0482 Physics & computingG06F 3/0484 Physics & computingMore from Google in AI & software
- Google patents video search that answers questions using AI descriptions Sep 15, 2026
- Google patents photo tool that removes people and their gear Sep 15, 2026
- Google patents AI that writes live meeting notes and summaries Sep 15, 2026
- Google patents AI tool that turns messy emergency notes into clear messages Sep 3, 2026
- Google patents AI that clicks through apps to finish tasks for you Sep 24, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.