Google patents AI that clicks through apps to finish tasks for you
Google has filed a patent for a system that uses a language model to automatically complete tasks on a computer or phone. The AI looks at a screenshot of the current screen and combines that visual data with the user's goal to decide which buttons to click or text to type next. It repeats this process, taking a new screenshot after each action, until the task is finished.
In the filing’s own words
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for automating the interaction with user interfaces for task execution using a language model neural network. In one aspect, a method comprises receiving an input that describes a task; determining data describing the task; and executing the task by using a language model neural network by repeatedly performing following operations: obtaining a first input subsequence that is derived from the data describing the task; obtaining a second input subsequence that is derived from an image of a current user interface; generating, by the language model neural network, based on the first input subsequence and the second input subsequence, output data that describes one or more actions to be performed within the current user interface; and performing the one or more actions within the current user interface.
What exactly they claim (claim 1)
1. A method comprising: receiving, at a computing device, an input that describes a task; determining, from the input, data describing the task; and executing the task by the computing device by using a language model neural network, wherein executing the task comprises repeatedly performing following operations until a termination criterion is met: obtaining a first input subsequence that is derived from the data describing the task; obtaining a second input subsequence that is derived from an image of a current user interface displayed on the computing device; generating, by the language model neural network, based on the first input subsequence and the second input subsequence, output data that describes one or more actions to be performed within the current user interface; and causing the current user interface to transition into a successive user interface by performing the one or more actions within the current user interface.
Facts
- Company
- Google (Google LLC)
- Official title
- TASK COMPLETION USING A LANGUAGE MODEL NEURAL NETWORK
- Publication number
- US20260289251A1
- Application number
- 19474033
- Filed
- Apr 11, 2023
- Published
- Sep 24, 2026
- Inventors
- 2
- Claims
- 20
Classified under
G06N 3/0475 Physics & computingG06N 3/088 Physics & computingMore from Google in AI & software
- Google patents video search that answers questions using AI descriptions Sep 15, 2026
- Google patents photo tool that removes people and their gear Sep 15, 2026
- Google patents AI that writes live meeting notes and summaries Sep 15, 2026
- Google patents AI tool that turns messy emergency notes into clear messages Sep 3, 2026
- Google patents editing photos by typing what you want to change Sep 24, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.