Microsoft patents AI assistant that reads app code to perform tasks
Microsoft describes a system where an AI assistant reads the code of an application to understand its structure. It uses this information to match user requests to specific actions, allowing the AI to click buttons or fill fields on the user's behalf.
In the filing’s own words
A computing system including one or more processing devices configured to execute a machine learning (ML) assistant interface. The processing devices receive a user input that specifies an interaction with an application interface. The processing devices receive application interface code of the application interface, and based on the application interface code, select an ontology schema from a plurality of ontology schemas. Based on the user input and the ontology schema, the processing devices select an intent schema from a plurality of intent schemas. The processing devices populate the ontology schema with interactable elements of the application interface identified in the application interface code, and, based on the populated ontology schema and the user input, populate the intent schema with one or more interface element interactions. At the application interface, the processing devices perform the interface element interactions at an interactable element specified in the populated ontology schema.
What exactly they claim (claim 1)
1. A computing system comprising: one or more processing devices configured to: execute a machine learning (ML) assistant interface over which a plurality of conversational turns are exchanged between a user and an ML system that includes a generative language model; at the ML system: via the ML assistant interface, receive a user input that specifies an interaction with an application interface of an application program; receive application interface code of the application interface; based at least in part on the application interface code, select an ontology schema from among a plurality of ontology schemas; based at least in part on the user input and the ontology schema, select an intent schema from among a plurality of intent schemas; populate the ontology schema with a plurality of interactable elements of the application interface identified in the application interface code; and based at least in part on the populated ontology schema and the user input, populate the intent schema with one or more interface element interactions; and at the application interface, programmatically perform the one or more interface element interactions specified by the populated intent schema at an interactable element of the plurality of interactable elements specified in the populated ontology schema.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- MACHINE LEARNING SYSTEM WITH ONTOLOGY SCHEMAS AND INTENT SCHEMAS
- Publication number
- US20260277985A1
- Application number
- 19080618
- Filed
- Mar 14, 2025
- Published
- Sep 17, 2026
- Inventors
- 2
- Claims
- 20
Classified under
G06F 16/367 Physics & computingG06F 16/3347 Physics & computingG06F 40/20 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.