Microsoft patents method to check if AI instructions help users succeed
Microsoft has patented a system that watches how users interact with software to judge whether AI-generated instructions are working. It compares the user's actual clicks and actions against key steps predicted by the AI to calculate a success score. If the score is too low, the system triggers a fix to improve the experience.
In the filing’s own words
A method of AI content evaluation includes receiving, from a generative artificial intelligence (AI) model, a set of AI-generated instructions that identifies steps for performing a task within an application, and selecting checkpoint interactions from an interaction index that define a plurality of interactions with a user interface. Each of the checkpoint interactions satisfies a similarity metric with a corresponding step in the set of AI-generated instructions. The method further includes determining, based on detected user interactions with the user interface, a subset of the checkpoint interactions completed by a user within an observation period, and evaluating a metric that to compute a quality score that quantifies user success with respect to performing the task associated with the AI-generated instructions. The metric depending at least in part on the subset of the checkpoint interactions completed by the user within the observation period. In response to determining that the quality score satisfies low-quality criteria, a remedial action is performed.
What exactly they claim (claim 1)
1. A method comprising: receiving, from a generative artificial intelligence (AI) model, a set of AI-generated instructions that identifies steps for performing a task within an application; selecting checkpoint interactions from an interaction index that defines a plurality of interactions with a user interface of the application, each of the checkpoint interactions satisfying a similarity metric with a corresponding step in the set of AI-generated instructions; determining, based on detected user interactions with the user interface, a subset of the checkpoint interactions completed by a user within an observation period; evaluating a metric that quantifies user success with respect to performing the task, the metric depending at least in part on the subset of the checkpoint interactions completed by the user within the observation period; and based on the metric satisfying predefined criteria, performing a remedial action that is effective to increase a user success rate with respect to performing the task within the application.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- User interface action tracking for quality evaluation of AI-generated content
- Patent number
- US12730705B2
- Application number
- 18822562
- Filed
- Sep 3, 2024
- Granted
- Sep 8, 2026
- Inventors
- 2
- Claims
- 20
Classified under
G06F 11/0793 Physics & computingG06F 11/0769 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.