Microsoft patents AI tool that visually tests software for bugs
Microsoft describes a system that uses a machine learning model to look at screenshots of a software environment. If the AI spots a specific object, it investigates that item; if not, it simulates user actions to explore the environment further to find errors.
In the filing’s own words
A computing system for pixel-based automated testing of a navigable simulated environment includes one or more processors configured to execute, in a run-time inference phase, an application testing program. The application testing program includes a machine learning model trained to detect errors in the application under test. A screen image of the simulated environment is processed by an object detection module to determine if a key object is present in the screen image. If a key object is present in the screen image, the application testing program executes an object investigation module to generate investigation inputs to investigate the key object. If a key object is not present in the screen image, the application testing program executes an environment exploration module to generate an environment exploration action to be provided to the application under test as simulated user input.
What exactly they claim (claim 1)
1. A computing system for pixel-based automated testing of a navigable simulated environment generated by an application under test, comprising: a computing device including one or more processors configured to execute instructions using portions of associated memory to implement, during an inference-time phase: a machine learning model trained to detect errors in the application under test, wherein the machine learning model is configured to: receive a screen image of the navigable simulated environment generated by the application under test; and determine if a key object is present in the screen image, wherein the one or more processors is further configured to: if a key object is determined to be present in the screen image, then generate investigation inputs to investigate the key object; and if a key object is not determined to be present in the screen image, then generate an environment exploration action to be provided to the application under test as simulated user input.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- PIXEL-BASED AUTOMATED TESTING OF A NAVIGABLE SIMULATED ENVIRONMENT
- Publication number
- US20260310593A1
- Application number
- 19462636
- Filed
- Jan 28, 2026
- Published
- Oct 8, 2026
- Inventors
- 6
- Claims
- 20
Classified under
G06F 11/3688 Physics & computingG06F 3/04815 Physics & computingG06T 7/70 Physics & computingG06T 19/003 Physics & computingG06T 2207/20081 Physics & computingG06T 2207/20084 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.