Microsoft patents method to test if AI models truly reason or just memorize
Microsoft describes a technique that turns word problems into mathematical symbols, changes those symbols to create new questions, and then asks an AI model to solve them. This allows developers to check if the model can actually think through the logic or if it is just recalling answers it has seen before.
In the filing’s own words
In one aspect herein, a symbolic representation is generated from a natural language reasoning problem, which can then be mutated to produce a symbolic problem variant and a natural language problem variant. This data synthesis techniques provides reasoning data that in some example applications is used to test a model's reasoning capabilities. Example applications include comparing model-generated answers against programmatically derived ground truth to distinguish genuine reasoning from memorized patterns. As another example, certain training applications repeatedly expose a model to newly synthesized data for iterative refinement.
What exactly they claim (claim 1)
1. A computer-implemented method, comprising: generating, based on a reasoning problem, a symbolic representation of the reasoning problem, the symbolic representation comprising a first variable, a second variable and a dependency between the first variable and the second variable; generating a mutation of the symbolic representation, resulting in a symbolic problem variant; generating based on the symbolic problem variant a reasoning problem variant; and testing, using the reasoning problem variant, reasoning capability of a reasoning model.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- TESTING, TRAINING OR VALIDATING REASONING MODELS
- Publication number
- US20260259817A1
- Application number
- 19096501
- Filed
- Mar 31, 2025
- Published
- Sep 3, 2026
- Inventors
- 10
- Claims
- 20
Classified under
G06F 11/3608 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.