Microsoft patents AI system that switches computing modes during text generation
Microsoft describes a system that manages how a generative AI model uses computer memory while running. It skips a specific memory component when processing the initial input but uses it when generating the response word by word.
In the filing’s own words
Methods, systems, and computer storage media for providing compute management using a compute management engine in an artificial intelligence (AI) system. A compute management engine supports dynamically switching between two modes of operation for an inference phase of a generative artificial AI model. The compute management engine employs a bypass engine that causes prompt stage operations to be executed without an in-memory compute engine and causes auto-regression stage operations to be executed with the in-memory compute engine. In operation, an inference phase operation is accessed. When the inference phase operation is a prompt stage operation, the inference phase operation is executed without an in-memory compute engine. When the inference phase operation is an auto-regressive stage operation, the inference phase operation is executed with the in-memory compute engine. Memory output is generated for the inference phase operation to cause a processor to output a processor output for the inference phase operation.
What exactly they claim (claim 1)
1. A computerized system comprising: one or more computer processors; and computer memory storing computer-useable instructions that, when used by the one or more computer processors, cause the one or more computer processors to perform operations, the operations comprising: accessing a prompt stage operation associated with a generative artificial intelligence (AI) model, the generative AI model is associated with memory that supports executing operations associated with the generative AI model; based on the prompt stage operation, causing the prompt stage operation to bypass an in-memory compute engine of the memory; executing the prompt stage operation without using the in-memory compute engine; accessing an auto-regressive stage operation associated with the generative AI model; based on the auto-regressive stage operation, cause the auto-regressive stage operation to be executed using the in-memory compute engine of the memory; and executing the auto-regressive stage operation using the in-memory compute engine.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- COMPUTE MANAGEMENT ENGINE IN AN ARTIFICIAL INTELLIGENCE SYSTEM
- Publication number
- US20260359563A1
- Application number
- 19736442
- Filed
- Jul 9, 2026
- Published
- Nov 12, 2026
- Inventors
- 2
- Claims
- 20
Classified under
G06F 40/20 Physics & computingG06F 17/16 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.
