MicrosoftPatent applicationAI & softwareInterest 6/10

Microsoft patents AI system that switches computing modes during text generation

Patent drawing
Patent drawing from the official filing.

Microsoft describes a system that manages how a generative AI model uses computer memory while running. It skips a specific memory component when processing the initial input but uses it when generating the response word by word.

Why it mattersThis approach could help AI systems run more efficiently by using different hardware resources for different parts of the task.

In the filing’s own words

Methods, systems, and computer storage media for providing compute management using a compute management engine in an artificial intelligence (AI) system. A compute management engine supports dynamically switching between two modes of operation for an inference phase of a generative artificial AI model. The compute management engine employs a bypass engine that causes prompt stage operations to be executed without an in-memory compute engine and causes auto-regression stage operations to be executed with the in-memory compute engine. In operation, an inference phase operation is accessed. When the inference phase operation is a prompt stage operation, the inference phase operation is executed without an in-memory compute engine. When the inference phase operation is an auto-regressive stage operation, the inference phase operation is executed with the in-memory compute engine. Memory output is generated for the inference phase operation to cause a processor to output a processor output for the inference phase operation.

What exactly they claim (claim 1)

1. A computerized system comprising: one or more computer processors; and computer memory storing computer-useable instructions that, when used by the one or more computer processors, cause the one or more computer processors to perform operations, the operations comprising: accessing a prompt stage operation associated with a generative artificial intelligence (AI) model, the generative AI model is associated with memory that supports executing operations associated with the generative AI model; based on the prompt stage operation, causing the prompt stage operation to bypass an in-memory compute engine of the memory; executing the prompt stage operation without using the in-memory compute engine; accessing an auto-regressive stage operation associated with the generative AI model; based on the auto-regressive stage operation, cause the auto-regressive stage operation to be executed using the in-memory compute engine of the memory; and executing the auto-regressive stage operation using the in-memory compute engine.

Facts

Company
Microsoft (Microsoft Technology Licensing, LLC)
Official title
COMPUTE MANAGEMENT ENGINE IN AN ARTIFICIAL INTELLIGENCE SYSTEM
Publication number
US20260359563A1
Application number
19736442
Filed
Jul 9, 2026
Published
Nov 12, 2026
Inventors
2
Claims
20

Classified under

G06F 40/20 Physics & computingG06F 17/16 Physics & computing

More from Microsoft in AI & software

Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.