Microsoft patents system to route AI requests to best available chips
Microsoft describes a method for managing AI requests by estimating how long a task will take based on the specific graphics processing units (GPUs) available. The system then routes the request to the AI model and hardware combination that can handle the job most efficiently, considering both the type of model and the expected output length.
In the filing’s own words
A generative artificial intelligence (AI) model application programming interface (API) receives a generative AI request and routes the generative AI request to a generative AI model. The generative AI model API identifies whether the generative AI request is an asynchronous or a synchronous request and identifies a likely length of generation requested by the generative AI request. The generative AI model API evaluates the available capacity of generative AI models in a shared pool of computing system resources based upon the generative AI model type requested by the generative AI request and based upon the length of the requested generation. The generative AI model API routes the generative AI request to a generative AI model based upon the available capacity, whether the generative AI request is synchronous or asynchronous, the generative AI model type, and the length of the requested generation.
What exactly they claim (claim 1)
1. A method, comprising: receiving, at an artificial intelligence (AI) communication interface, from a calling system, an AI request; determining an AI model type configured to execute the AI request; determining a subset of AI models from a plurality of different AI models that correspond to the AI model type; identifying dedicated graphics processing units (GPUs) associated with the subset of AI models, individual dedicated GPUs being dedicated to individual AI models prior to receiving the AI request; estimating a plurality of estimated lengths of generated outputs for executing the AI request by the dedicated GPUs, estimating the plurality of estimated lengths of the generated outputs being based at least in part on a hardware characteristic of the dedicated GPUs; routing the AI request to a selected AI model, of the subset of AI models, based on the plurality of estimated lengths of the generated outputs; receiving a response from the selected AI model; and returning the response to the calling system.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- GENERATIVE ARTIFICIAL INTELLIGENCE (AI) SYSTEM
- Publication number
- US20260310946A1
- Application number
- 19711554
- Filed
- Jun 17, 2026
- Published
- Oct 8, 2026
- Inventors
- 4
- Claims
- 20
Classified under
G06N 3/0475 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.