Microsoft patents system to share GPU resources between different AI models
Microsoft describes a platform that manages shared graphics processing units (GPUs) for running various large language models. It uses a standard measurement method to compare how different AI models use resources, allowing the system to automatically shift computing power to where it is needed most.
In the filing’s own words
A model-as-a-service (MaaS) platform performs cross-model resources allocation from a shared pool of GPU resources based on model-agnostic metrics generated by a metric standardizer. The metric standardizer receives, from model providers, model-specific benchmark metrics that define relationships between resource utilization and token processing according to the different model-specific tokenization schemes; receives, from one or more MaaS components, token-based job metrics pertaining to LLM processing tasks; and determines, based on the model-specific benchmark metrics and token-based job metrics, the model-agnostic metrics for multiple model pools executing instances of different large language models (LLMs) that generate and process text according to different model-specific tokenization schemes. The MaaS platform further includes one or more resource allocation components that dynamically reallocates resources of the shared pool based on the model-agnostic metric.
What exactly they claim (claim 1)
1. A model-as-a-service platform including: a metric standardizer that: determines model-specific benchmark metrics that define relationships between GPU utilization and token processing according to different tokenization schemes; receives token-based job metrics identifying quantities of tokens processed by different LLMs according to the different tokenization schemes; and utilizes the model-specific benchmark metrics and the token-based job metrics to generate model pool utilization metrics that each quantifies resource utilization for a corresponding one of two or more model pools supporting instances of different LLMs, the model pool utilization metrics being expressed in units of a model-agnostic unit type; and an autoscaler that dynamically reallocates GPU resources among the model pools based on the model pool utilization metrics.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- Utilization-based resource allocation in model-as-a-service platform
- Patent number
- US12801508B2
- Application number
- 18652445
- Filed
- May 1, 2024
- Granted
- Nov 17, 2026
- Inventors
- 7
- Claims
- 20
Classified under
G06F 11/3428 Physics & computingG06F 11/3414 Physics & computingG06F 11/3447 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.
