MicrosoftGranted patentAI & softwareInterest 6/10

Microsoft patents system to share GPU resources between different AI models

Patent drawing
Patent drawing from the official filing.

Microsoft describes a platform that manages shared graphics processing units (GPUs) for running various large language models. It uses a standard measurement method to compare how different AI models use resources, allowing the system to automatically shift computing power to where it is needed most.

Why it mattersThis could help companies run multiple AI services more efficiently by preventing some servers from sitting idle while others are overloaded.

In the filing’s own words

A model-as-a-service (MaaS) platform performs cross-model resources allocation from a shared pool of GPU resources based on model-agnostic metrics generated by a metric standardizer. The metric standardizer receives, from model providers, model-specific benchmark metrics that define relationships between resource utilization and token processing according to the different model-specific tokenization schemes; receives, from one or more MaaS components, token-based job metrics pertaining to LLM processing tasks; and determines, based on the model-specific benchmark metrics and token-based job metrics, the model-agnostic metrics for multiple model pools executing instances of different large language models (LLMs) that generate and process text according to different model-specific tokenization schemes. The MaaS platform further includes one or more resource allocation components that dynamically reallocates resources of the shared pool based on the model-agnostic metric.

What exactly they claim (claim 1)

1. A model-as-a-service platform including: a metric standardizer that: determines model-specific benchmark metrics that define relationships between GPU utilization and token processing according to different tokenization schemes; receives token-based job metrics identifying quantities of tokens processed by different LLMs according to the different tokenization schemes; and utilizes the model-specific benchmark metrics and the token-based job metrics to generate model pool utilization metrics that each quantifies resource utilization for a corresponding one of two or more model pools supporting instances of different LLMs, the model pool utilization metrics being expressed in units of a model-agnostic unit type; and an autoscaler that dynamically reallocates GPU resources among the model pools based on the model pool utilization metrics.

Facts

Company
Microsoft (Microsoft Technology Licensing, LLC)
Official title
Utilization-based resource allocation in model-as-a-service platform
Patent number
US12801508B2
Application number
18652445
Filed
May 1, 2024
Granted
Nov 17, 2026
Inventors
7
Claims
20

Classified under

G06F 11/3428 Physics & computingG06F 11/3414 Physics & computingG06F 11/3447 Physics & computing

More from Microsoft in AI & software

Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.