/Patent Tracker
MicrosoftPatent applicationAI & softwareInterest 7/10

Microsoft patents method to run big AI models on small devices

Microsoft describes a system where a server holds a full AI model and sends only small pieces to a device with limited memory. The device runs one piece at a time while the server prepares the next, allowing large models to work on hardware that cannot store the whole program.

Why it mattersThis approach could help run advanced AI features on phones or laptops without needing expensive, high-capacity chips.

In the filing’s own words

Methods, systems, apparatuses, and computer program products are described herein that enable execution of a large AI model on a memory-constrained target device that is communicatively connected to a parameter server, which stores a master copy of the AI model. The AI model may be dissected into smaller portions (e.g., layers or sub-layers), and each portion may be executed as efficiently as possible on the target device. After execution of one portion of the AI model is finished, another portion of the AI model may be downloaded and executed at the target device. To improve efficiency, the input samples may be divided into microbatches, and a plurality of microbatches executing in sequential order may form a minibatch. The size of the group of microbatches or minibatch can be manually or automatically adjusted to reduce the communication overhead.

What exactly they claim (claim 1)

1. A method performed by a system that comprises a parameter server, the parameter server being communicatively connected to a group of target devices and including a data manager, a batch manager, a transmitter, and a weight updater, the group of target devices comprising one or more target devices, the group of target devices being configured to run an instance of an artificial intelligence (AI) model, at least one target device comprising an integrated circuit chip having an on-chip memory of a size less than an entirety of the AI model, the method comprising: storing by the data manager a master copy the AI model, determining by the batch manager a microbatch size suitable for each target device of the group of target devices, and transmitting by the transmitter a portion of the AI model to the group of target devices, a size of the portion being based at least on the on-chip memory size and a size of one or more layers of the AI model; and contemporaneously, with a set of microbatches of a training dataset being executed by the integrated circuit chip at the at least one target device on a first subportion of the transmitted portion of the AI model stored in the on-chip memory to generate activations, performing by the weight updater a reduction of parameters for a second subportion of the transmitted portion of the AI model, and sending by the transmitter weights for a third subportion of the transmitted portion of the AI model to the target device.

Facts

Company
Microsoft (Microsoft Technology Licensing, LLC)
Official title
EXECUTING LARGE ARTIFICIAL INTELLIGENCE MODELS ON MEMORY-CONSTRAINED DEVICES
Publication number
US20260288469A1
Application number
19692719
Filed
May 29, 2026
Published
Sep 24, 2026
Inventors
7
Claims
20

Classified under

G06F 9/3877 Physics & computingH04L 67/289 Electricity & electronicsH04L 67/34 Electricity & electronicsG06F 3/0673 Physics & computingG06F 9/5027 Physics & computingG06F 15/163 Physics & computing
Read the full filing (official PDF) ↗ Open in the Patent Tracker

More from Microsoft in AI & software

Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.