Microsoft patents method to keep AI models loaded in memory
This patent describes a way to run artificial intelligence models on a network of computers by keeping the model's data permanently stored in fast on-chip memory. Instead of reloading the data for every request, the system locks the information in place until the service is stopped or the model changes.
In the filing’s own words
Systems and methods for neural network processing are provided. A method in a system comprising a plurality of nodes interconnected via a network, where each node includes a plurality of on-chip memory blocks and a plurality of compute units, is provided. The method includes upon service activation receiving an N by M matrix of coefficients corresponding to the neural network model. The method includes loading the coefficients corresponding to the neural network model into the plurality of the on-chip memory blocks for processing by the plurality of compute units. The method includes regardless of a utilization of the plurality of the on-chip memory blocks as part of an evaluation of the neural network model, maintaining the coefficients corresponding to the neural network model in the plurality of the on-chip memory blocks until the service is interrupted or the neural network model is modified or replaced.
What exactly they claim (claim 1)
1. A method for evaluating a neural network model corresponding to a service in a distributed hardware system comprising a plurality of nodes interconnected via a datacenter network, wherein each node comprises a plurality of on-chip memory blocks and a plurality of compute units, the method comprising: upon service activation, partitioning the neural network model into separate lavers, wherein each layer comprises an N by M matrix of coefficients corresponding to the neural network model, and wherein N is an integer equal to or greater than 4 and M is an integer equal to or greater than 4; loading the N by M matrix of coefficients corresponding to the neural network model into the plurality of the on-chip memory blocks of the plurality of nodes of the distributed hardware system for processing by the plurality of compute units, wherein each of the plurality of nodes comprises a field programable gate array (FPGA) within the datacenter network, and wherein each of the plurality of the on-chip memory blocks comprises a static random access memory block, and wherein each of the plurality of compute units comprises a set of pre-configured resources on the FPGA; allowing the plurality of nodes interconnected via the datacenter network to communicate directly with each other using a transport protocol to process even a single request for evaluating the neural network corresponding to the service; and regardless of a utilization of the plurality of the on-chip memory blocks as part of an evaluation of the neural network model, keeping the N by M matrix of coefficients corresponding to the neural network model pinned to the plurality of the on-chip memory blocks until the service is interrupted or the neural network model is modified or replaced.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- Neural network processing with model pinning
- Patent number
- US12737598B2
- Application number
- 17483313
- Filed
- Sep 23, 2021
- Granted
- Sep 15, 2026
- Inventors
- 4
- Claims
- 20
Classified under
G06N 3/063 Physics & computingG06F 9/30098 Physics & computingG06F 9/3836 Physics & computingG06F 9/3867 Physics & computingG06N 3/04 Physics & computingG06F 17/16 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.