Microsoft patents method to shrink AI models for smaller devices
Microsoft describes a technique to make large AI models run faster on devices with limited power by treating different parts of the model differently. It uses a math method called Singular Value Decomposition to separate the most important data from the less important data, keeping the important parts precise while simplifying the rest.
In the filing’s own words
The present invention relates to a method for differential quantization of neural networks, such as large language models (LLMs), to improve operational efficiency without significant performance degradation. The method involves decomposing a neural network layer's matrix into multiple matrices using decomposition to identify significant components and insignificant components. The significant components undergo minimal or no quantization, while the less significant components undergo higher degrees of quantization. This differential quantization allows large neural networks to operate efficiently on edge devices with limited computing resources. The differential quantization technique enhances the deployment of neural networks in various hardware environments by reducing computational and memory bandwidth demands.
What exactly they claim (claim 1)
1. One or more computer storage media comprising computer-executable instructions that when executed by computing device performs a method of generating a differentially quantized neural network, the method comprising: decomposing a matrix from a first layer of a neural network into an orthogonal matrix of singular values; identifying a first plurality of significant columns and/or rows in the orthogonal matrix that make above a threshold contribution to producing an accurate final inference; forming a significant matrix using the first plurality of significant columns and/or rows; forming a less significant matrix using a second plurality of columns and/or rows from the orthogonal matrix that are not in the first plurality of significant columns and/or rows; quantizing the significant matrix at a first precision level to form a significant quantized matrix; quantizing the less significant matrix at a second precision level to form a less significant quantized matrix; and generating the differentially quantized neural network from the significant matrix and the less significant matrix.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- DIFFERENTIAL QUANTIZATION OF A NEURAL NETWORK BASED ON SVD INFORMATION RANKING
- Publication number
- US20260300700A1
- Application number
- 19095166
- Filed
- Mar 31, 2025
- Published
- Oct 1, 2026
- Inventors
- 3
- Claims
- 20
Classified under
G06N 3/0495 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.