Microsoft patents method to shrink AI models for faster hardware
Microsoft describes a system that tests different ways to reduce the data size of parts of a language model. It measures which parts are most important to the model's accuracy and assigns smaller, faster data types to the less important parts. This allows the AI to run more efficiently on specialized hardware.
In the filing’s own words
A method for recommending a mixed-precision data type configuration to accelerate execution of a language model includes identifying a plurality of mixed-precision test configurations that assign mixed-precision data types among components of the language model; measuring saliency metrics for the language model while the language model is executed in a test environment configured in accord with multiple of the plurality of mixed-precision test configurations; generating a saliency ranking that ranks the components based on the saliency metrics; and generating the recommended mixed-precision data type configuration. The recommended mixed-precision data type configuration assigns a first, reduced-precision data type to a least salient subset of the components identified in the saliency ranking while assigning a second, higher-precision data type to a most salient subset of the components identified in the saliency ranking.
What exactly they claim (claim 1)
1. A system for configuring a hardware accelerator to execute a language model, the system comprising: a test configuration identifier stored in memory and executable by a processing system to identify a plurality of mixed-precision test configurations that assign mixed-precision data types among components of the language model, each configuration of the plurality of mixed-precision test configurations assigning a reduced-precision data type to a select one of the components while assigning a comparatively increased-precision data type to another one of the components; a salience analyzer stored in memory and executable by the processing system to: measure saliency metrics for the language model while the language model is executed according to different configurations selected from the plurality of mixed-precision test configurations; and based on the saliency metrics, generate a saliency ranking that ranks at least a subset of the components according to relative salience; a quantized model recommender stored in memory and executable by the processing system to: generate a recommended mixed-precision data type configuration for an artificial intelligence (AI) hardware accelerator to implement while executing the language model, the recommended mixed-precision data type configuration assigning the reduced-precision data type to a least salient subset of the components identified in the saliency ranking while assigning the comparatively increased-precision data type to a most salient subset of the components identified in the saliency ranking.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- FINE GRAIN POST-TRAINING QUANTIZATION FOR MIXED-PRECISION ARTIFICIAL INTELLIGENCE HARDWARE ACCELERATION
- Publication number
- US20260319938A1
- Application number
- 19176562
- Filed
- Apr 11, 2025
- Published
- Oct 15, 2026
- Inventors
- 4
- Claims
- 20
Classified under
G06F 11/3698 Physics & computingG06F 11/3688 Physics & computingG06F 17/16 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.