Amazon patents method to save memory in AI chips by varying data precision
Amazon has filed a patent for a technique that reduces the amount of memory needed to run neural networks on specialized hardware. The system identifies unusual data points, called outliers, within a layer of the network and keeps them in high detail, while compressing the rest of the data to save space.
In the filing’s own words
Devices and techniques are generally described for intra-layer mixed precision quantization. In some examples, a first activation matrix comprising a plurality of rows may be computed. A set of outlier rows may be determined from the first activation matrix. A set of non-outliers may be determined from the first activation matrix. A first quantization algorithm may be selected for the set of outlier rows. A second quantization algorithm may be selected for the set of non-outlier rows. A first quantized activation matrix may be generated using the first quantization algorithm and the second quantization algorithm. A second activation matrix for a second layer of the neural network may be computed using the first quantized activation matrix and a first weight matrix.
What exactly they claim (claim 1)
1. A system comprising: a neural network accelerator apparatus comprising: one or more processors; a compute engine; and one or more computer-readable media comprising static random access memory (SRAM) and dynamic random access memory (DRAM), the one or more computer-readable media storing processor-executable instructions which, when executed using the one or more processors, perform operations comprising: computing, for a first layer of a neural network, a first activation matrix, wherein the first activation matrix comprises a first plurality of rows, wherein a first row of the first plurality of rows comprises a plurality of activation values; determining, based on the plurality of activation values, a set of outlier rows; determining, based on the plurality of activation values, a set of non-outlier rows; selecting, for the set of outlier rows, a first quantization algorithm, wherein the first quantization algorithm generates output having a first bitwidth; selecting, for the set of non-outlier rows, a second quantization algorithm, wherein the second quantization algorithm generates output having a second bitwidth, wherein the second bitwidth is less than the first bitwidth; generating, a first quantized activation matrix using the first quantization algorithm to quantize activation values of the set of outlier rows and using the second quantization algorithm to quantize activation values of the set of non-outlier rows; and computing, by the compute engine, a second activation matrix for a second layer of the neural network using the first quantized activation matrix and a first weight matrix.
Facts
- Company
- Amazon (Amazon Technologies, Inc.)
- Official title
- INTRA-LAYER MIXED PRECISION QUANTIZATION FOR NEURAL NETWORKS
- Publication number
- US20260300701A1
- Application number
- 19096233
- Filed
- Mar 31, 2025
- Published
- Oct 1, 2026
- Inventors
- 3
- Claims
- 22
Classified under
G06N 3/0495 Physics & computingG06F 17/16 Physics & computingG06N 3/048 Physics & computingMore from Amazon in AI & software
- Amazon patents system that cites sources for AI answers Sep 29, 2026
- Amazon patents tech to swap words in movies using actor's voice Sep 8, 2026
- Amazon patents ML tool to add graphics to live sports from moving cameras Sep 15, 2026
- Amazon patents shopping assistant that remembers your search filters Oct 6, 2026
- Amazon patents system where AI agents split tasks and talk to each other Oct 20, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.