Amazon patents method to save memory in neural network models
Amazon has been granted a patent for a technique that optimizes how neural network models use computer memory. The system identifies specific calculations that are repeated and recalculates them on the fly rather than storing the results, which reduces the amount of memory needed to run the model.
In the filing’s own words
A technique for rematerializing a tensor in a neural network model may include identifying a candidate operator for rematerialization in a representation of a data flow graph for a neural network model. An operator can be identified as a candidate operator if it generates a tensor that is used multiple times in the neural network model and belongs to a set of permitted operators for rematerialization. An operator chain that starts at one or more source operators and ends at the candidate operator is determined. A profitability of rematerializing the operator chain is determined based on a comparison of a memory footprint of source data for the one or more source operators, and a memory footprint of the tensor. If it is determined that rematerializing the operator chain is profitable, the operator chain is inserted in the representation of the data flow graph where the tensor is reused.
What exactly they claim (claim 1)
1. A computer-implemented method, comprising: identifying a candidate operator for rematerialization in an intermediate representation of a data flow graph for a neural network model, the candidate operator generating a tensor that is used multiple times in the neural network model, and belonging to a set of permitted operators for rematerialization; determining an operator chain that starts at one or more source operators and ends at the candidate operator, wherein each operator in the operator chain belongs to the set of permitted operators; determining a profitability of rematerializing the operator chain based on a comparison of a memory footprint of source data for the one or more source operators, and a memory footprint of the tensor; in response to determining that rematerializing the operator chain is profitable, generating a modified representation of the data flow graph by inserting the operator chain in the intermediate representation of the data flow graph where the tensor is reused to rematerialize the operator chain; and compiling the modified representation of the data flow graph into machine code implementing the neural network model having a reduced runtime memory footprint as compared to the intermediate representation.
Facts
- Company
- Amazon (Amazon Technologies, Inc.)
- Official title
- Tensor rematerialization
- Patent number
- US12737601B1
- Application number
- 18067510
- Filed
- Dec 16, 2022
- Granted
- Sep 15, 2026
- Inventors
- 4
- Claims
- 23
Classified under
G06N 3/063 Physics & computingG06F 9/5016 Physics & computingG06N 3/082 Physics & computingMore from Amazon in AI & software
- Amazon patents system that cites sources for AI answers Sep 29, 2026
- Amazon patents tech to swap words in movies using actor's voice Sep 8, 2026
- Amazon patents ML tool to add graphics to live sports from moving cameras Sep 15, 2026
- Amazon patents shopping assistant that remembers your search filters Oct 6, 2026
- Amazon patents system where AI agents split tasks and talk to each other Oct 20, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.