Amazon patents method to save memory when running neural networks
This patent describes a way to make neural network models run more efficiently by managing how they use computer memory. It identifies large data blocks that are used far apart in the calculation process and temporarily moves them out of the main memory, freeing up space for other tasks.
In the filing’s own words
Techniques for reducing compilation time of a neural network model can include identifying boundary markers in a representation of the neural network model to determine proxy layers in the neural network model. A tensor that is used in a first proxy layer and is used next in a second proxy layer separated from the first proxy layer by at least a threshold distance can be identified, and a spilling operation can be inserted for the tensor to remove the tensor from a memory allocation process. A memory allocator can then be invoked to perform the memory allocation process without the tensor associated with the inserted spilling operation.
What exactly they claim (claim 1)
1. A computer-implemented method performed by a compiler, the method comprising: receiving a description of a neural network model; generating an intermediate representation of the neural network model representing a data flow graph, the data flow graph comprising a plurality of operators; identifying collective compute operations to identify proxy layers in the data flow graph; determining long-distance tensors that are used sequentially in proxy layers separated by a threshold number of at least two proxy layers; inserting spilling operations for the long-distance tensors in the intermediate representation of the neural network model to generate a modified intermediate representation; providing the modified intermediate representation to a memory allocator to assign memory locations for tensors in the modified intermediate representation that have not been spilled; and generating machine code for the neural network model based on the modified intermediate representation and the assigned memory locations.
Facts
- Company
- Amazon (Amazon Technologies, Inc.)
- Official title
- Layer-based spilling
- Patent number
- US12762384B1
- Application number
- 18194561
- Filed
- Mar 31, 2023
- Granted
- Oct 13, 2026
- Inventors
- 2
- Claims
- 20
Classified under
G06N 3/0499 Physics & computingG06F 8/433 Physics & computingG06F 8/443 Physics & computingMore from Amazon in AI & software
- Amazon patents system that cites sources for AI answers Sep 29, 2026
- Amazon patents tech to swap words in movies using actor's voice Sep 8, 2026
- Amazon patents ML tool to add graphics to live sports from moving cameras Sep 15, 2026
- Amazon patents shopping assistant that remembers your search filters Oct 6, 2026
- Amazon patents system where AI agents split tasks and talk to each other Oct 20, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.