Nvidia patents chip design that speeds up AI calculations by keeping data local
Nvidia describes a computer chip layout designed to run AI models faster by minimizing how much data has to move around. The design uses small, self-contained processing blocks that store data right next to the math units, allowing them to work in parallel without waiting for slow memory access.
In the filing’s own words
A low-latency processing system reduces data movement, improves memory bandwidth utilization, and enables, e.g., scalable, low-latency neural network inference. Compute tiles are configured to efficiently execute activation functions in parallel. Each compute tile includes an array of basic units equipped with local SRAM, vector registers, and multiply-accumulate (MAC) units, enabling localized computation to minimize data movement and reduce latency. IMLP tiles efficiently perform data format conversions and reduction operations as part of intermediate layer processing. A configurable network transmits data directly from source to destination register(s) within compute and IMLP tiles. Connections are, e.g., pre-configured for one or more cycles of neural network inference.
What exactly they claim (claim 1)
1. A method of performing tensor operations for low latency inference (LLI) operations using a set of one or more gather groups, comprising: distributing a number of inputs to the set of gather groups to compute partial results, at least one gather group comprising a plurality of compute tiles for performing the tensor operations, at least one compute tile comprising: shared control logic that configures the set of gather groups to process inputs in parallel; reduction and element-wise computation logic; and an array of basic units, each basic unit comprising: RAM; and arithmetic logic directly coupled to the RAM; configuring one or more basic units to compute, in parallel, per-basic unit results; combining the per-basic unit results by the reduction computation logic to produce a per-group result; and transmitting the per-group results to a configurable network coupled to at least one compute tile to perform a next processing operation when the per-group result completes a tensor operation or to combine the per-group results with other per-group results to complete the tensor operation.
Facts
- Company
- Nvidia (NVIDIA Corporation)
- Official title
- LOW-LATENCY COMPUTE ARCHITECTURES AND USE FOR NEURAL NETWORK INFERENCE
- Publication number
- US20260330206A1
- Application number
- 19371580
- Filed
- Oct 28, 2025
- Published
- Oct 22, 2026
- Inventors
- 4
- Claims
- 20
Classified under
G06N 3/063 Physics & computingMore from Nvidia in Chips & hardware
- Nvidia patents method to skip zero data in neural network math Oct 13, 2026
- Nvidia patents hardware to speed up motion blur in ray tracing Oct 13, 2026
- Nvidia patents method to sync data across split processor chips Sep 17, 2026
- Nvidia patents chip math trick to speed up AI calculations Oct 8, 2026
- Nvidia patents data center cooling that uses cryogenic fluids Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.