Nvidia patents chip design to speed up AI text generation
Nvidia describes a computer chip layout designed to run large language models faster by keeping data close to the processing units. The design uses specific blocks called compute tiles to handle math operations locally, reducing the time it takes to move information around the chip.
In the filing’s own words
A low-latency processing system reduces data movement, improves memory bandwidth utilization, and enables, e.g., scalable, low-latency neural network inference. Compute tiles are configured to efficiently execute matrix operations in parallel. Each compute tile includes an array of basic units equipped with local SRAM, vector registers, and multiply-accumulate (MAC) units, enabling localized computation to minimize data movement and reduce latency. IMLP tiles efficiently perform data format conversions and reduction operations as part of intermediate layer processing. A configurable network transmits data directly from source to destination register(s) within compute and IMLP tiles. Connections are, e.g., pre-configured for one or more cycles of neural network inference.
What exactly they claim (claim 1)
1. A method for performing large language model (LLM) decode operations via compute tiles organized into tile groups, respective tile groups comprising a respective group of compute tiles and a respective inter-matrix layer processing (IMLP) tile, the method comprising: determining a first spatial distribution of computations corresponding to a first batch matrix multiplication (BMM) operation to tile groups; determining a second spatial distribution of computations corresponding to a second BMM operation to tile groups; determining a mapping of IMLP tiles to compute tiles based on the first spatial distribution and the second spatial distribution; performing the first BMM operation to calculate dot-product attention scores for one or more attention heads of the LLM, and distributing, to compute tiles from IMLP tiles based on the mapping, subvectors that form portions of normalized attention scores; and performing the second BMM operation to calculate a weighted sum of values based on the normalized attention scores.
Facts
- Company
- Nvidia (NVIDIA Corporation)
- Official title
- LOW-LATENCY COMPUTE ARCHITECTURES AND USE FOR NEURAL NETWORK INFERENCE
- Publication number
- US20260330199A1
- Application number
- 19372221
- Filed
- Oct 28, 2025
- Published
- Oct 22, 2026
- Inventors
- 4
- Claims
- 23
Classified under
G06N 3/048 Physics & computingG06F 17/16 Physics & computingMore from Nvidia in Chips & hardware
- Nvidia patents method to skip zero data in neural network math Oct 13, 2026
- Nvidia patents hardware to speed up motion blur in ray tracing Oct 13, 2026
- Nvidia patents method to sync data across split processor chips Sep 17, 2026
- Nvidia patents chip math trick to speed up AI calculations Oct 8, 2026
- Nvidia patents data center cooling that uses cryogenic fluids Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.