Nvidia patents chip instructions to speed up AI math by mixing precision types
Nvidia describes a hardware feature for neural network processors that handles math in two different levels of detail at once. This allows the chip to calculate complex probability steps, like SoftMax, without wasting time switching between data formats. The filing says this approach can cut the number of steps needed for these calculations in half.
In the filing’s own words
Hardware-accelerated fused mixed precision floating point instructions include built-in precision conversion capabilities to eliminate overhead associated with switching between precisions. Such mixed precision operations that can be used to accelerate probability distribution function operations like SoftMax and Flash Attention, and/or for accelerated conversion to MX microscaling formats in other contexts. Using the new mixed precision instructions reduces the instruction count in half in example SoftMax/Flash Attention implementations.
What exactly they claim (claim 1)
1. A neural network processor comprising: a hardware calculator that receives a vector of logits; exponentiates each element of the vector; sums exponentiated values to provide a sum; and divides each exponentiated element by the sum to produce a probability distribution of results each of which are between 0 and 1 and all of which together sum to 1, wherein the hardware calculator includes a hardware instruction decoder configured to decode a single instruction call controlling hardware to (a) compute a value in first precision floating point and (b) convert the computed first precision floating point value to a second precision floating point value having a second floating point precision different than the first precision floating point.
Facts
- Company
- Nvidia (NVIDIA CORPORATION)
- Official title
- Accelerating SoftMax with Mixed Precision Operations
- Publication number
- US20260278351A1
- Application number
- 19557409
- Filed
- Mar 5, 2026
- Published
- Sep 17, 2026
- Inventors
- 10
- Claims
- 28
Classified under
G06N 3/048 Physics & computingG06F 7/485 Physics & computingG06F 7/4873 Physics & computingMore from Nvidia in Chips & hardware
- Nvidia patents method to skip zero data in neural network math Oct 13, 2026
- Nvidia patents hardware to speed up motion blur in ray tracing Oct 13, 2026
- Nvidia patents method to sync data across split processor chips Sep 17, 2026
- Nvidia patents chip math trick to speed up AI calculations Oct 8, 2026
- Nvidia patents data center cooling that uses cryogenic fluids Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.