Nvidia patents method to speed up AI text generation with parallel processing
Nvidia describes a technique where one group of neural networks starts generating a sequence of words while a second group prepares the next word in parallel. This overlapping process, called prefill, aims to reduce the waiting time between generating each part of the output.
In the filing’s own words
Apparatuses, systems, and techniques to cause one or more neural networks to perform one or more prefill operations on a token of a set of tokens before the set of tokens is completely generated. In at least one embodiment, a processor comprises circuitry to cause one or more first neural networks to begin generating a set of tokens and cause one or more second neural networks to perform one or more prefill operations on a token of the set of tokens before the set of tokens is completely generated.
What exactly they claim (claim 1)
1. One or more processors comprising processing circuitry to: cause one or more first neural networks to begin generating a set of tokens; and cause one or more second neural networks to perform one or more prefill operations on a first token of the set of tokens in parallel with the generation of a subsequent token of the set of tokens.
Facts
- Company
- Nvidia (NVIDIA Corporation)
- Official title
- GENERATING TOKENS USING ONE OR MORE NEURAL NETWORKS
- Publication number
- US20260278386A1
- Application number
- 19180502
- Filed
- Apr 16, 2025
- Published
- Sep 17, 2026
- Inventors
- 3
- Claims
- 20
Classified under
G06N 3/0895 Physics & computingMore from Nvidia in AI & software
- Nvidia patents AI coach that analyzes your gameplay to suggest strategies Oct 13, 2026
- Nvidia patents AI system that writes and voices character conversations Sep 10, 2026
- Nvidia patents system that uses AI to find bugs in game videos Sep 17, 2026
- Nvidia patents AI method to make digital characters move realistically Sep 8, 2026
- Nvidia patents AI system that builds image generators from text prompts Sep 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.