Nvidia patents data sorting method to speed up AI training on GPUs
Nvidia describes a system that organizes variable-length data into two layers of buckets before sending it to graphics processing units. This structure helps fit the data into the GPU's memory more efficiently by reducing wasted space, or padding, during the training of machine learning models.
In the filing’s own words
In various examples, systems and methods are disclosed for a two-dimensional (2D) dynamic bucket pre-processor that enhances the efficiency of machine learning model training on GPUs. The 2D dynamic bucket pre-processor includes a bucket stratification optimizer that configures a 2D bucket buffer for performing stratified 2D bucketing by organizing data into first-tier and second-tier sub-buckets, and a batch size optimizer to identify optimal batch sizes that fit into GPU memory. During training of a machine learning model, micro-batches comprise data served from selected sub-buckets to a model training platform. Organizing data into first and second tiers based on sequence length dimensions minimizes padding and ensures efficient GPU processing. In some embodiments, the 2D dynamic bucket pre-processor may perform 2D bucketing as a dynamic process that streams data into the 2D bucket buffer, as micro-batches of previously bucketed data are being served to execute training steps of a model training process.
What exactly they claim (claim 1)
1. One or more processors comprising processing circuitry to: instantiate a buffer structure comprising at least a first bucketing tier that includes a first plurality of stratified data buckets, where individual stratified data buckets of the first plurality of stratified data buckets comprise a second bucketing tier that includes a second plurality of stratified data buckets; obtain variable length sequenced data comprising a first sequence length dimension and a second sequence length dimension; store the variable length sequenced data to the buffer structure based at least on correlating the first sequence length dimension with the first bucketing tier and correlating the second sequence length dimension with the second bucketing tier; select a second-tier bucket from the second plurality of stratified data buckets; and serve a batch of data samples of the variable length sequenced data from the selected second-tier bucket to one or more graphics processing units (GPUs) via transmission over one or more communication links, the batch of data samples having a data sample batch size determined based at least on a memory utilization of the one or more GPUs and a two-dimensional bucket size of the second-tier bucket.
Facts
- Company
- Nvidia (NVIDIA Corporation)
- Official title
- BUFFER MANAGEMENT
- Publication number
- US20260278000A1
- Application number
- 19076691
- Filed
- Mar 11, 2025
- Published
- Sep 17, 2026
- Inventors
- 4
- Claims
- 20
Classified under
G06F 16/901 Physics & computingG06N 20/00 Physics & computingMore from Nvidia in AI & software
- Nvidia patents AI coach that analyzes your gameplay to suggest strategies Oct 13, 2026
- Nvidia patents AI system that writes and voices character conversations Sep 10, 2026
- Nvidia patents system that uses AI to find bugs in game videos Sep 17, 2026
- Nvidia patents AI method to make digital characters move realistically Sep 8, 2026
- Nvidia patents AI system that builds image generators from text prompts Sep 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.