Nvidia patents system that auto-fixes AI training crashes
Nvidia describes a method for monitoring large computer clusters used to train AI models. When a problem occurs, the system checks both the software code and the physical hardware to identify the cause. It then automatically applies a fix, such as restarting the job with updated settings, to keep the work running.
In the filing’s own words
Systems and methods for autonomous failure response to enhance compute cluster resilience and efficiency. The systems and methods are suitable, e.g., for enhancing resilience and efficiency of computational workflows, e.g., deep learning/AI model training jobs, performed, e.g., via GPU-accelerated supercomputers. According to one or more embodiments, systems and methods obtain both application-level and hardware/infrastructure level health data associated with a workflow and implement, following the detection of a fault in the workflow, automated interventions, e.g. updating the workflow for a restart, for fault remediation.
What exactly they claim (claim 1)
1. A computer-implemented method for monitoring a computational workflow on a computing cluster, the method comprising: obtaining application-level health data pertaining to the computational workflow; obtaining infrastructure-level health data pertaining to the computational workflow; detecting, based on the application-level health data and/or the infrastructure-level health data, a fault; after detecting the fault, correlating the application-level health data and the infrastructure-level health data to provide correlated application-and infrastructure-level health data; classifying, based on the correlated application-and infrastructure-level health data, the fault as being one of: application-level, infrastructure-level, or ambiguous; determining, based on the classification of the fault, an intervention; and executing the intervention.
Facts
- Company
- Nvidia (NVIDIA Corporation)
- Official title
- AUTONOMOUS FAILURE RESPONSE FOR ENHANCED COMPUTE CLUSTER RESILIENCE AND EFFICIENCY
- Publication number
- US20260277731A1
- Application number
- 19080001
- Filed
- Mar 14, 2025
- Published
- Sep 17, 2026
- Inventors
- 7
- Claims
- 21
Classified under
G06F 11/0793 Physics & computingG06F 11/0721 Physics & computingG06F 11/079 Physics & computingG06T 1/20 Physics & computingMore from Nvidia in AI & software
- Nvidia patents AI coach that analyzes your gameplay to suggest strategies Oct 13, 2026
- Nvidia patents AI system that writes and voices character conversations Sep 10, 2026
- Nvidia patents system that uses AI to find bugs in game videos Sep 17, 2026
- Nvidia patents AI method to make digital characters move realistically Sep 8, 2026
- Nvidia patents AI system that builds image generators from text prompts Sep 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.