Nvidia patents smarter routing for AI agent workloads
Nvidia describes a system that manages how AI agents handle tasks by looking ahead at upcoming work. It checks how busy different computing units are and how much memory they are using to decide which unit gets the next job.
In the filing’s own words
A system comprising one or more processors performs operations to optimize workload distribution among artificial intelligence (AI) agents. The system receives inference requests from AI agent workflows, which include metadata on future workload characteristics. The system accesses current load patterns and key-value cache utilization of multiple inference workers and routes the requests to selected subsets of workers based on these factors, ensuring efficient allocation of workloads and effective use of resources.
What exactly they claim (claim 1)
1. One or more processors comprising processing circuitry to perform operations comprising: receiving, by a processor of an artificial intelligence (AI) agent system, one or more inference requests from one or more AI agent workflows, the one or more inference requests comprising metadata indicating one or more future workload characteristics of the one or more inference requests; accessing, by the processor of the AI agent system, one or more current load patterns and key-value (KV) cache utilization of a plurality of inference workers of the AI agent system; and routing, by the processor of the AI agent system, the one or more inference requests to a selected subset of the plurality of inference workers based on the one or more current load patterns and the one or more future workload characteristics.
Facts
- Company
- Nvidia (NVIDIA Corporation)
- Official title
- WORKLOAD AWARE OPTIMIZATIONS FOR INFERENCE TIME SCALING IN AGENTS
- Publication number
- US20260278422A1
- Application number
- 19358991
- Filed
- Oct 15, 2025
- Published
- Sep 17, 2026
- Inventors
- 9
- Claims
- 20
Classified under
G06N 5/043 Physics & computingG06F 17/16 Physics & computingMore from Nvidia in AI & software
- Nvidia patents AI coach that analyzes your gameplay to suggest strategies Oct 13, 2026
- Nvidia patents AI system that writes and voices character conversations Sep 10, 2026
- Nvidia patents system that uses AI to find bugs in game videos Sep 17, 2026
- Nvidia patents AI method to make digital characters move realistically Sep 8, 2026
- Nvidia patents AI system that builds image generators from text prompts Sep 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.