OpenAI patents system to swap AI models in and out of memory
OpenAI has filed a patent for a method that manages how multiple customized AI models run on a server. The system prioritizes requests for models already in the computer's fast memory and swaps in new models only when needed, keeping unused models in slower storage to save space.
In the filing’s own words
The present technology pertains to a generative response system (“system”) that includes generative inference engines (“engines”) on each of which multiple fine-tuned models can be simultaneously loaded. A controller determines the order in which requests for respective fine-tuned models are queued/batched. Requests for currently loaded models (e.g., models currently on an engine) are processed first. Requests for a model that is not currently loaded are processed after swapping the model onto an engine, according to a caching hierarchy, to replace a long-dormant (or infrequently used) model. After being swapped off the inference engine, a system copy of the long-dormant model is maintained in system memory (e.g., CPU RAM) for easy retrieval in case the model is subsequently requested. After not being requested for an extended time, the system copy of the long-dormant model is removed and only a network copy is maintained in network storage.
What exactly they claim (claim 1)
1. A method, comprising: receiving, at a generative response system, a plurality of requests to generate responses based on respective fine-tuned models; organizing the plurality of requests into first requests and second requests based on which requests of the plurality of requests use currently-loaded fine-tuned models that are on one or more inference engines of the generative response system when the plurality of requests are received, wherein the first requests use the currently-loaded fine-tuned models and the second requests do no use the currently-loaded fine-tuned models; and processing the first requests; and processing the second requests, after processing the first requests.
Facts
- Company
- OpenAI (OpenAI OpCo, LLC)
- Official title
- PARALLEL PROCESSING FOR FINE-TUNED MODEL GENERATIVE RESPONSE MODELS AND THE MEMORY MANAGEMENT THEREOF
- Publication number
- US20260310938A1
- Application number
- 19170532
- Filed
- Apr 4, 2025
- Published
- Oct 8, 2026
- Inventors
- 4
- Claims
- 20
Classified under
G06N 3/0455 Physics & computingMore from OpenAI in AI & software
- OpenAI patents voice system that detects when you stop talking Sep 22, 2026
- OpenAI patents system to catch AI models cheating on tasks Sep 10, 2026
- OpenAI patents method to check AI thinking for cheating Sep 10, 2026
- OpenAI patents system that turns photos into new AI-generated images Sep 10, 2026
- OpenAI patents AI that answers questions using only your company's secure files Sep 17, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.