Nvidia patents image-to-text system using multiple specialized AI models
Nvidia describes a vision-language model that combines several smaller, specialized neural networks, or experts, instead of one massive network. Each expert focuses on a specific task, and the system uses a process called cross-attention to turn their variable-length outputs into a fixed format that a language decoder can use to describe an image.
In the filing’s own words
A vision-language model learns skills and domain knowledge via distinct and separate task-specific neural networks, referred to as experts. Each expert is independently optimized for a specific task, facilitating the use of domain-specific data and architectures that are not feasible with a single large neural network trained for multiple tasks. The vision-language model implemented as an ensemble of pre-trained experts and is more efficiently trained compared with the single large neural network. During training, the vision-language model integrates specialized skills and domain knowledge, rather than trying to simultaneously learn multiple tasks, resulting in effective multi-modal learning.
What exactly they claim (claim 1)
1. A computer-implemented method, comprising: receiving multi-modal labels corresponding to an image, wherein the multi-modal labels are variable-length and are generated by two or more pre-trained domain-specific neural network models; mapping the multi-modal labels to a fixed length sequence of multi-modal features using cross-attention between a pre-defined number of learned latent queries and embeddings of the multi-modal labels; processing the image and the fixed length sequence of multi-modal features by a vision encoder and language decoder to predict text corresponding to the image; and providing the predicted text to a device for presentation to a user.
Facts
- Company
- Nvidia (NVIDIA Corporation)
- Official title
- Vision-language model with an ensemble of experts
- Patent number
- US12743879B2
- Application number
- 18544840
- Filed
- Dec 19, 2023
- Granted
- Sep 22, 2026
- Inventors
- 5
- Claims
- 21
Classified under
G06V 10/82 Physics & computingG06V 10/811 Physics & computingMore from Nvidia in AI & software
- Nvidia patents AI coach that analyzes your gameplay to suggest strategies Oct 13, 2026
- Nvidia patents AI system that writes and voices character conversations Sep 10, 2026
- Nvidia patents system that uses AI to find bugs in game videos Sep 17, 2026
- Nvidia patents AI method to make digital characters move realistically Sep 8, 2026
- Nvidia patents AI system that builds image generators from text prompts Sep 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.