Nvidia patents method to swap objects in images using depth and text cues
Nvidia describes a system that uses neural networks to replace objects in computer-generated images. It uses depth maps and text instructions to guide the creation of new object details, which are then blended into the original scene.
In the filing’s own words
Synthetic data generation systems and methods are disclosed for augmenting synthetic scenes using neural networks that are conditioned on depth information. The synthetic data generation system may use a guided latent diffusion model to generate (or augment) synthetic images, which can subsequently be used to train other models to perform tasks such as object detection. Input of the model may be an image rendered by a graphic engine with coarsely rendered objects. When the image is rendered, segmentation masks may also be generated for objects in the image. The synthetic data generation system may generate a monocular depth image. During the image generation phase, the corresponding segmentation mask, depth map, and any guiding textual input serve as constraints for the denoising process. During the denoising step, the synthetic data generation system may also crop and adjust the resolution of individual objects during diffusion to enhance the results. The newly regenerated object is then blended back into the original image to produce a synthetic image.
What exactly they claim (claim 1)
1. A computer-implemented method, comprising: identifying a region of interest (ROI) associated with a first object in an image; generating, using a generative model and based on pixel data from the ROI and one or more of a segmentation mask for the first object, a depth map for the image, or an input specifying at least one aspect of a second object, image data for the second object independently of pixel data outside the ROI; and inserting the image data for the second object into the ROI of the image to cause the first object to be replaced with the second object in the image.
Facts
- Company
- Nvidia (Nvidia Corporation)
- Official title
- Data augmentation using conditioned generative models for synthetic content generation
- Patent number
- US12755516B2
- Application number
- 18349775
- Filed
- Jul 10, 2023
- Granted
- Oct 6, 2026
- Inventors
- 2
- Claims
- 20
Classified under
G06V 10/774 Physics & computingG06T 5/50 Physics & computingG06T 7/194 Physics & computingG06V 10/25 Physics & computingG06V 10/82 Physics & computingG06T 2207/20221 Physics & computingMore from Nvidia in AI & software
- Nvidia patents AI coach that analyzes your gameplay to suggest strategies Oct 13, 2026
- Nvidia patents AI system that writes and voices character conversations Sep 10, 2026
- Nvidia patents system that uses AI to find bugs in game videos Sep 17, 2026
- Nvidia patents AI method to make digital characters move realistically Sep 8, 2026
- Nvidia patents AI system that builds image generators from text prompts Sep 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.