Nvidia patents method to build 3D scene maps from camera and LiDAR data
Nvidia describes a system that combines camera images and 3D sensor data to create a detailed 3D map of a scene. It uses software to identify objects in 2D images, projects them into 3D space, and refines these shapes using sensor data to train a computer model that can fill in missing parts of the scene.
In the filing’s own words
In an example pseudo-labeling pipeline, LiDAR data (or other 3D measurements) and image data may be collected and used to construct a 3D occupancy grid representing an observed scene comprising accumulated 3D masks and corresponding semantic features (e.g., vision-language features) for each object instance. The 3D occupancy grid may be incrementally updated using accumulated observations over time. For example, image and video segmentation foundation models may be used to generate a sequence of two-dimensional (2D) masks for each object, and the 2D masks may be lifted to 3D and temporally accumulated. Semantic features may be aggregated for each masklet in a joint embedding space, effectively combining shape completions and semantic information. As such, the resulting pseudo-labels may be used to train a neural network to perform scene-scale object-level completion.
What exactly they claim (claim 1)
1. A system comprising one or more processors to: generate one or more object tracks based at least on processing image data representing an environment using one or more segmentation models; generate one or more three-dimensional (3D) object masks based at least on aggregating a projected representation of the one or more object tracks in a 3D grid; refine the one or more 3D object masks based at least on binary occupancy data derived from 3D sensor data corresponding to the image data; and execute one or more iterations of training a neural network based at least on training data associating the 3D sensor data with the one or more refined 3D object masks representing one or more object instances.
Facts
- Company
- Nvidia (NVIDIA CORPORATION)
- Official title
- SCENE RECONSTRUCTION FOR PANOPTIC SCENE COMPLETION
- Publication number
- US20260320803A1
- Application number
- 19643463
- Filed
- Apr 9, 2026
- Published
- Oct 15, 2026
- Inventors
- 7
- Claims
- 20
Classified under
G06T 7/12 Physics & computingG06N 3/08 Physics & computingG06T 7/20 Physics & computingG06V 10/82 Physics & computingG06V 20/70 Physics & computingMore from Nvidia in AI & software
- Nvidia patents AI coach that analyzes your gameplay to suggest strategies Oct 13, 2026
- Nvidia patents AI system that writes and voices character conversations Sep 10, 2026
- Nvidia patents system that uses AI to find bugs in game videos Sep 17, 2026
- Nvidia patents AI method to make digital characters move realistically Sep 8, 2026
- Nvidia patents AI system that builds image generators from text prompts Sep 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.