Nvidia patents 3D scene mapping that lets robots understand objects by name
Nvidia describes a system that uses LiDAR data and a neural network to create a 3D map of the environment, identifying individual objects without pre-defined labels. It then uses text descriptions to match these objects to specific categories, allowing the machine to understand what it sees and decide how to move or interact with items.
In the filing’s own words
Panoptic scene completion may be achieved using a neural network comprising a sparse generative encoder-decoder network that performs scene-level occupancy prediction in a fixed-size voxel grid. To segment individual object instances, an instance decoder may segment individual object instances from the scene-level occupancy predictions to predict instance masks in the (occupied) voxel space in a class-agnostic fashion. For each predicted instance (e.g., transformer query), the instance decoder may regress a semantic embedding or token representing the semantic features (e.g., a CLIP token) for a corresponding object instance, enabling zero-shot prompting at test and/or inference time. As such, the neural network may produce semantic features that may be used for open-vocabulary querying directly from 3D measurements, and a representation of the resulting labeled masks may be provided to one or more downstream components such as a control policy of an ego-machine to make decisions on navigation, grasping, and/or manipulation actions.
What exactly they claim (claim 1)
1. A system comprising one or more processors to: generate, based at least on applying a representation of LiDAR data generated using one or more LiDAR sensors of an ego-machine to a neural network, one or more detected object instances comprising one or more predicted three-dimensional (3D) masks and one or more one or more semantic embeddings; generate one or more labeled object instances based at least on querying the one or more semantic embeddings of the one or more detected object instances using one or more textual descriptions of one or more target classes; and execute one or more operations of the ego-machine based at least on a representation of the one or more labeled object instances.
Facts
- Company
- Nvidia (NVIDIA CORPORATION)
- Official title
- PANOPTIC SCENE COMPLETION
- Publication number
- US20260321031A1
- Application number
- 19643489
- Filed
- Apr 9, 2026
- Published
- Oct 15, 2026
- Inventors
- 7
- Claims
- 20
Classified under
G06V 20/64 Physics & computingG01S 7/4915 Physics & computingG01S 17/89 Physics & computingG01S 17/931 Physics & computingG06V 10/762 Physics & computingG06V 10/764 Physics & computingMore from Nvidia in Robots & automation
- Nvidia patents robot control using language models to predict 3D positions Sep 10, 2026
- Nvidia patents method to teach humanoid robots to use hands via simulation Oct 15, 2026
- Nvidia patents robot system that learns tasks from visual data Sep 29, 2026
- Nvidia patents method for robots to estimate object positions from images Oct 13, 2026
- Nvidia patents method to create new robot training videos from existing footage Sep 10, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.