Nvidia patents method to turn photos into editable 3D scenes
Nvidia describes a process that takes photos or scans of a real-world space and turns them into a digital 3D model. The system identifies objects in the scene, swapping in high-quality digital copies for common items and creating new 3D shapes for anything it does not recognize.
In the filing’s own words
Approaches presented herein can provide for the automatic generation of a digital representation of an environment that may include multiple objects of various object types. An initial representation (e.g., a point cloud) of the environment can be generated from registered image or scan data, for example, and objects in the environment can be segmented and identified based at least on that initial representation. For objects that are recognized based on these segmentations, stored accurate representations can be substituted for those objects in the representation of the environment, and if no such model is available then a mesh or other representation of that object can be generated and positioned in the environment. A result can then include a 3D representation of a scene or environment in which objects are identified and segmented as individual objects, and representations of the scene or environment can be viewed, and interacted with, through various viewports, positions, and perspectives.
What exactly they claim (claim 1)
1. A computer-implemented method, comprising: generating, based at least on image data representative of an environment including one or more objects, a three-dimensional (3D) representation of the environment; analyzing the 3D representation to determine one or more segments corresponding to the one or more objects in the environment; classifying, based at least on data for the one or more segments and the image data, first individual ones of the one or more objects as identified objects and second individual ones of the one or more objects as unidentified objects, wherein the classifying of the second individual ones of the one or more objects as the unidentified objects comprises determining that at least some of the second individual ones of the one or more objects do not have a corresponding 3D model available within an existing repository of 3D models; generating, based on at least classifying the second individual ones of the one or more objects as unidentified objects, one or more mesh representations of the unidentified objects to be included in the 3D representation of the environment; replacing at least a portion of the one or more segments corresponding to the identified objects with a stored portion of a 3D model selected from the existing repository of 3D models, wherein the replacing changes a number of points in a mesh representation of the environment; and providing the 3D representation of the environment including the stored portion of the 3D model and the one or more mesh representations.
Facts
- Company
- Nvidia (Nvidia Corporation)
- Official title
- Replicating physical environments and generating 3D assets for synthetic scene generation
- Patent number
- US12802356B2
- Application number
- 18066135
- Filed
- Dec 14, 2022
- Granted
- Nov 17, 2026
- Inventors
- 9
- Claims
- 19
Classified under
G06T 17/20 Physics & computingG06T 7/33 Physics & computingG06V 10/82 Physics & computingG06V 2201/07 Physics & computingMore from Nvidia in AI & software
- Nvidia patents AI coach that analyzes your gameplay to suggest strategies Oct 13, 2026
- Nvidia patents AI system that writes and voices character conversations Sep 10, 2026
- Nvidia patents system that uses AI to find bugs in game videos Sep 17, 2026
- Nvidia patents chip that turns text into video using noise reduction Nov 5, 2026
- Nvidia patents system that translates specific speakers in group conversations Nov 12, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.
