Apple patents method to build 3D maps from camera frames
Apple describes a system that turns video or camera data into a 3D model of a room or space. It selects specific 'key frames' to build the map and blends in other frames to keep details sharp. The system can also adjust the 3D model if the device moves or if the data needs correcting.
In the filing’s own words
Various implementations disclosed herein include devices, systems, and methods that use frames of sensor data to create key frames that are used to provide a 3D representation of a physical environment (e.g., a 3D mesh, planar representations, object bounding boxes, a combo of these, etc.). Some implementations use key frames (e.g., bundles of 2D image data deemed to be representative or useful for reconstruction/3D representation generation). Such key frames may be created by evaluating multiple frames of sensor data and determining whether to convert a given frame into a key frame or fuse it, e.g., fuse it with a nearest existing key frame. Such fusing may reduce loss of information which would occur if frames not selected as key frames are simply discarded.
What exactly they claim (claim 1)
1. A method comprising: at a device having a processor: obtaining a plurality of frames of sensor data of a physical environment, the frames comprising sensor data and frame-to-frame motion data corresponding to device poses associated with the frames; generating a 3D representation of the physical environment based on the plurality of frames; receiving correction data corresponding to a correction associated with the device poses; determining a global transformation and a change to a non-rigid component based on the correction data; and updating the 3D representation by: applying a transform based on the global transformation to update the 3D representation; and regenerating a subset of the 3D representation based on the change to the non-rigid component.
Facts
- Company
- Apple (Apple Inc.)
- Official title
- GENERATING 3D REPRESENTATIONS OF PHYSICAL ENVIRONMENTS USING KEY FRAMES
- Publication number
- US20260311432A1
- Application number
- 19709435
- Filed
- Jun 16, 2026
- Published
- Oct 8, 2026
- Inventors
- 5
- Claims
- 19
Classified under
G06T 17/205 Physics & computingG06T 5/50 Physics & computingG06T 7/12 Physics & computingG06T 7/579 Physics & computingG06T 7/70 Physics & computingG06V 10/25 Physics & computingMore from Apple in AR, VR & displays
- Apple patents AR glasses that zoom in on what you look at Sep 8, 2026
- Apple patents finger ring that controls VR with taps and waves Sep 15, 2026
- Apple patents head-mounted device that types via eye swipes and hand pinches Oct 6, 2026
- Apple patents eye-tracking and hand gestures to select screen items Oct 20, 2026
- Apple patents AR glasses that adjust lenses to fit your eyes Sep 17, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.