Apple patents method for tracking real-world objects in AR/VR scenes
Apple has filed a patent application describing a method for electronic devices to identify specific features in real-world images captured by a camera. The system analyzes pixel data to find distinguishable objects and fits geometric planes to them, allowing the device to track these features as the user moves. This tracking enables the device to maintain the position of augmented reality markers relative to the physical object.
In the filing’s own words
A method includes obtaining first pass-through image data characterized by a first pose. The method includes obtaining respective pixel characterization vectors for pixels in the first pass-through image data. The method includes identifying a feature of an object within the first pass-through image data in accordance with a determination that pixel characterization vectors for the feature satisfy a feature confidence threshold. The method includes displaying the first pass-through image data and an AR display marker that corresponds to the feature. The method includes obtaining second pass-through image data characterized by a second pose. The method includes transforming the AR display marker to a position associated with the second pose in order to track the feature. The method includes displaying the second pass-through image data and maintaining display of the AR display marker that corresponds to the feature of the object based on the transformation.
What exactly they claim (claim 1)
1. A method comprising: at an electronic device with one or more processors, a non-transitory memory, and a display: identifying, in pass-through image data characterized by a pose associated with a field of view of an image sensor, a first set of pixels associated with a distinguishable set of features; fitting a first plane to the first set of pixels according to a determination that the first set of pixels satisfy a planar criterion; obtaining pixel characterization vectors for pixels in the pass-through image data, wherein each of the pixel characterization vectors includes one or more labels; identifying a second set of pixels proximate to the first set of pixels, wherein pixel characterization vectors for the second set of pixels and pixel characterization vectors for the first set of pixels satisfy an object confidence threshold; and fitting a second plane to the first set of pixels and the second set of pixels, wherein the first plane is coplanar with the second plane.
Facts
- Company
- Apple (Apple Inc.)
- Official title
- Methods and Devices for Detecting and Identifying Features in an AR/VR Scene
- Publication number
- US20260289814A1
- Application number
- 19691170
- Filed
- May 28, 2026
- Published
- Sep 24, 2026
- Inventors
- 8
- Claims
- 20
Classified under
G06T 7/73 Physics & computingG02B 27/017 Physics & computingG06V 10/245 Physics & computingG06V 10/462 Physics & computingG06V 20/20 Physics & computingG06T 2207/30204 Physics & computingMore from Apple in AR, VR & displays
- Apple patents AR glasses that zoom in on what you look at Sep 8, 2026
- Apple patents finger ring that controls VR with taps and waves Sep 15, 2026
- Apple patents head-mounted device that types via eye swipes and hand pinches Oct 6, 2026
- Apple patents eye-tracking and hand gestures to select screen items Oct 20, 2026
- Apple patents AR glasses that adjust lenses to fit your eyes Sep 17, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.