Apple patents hand tracking that switches to a simple icon in low light
This patent describes a method for tracking hands in mixed reality environments by measuring the brightness of the camera feed. If the light is sufficient, the system cuts out the hand from the video to place it over virtual objects. If the light is too dim for accurate tracking, the system stops cutting out the hand and instead displays a simple icon to show where the hand is.
In the filing’s own words
Managing hands matting based on light conditions includes receiving image data comprising a hand, and determining a brightness in the image data. If the brightness in the first image data satisfies the brightness threshold, hands matting is performed on the image data to extract the hand from the image data and the image data is blended with virtual content in accordance with the extracted hand. If the brightness in the first image data satisfies a disablement threshold, hands matting is disabled and the image data is blended with virtual content without extracting the hand. An alternative representation of the location of the hand is depicted in the final image.
What exactly they claim (claim 1)
1. A non-transitory computer readable medium comprising computer readable code executable by one or more processors to: receive an image data stream comprising a series of frames in which a hand is captured; and in accordance with a first brightness value for a first portion of the image data stream satisfying a first brightness threshold: identify a set of pixels of one or more of the series of frames corresponding to the hand, and blend the image data stream with virtual content to provide a visual indication of the hand relative to the virtual content based on the set of pixels.
Facts
- Company
- Apple (Apple Inc.)
- Official title
- Hands matting responsive to low light conditions
- Patent number
- US12770321B2
- Application number
- 18675828
- Filed
- May 28, 2024
- Granted
- Oct 20, 2026
- Inventors
- 2
- Claims
- 20
Classified under
G06T 15/503 Physics & computingG06T 19/006 Physics & computingMore from Apple in AR, VR & displays
- Apple patents AR glasses that zoom in on what you look at Sep 8, 2026
- Apple patents finger ring that controls VR with taps and waves Sep 15, 2026
- Apple patents head-mounted device that types via eye swipes and hand pinches Oct 6, 2026
- Apple patents eye-tracking and hand gestures to select screen items Oct 20, 2026
- Apple patents AR glasses that adjust lenses to fit your eyes Sep 17, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.