Apple patents head-tracking tech to sharpen mixed reality video
Apple describes a method for improving the clarity of live video feeds in mixed reality devices by combining current and past camera frames. The system uses sensors to predict where the user's head is pointing and estimates the distance of objects in the scene to align these frames accurately. This process reduces visual noise, resulting in a sharper image of the real world overlaid with digital content.
In the filing’s own words
Various implementations disclosed herein include devices, systems, and methods that improve passthrough video image quality using a hardware-based temporal noise reduction process. For example, a process may obtain image content. The image content may include frames corresponding to views of a three-dimensional ( 3 D) environment. The frames may include a current frame and previously-captured frames. The process may further predict a user head pose corresponding to a viewpoint of an image sensor with respect to the 3 D environment. The process may further estimate depth attributes corresponding to distances of elements of a scene depicted in the current frame and generate an enhanced frame by combining the current frame with the previously-captured frames. Combining the frames may be based on aligning the previously-captured frames with the current frame based on the predicted user head pose and the estimated depth attributes. The process may further present the enhanced frame.
What exactly they claim (claim 1)
1. A method comprising: at an electronic device having a processor and one or more displays: obtaining image content from an image sensor, the image content comprising frames corresponding to views of a three-dimensional (3D) environment, the frames comprising a current frame and one or more previously-captured frames; predicting a user head pose based on sensor data obtained via one or more sensors, the user head pose corresponding to a viewpoint of the image sensor with respect to the 3D environment during capture of the current frame; estimating depth attributes corresponding to distances of one or more elements of a scene depicted in the current frame; generating an enhanced frame by combining the current frame with the one or more previously-captured frames, wherein the combining is based on aligning at least one of the previously-captured frames with the current frame based on the predicted user head pose and the estimated depth attributes; and presenting, to the user via the one or more displays, the enhanced frame.
Facts
- Company
- Apple (Apple Inc.)
- Official title
- GAZE AND HEAD TRACKING INFORMED HARDWARE-BASED PASSTHROUGH NOISE REDUCTION
- Publication number
- US20260301133A1
- Application number
- 19559053
- Filed
- Mar 6, 2026
- Published
- Oct 1, 2026
- Inventors
- 3
- Claims
- 20
Classified under
G06T 5/70 Physics & computingG06F 3/012 Physics & computingG06F 3/013 Physics & computingG06T 3/18 Physics & computingG06T 5/50 Physics & computingG06T 7/50 Physics & computingMore from Apple in AR, VR & displays
- Apple patents AR glasses that zoom in on what you look at Sep 8, 2026
- Apple patents finger ring that controls VR with taps and waves Sep 15, 2026
- Apple patents head-mounted device that types via eye swipes and hand pinches Oct 6, 2026
- Apple patents eye-tracking and hand gestures to select screen items Oct 20, 2026
- Apple patents AR glasses that adjust lenses to fit your eyes Sep 17, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.