Apple patents camera trick to read text from moving head-mounted displays
Apple describes a method for improving text recognition in images taken by cameras attached to the head, such as in mixed reality headsets. By using motion sensors to track how the device moves between frames, the system combines multiple low-resolution images into a single, sharper picture of the text. This allows the device to read text clearly even when the user's head is moving, compensating for the lack of natural eye movements that humans use to focus.
In the filing’s own words
Electronic devices, methods, and program storage devices for achieving improved optical character recognition (OCR) operations are disclosed. Performing OCR operations on captured images, e.g., images captured by cameras that are affixed to a user's body (e.g., from mixed reality devices, such as smart HMDs) requires a low-power, robust camera design. Obtaining high spatial resolution in such captured images faces many challenges. However, images with higher spatial resolution can be created by combining information extracted from multiple images captured by such devices, leveraging information obtained from positional sensors of such devices, and performing SR post-processing operations. Such higher spatial resolution images may then be used to enable high-acuity OCR capabilities. The solutions disclosed herein also compensate for the missing ability of such devices due to the lack of a vestibulo-ocular reflex (i.e., the human visual system's ability to use compensating eye movement to fixate and read text clearly, despite head movement).
What exactly they claim (claim 1)
1. A device, comprising: a memory; a positional sensor; a first image capture device; and one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors to: obtain a first video image stream from the first image capture device, wherein the first video image stream comprises a first plurality of captured images; obtain a first positional information data stream from the positional sensor, wherein the first positional information data stream comprises data indicative of a position of the device during the capture of the first plurality of captured images; identify a location of a region of interest (ROI) in a first image of the first plurality of captured images, wherein the ROI includes textual information; track the location of the ROI in a second image of the first plurality of captured images, based, at least in part, on the data from the first positional information data stream indicative of the position of the device during the capture of the second image, wherein the second image is captured after the first image; reconstruct a super-resolution (SR) version of the ROI based, at least in part, on aligning and fusing the ROI in the first image with the ROI in the second image; and perform optical character recognition (OCR) on the textual information in the reconstructed SR version of the ROI.
Facts
- Company
- Apple (Apple Inc.)
- Official title
- Optical character recognition (OCR) enhancement via inertial measurement unit (IMU)-supported super-resolution imaging
- Patent number
- US12770421B2
- Application number
- 18884605
- Filed
- Sep 13, 2024
- Granted
- Oct 20, 2026
- Inventors
- 5
- Claims
- 20
Classified under
G06V 30/10 Physics & computingG06T 3/4053 Physics & computingG06T 5/50 Physics & computingG06T 7/20 Physics & computingG06V 10/25 Physics & computingG06T 2207/20132 Physics & computingMore from Apple in AR, VR & displays
- Apple patents AR glasses that zoom in on what you look at Sep 8, 2026
- Apple patents finger ring that controls VR with taps and waves Sep 15, 2026
- Apple patents head-mounted device that types via eye swipes and hand pinches Oct 6, 2026
- Apple patents eye-tracking and hand gestures to select screen items Oct 20, 2026
- Apple patents AR glasses that adjust lenses to fit your eyes Sep 17, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.