Meta patents selecting objects with eyes and hand gestures in AR
Meta describes a system where a user selects items in their field of view by combining where they are looking with a specific hand gesture. The system uses cameras on a head-wearable device to track eye movement and a wrist-wearable device to detect the size and shape of the hand gesture. It then identifies the object being looked at and confirms the selection based on these combined inputs.
In the filing’s own words
A method of performing an object selection procedure is described. The method includes, in response to a user performing a selection input: receiving, from one or more cameras of a head-wearable device worn by the user, image data of a field-of-view of the user, the image data including one or more objects, receiving, from the head-wearable device, gaze data comprising a user gaze at one or more of the one or more objects, receiving, from a wrist-wearable device worn by the user, hand gesture data indicating the user performing a selection hand gesture, wherein the selection hand gesture has a gesture size and gesture shape, determining, based on the gaze data, the image data, the gesture size, and the gesture shape, at least a portion of the one or more objects, and presenting, to the user, an indication that the portion of the one or more objects is selected.
What exactly they claim (claim 1)
1. A method, comprising: in response to a user performing a selection input: receiving, from one or more cameras of a head-wearable device worn by the user, image data of a field-of-view of the user, the image data including one or more objects; receiving, from the head-wearable device, gaze data comprising a user gaze at one or more of the one or more objects; receiving, from a wrist-wearable device worn by the user, hand gesture data indicating the user performing a selection hand gesture, wherein the selection hand gesture has a gesture size and a gesture shape; determining, based on the gaze data, the image data, the gesture size, and the gesture shape, at least a portion of the one or more objects; and presenting, to the user, an indication that the portion of the one or more objects is selected.
Facts
- Company
- Meta (Meta Platforms Technologies, LLC)
- Official title
- SYSTEMS AND METHODS FOR MULTIMODAL FREEFORM OBJECT IDENTIFICATION
- Publication number
- US20260299683A1
- Application number
- 19632263
- Filed
- Mar 28, 2026
- Published
- Oct 1, 2026
- Inventors
- 3
- Claims
- 20
Classified under
G06F 3/013 Physics & computingG06F 3/017 Physics & computingG06F 3/0346 Physics & computingMore from Meta in AR, VR & displays
- Meta patents VR avatars that mimic facial expressions from voice Sep 15, 2026
- Meta patents spatial mouse for pointing at and controlling real-world objects Sep 24, 2026
- Meta patents AR notifications that grow when you look at them Sep 1, 2026
- Meta patents AR glasses with antennas built into the lenses Sep 8, 2026
- Meta patents auto-pause for VR headsets when you take them off Oct 6, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.