Apple patents system that mixes real and virtual objects to run commands
Apple describes a method where a device uses sensors to map a physical space and creates a single model that links real items with virtual ones. This combined map allows the system to understand how objects relate to each other, enabling it to carry out specific user commands based on those connections.
In the filing’s own words
Various implementations disclosed herein include devices, systems, and methods that uses object relationships represented in a scene graph to execute an instruction. For example, an example process may include obtaining sensor data for a physical environment including physical objects, determining a scene understanding of the physical environment based on the sensor data, generating a unified scene graph based on the scene understanding that represents the physical objects, virtual objects, and relationships amongst the physical objects and the virtual objects, receiving an instruction for an action, and implementing the action based on the unified scene graph.
What exactly they claim (claim 1)
1. A method comprising: at an electronic device having a processor: obtaining sensor data for a physical environment comprising physical objects; determining a scene understanding of the physical environment based on the sensor data; generating a unified object relationship model based on the scene understanding, the unified object relationship model representing the physical objects, virtual objects, and relationships amongst the physical objects and the virtual objects; receiving an instruction for an action; and implementing the action based on the unified object relationship model.
Facts
- Company
- Apple (Apple Inc.)
- Official title
- Instructional interactions using scene graphs
- Patent number
- US12802363B1
- Application number
- 17678629
- Filed
- Feb 23, 2022
- Granted
- Nov 17, 2026
- Inventors
- 3
- Claims
- 20
Classified under
G06T 19/006 Physics & computingG06F 3/04815 Physics & computingG06F 3/167 Physics & computingG06T 19/20 Physics & computingG06T 2219/2004 Physics & computingMore from Apple in AR, VR & displays
- Apple patents AR glasses that zoom in on what you look at Sep 8, 2026
- Apple patents finger ring that controls VR with taps and waves Sep 15, 2026
- Apple patents head-mounted device that types via eye swipes and hand pinches Oct 6, 2026
- Apple patents eye-tracking and hand gestures to select screen items Oct 20, 2026
- Apple patents AR glasses that adjust lens spacing to fit your eyes Nov 3, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.
