Nvidia patents system that suggests tasks based on where you look
Nvidia describes a system that uses a vision-language model to analyze what you are looking at in a visual scene. It identifies the object in your line of sight and suggests specific actions you can take with it, such as sending a command to control that device.
In the filing’s own words
Disclosed are apparatuses, systems, and techniques that implement attention-driven task suggestion with adaptive presentation using vision-language models. The system may receive image data representing a visual scene and an attention signal indicating a location within the visual scene, process the image data and the attention signal using a vision-language model to identify an object corresponding to the location, and generate task identifiers representing tasks associated with the object. The system may cause presentation of the task identifiers, and responsive to a selection of a task identifier, transmit a command to a device control interface associated with the object to execute a task.
What exactly they claim (claim 1)
1. A system comprising one or more processors to: receive image data representing a visual scene and spatial telemetry data comprising a gaze coordinate indicating a location within the visual scene; generate, using a vision-language model that processes the image data and the spatial telemetry data, one or more task identifiers of one or more tasks associated with an object at the location; cause presentation of the one or more task identifiers; and responsive to a selection of a task identifier from the one or more task identifiers, transmit a command to a device control interface associated with the object to execute a task corresponding to the selected task identifier.
Facts
- Company
- Nvidia (NVIDIA Corporation)
- Official title
- ATTENTION-DRIVEN TASK SUGGESTION WITH ADAPTIVE PRESENTATION
- Publication number
- US20260267458A1
- Application number
- 19559839
- Filed
- Mar 6, 2026
- Published
- Sep 10, 2026
- Inventors
- 3
- Claims
- 20
Classified under
G06F 3/04815 Physics & computingG06F 3/013 Physics & computingG06F 3/0484 Physics & computingG06F 9/453 Physics & computingG06F 40/30 Physics & computingG06F 40/40 Physics & computingMore from Nvidia in AI & software
- Nvidia patents AI coach that analyzes your gameplay to suggest strategies Oct 13, 2026
- Nvidia patents AI system that writes and voices character conversations Sep 10, 2026
- Nvidia patents system that uses AI to find bugs in game videos Sep 17, 2026
- Nvidia patents AI method to make digital characters move realistically Sep 8, 2026
- Nvidia patents AI system that builds image generators from text prompts Sep 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.