Amazon patents camera system that logs activities using language models
Amazon describes a system where a device uses a language model to watch camera footage and identify specific physical activities, such as a user interacting with an object. When the model confirms the activity is happening, it saves a record of the event, allowing users to later ask questions about these logged moments in plain language.
In the filing’s own words
Systems and methods for visual event processing using multi-modal language models include receiving, at a first device, first user input data representing a first user command to store occurrence of a type of visual, auditory or other event over time, the event associated with a physical activity involving movement, sound, or other type of detectable physical stimuli. An activity indicator describing the activity may be stored in the first device. The first device's functionality can be extended to process data from the sensors of a second device. As such, the activity indicator stored in the first device can be used to process a first query requesting that a model identify the activity from the data. The model may determine that the data depicts the activity and a first output (natural language, notification, smart home device response, etc.) may be generated indicating occurrence of the event.
What exactly they claim (claim 1)
1. A device, comprising: one or more processors; and non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: receiving first user input data representing a first user command to store occurrence of a visual event, the visual event associated with an activity physically performed by a user in association with an object other than the device; generating, utilizing the first user input data, an activity indicator describing the visual event from the first user command; storing the activity indicator in an activity catalog disposed on the device as one of multiple activity indicators; receiving image data representing images in a field of view of a camera to the device, wherein the image data is received in response to motion being detected; generating, utilizing the activity indicator stored in the activity catalog, a first query requesting that a language model (LM) identify the activity in association with the object and the user from the image data, wherein the LM is disposed on the device and is configured to determine results utilizing the image data as the input; determining, at the device utilizing the LM and the first query, that the image data depicts the activity being performed in association with the object and the user; storing first data indicating occurrence of the visual event in response to the activity being performed in association with the object and the user; receiving second user input data representing a natural language request associated with stored visual events; determining, utilizing the second user input data, that the natural language request is associated with the visual event; and generating, utilizing the LM, a first natural language response to the second user input data, wherein generating the first natural language response is in response to the visual event being one of the visual events from the first data.
Facts
- Company
- Amazon (Amazon Technologies, Inc.)
- Official title
- Visual event processing using language models
- Patent number
- US12738060B1
- Application number
- 18759294
- Filed
- Jun 28, 2024
- Granted
- Sep 15, 2026
- Inventors
- 6
- Claims
- 20
Classified under
G06V 20/47 Physics & computingG06F 16/7328 Physics & computingG06V 20/52 Physics & computingMore from Amazon in AI & software
- Amazon patents system that cites sources for AI answers Sep 29, 2026
- Amazon patents tech to swap words in movies using actor's voice Sep 8, 2026
- Amazon patents ML tool to add graphics to live sports from moving cameras Sep 15, 2026
- Amazon patents shopping assistant that remembers your search filters Oct 6, 2026
- Amazon patents system where AI agents split tasks and talk to each other Oct 20, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.