Microsoft patents system that answers questions using your screen history
Microsoft has filed a patent for a system that takes screenshots of your computer screen at regular intervals to create a digital memory of your work. It uses AI to summarize these screens into events, allowing a large language model to search this history and answer questions about what you were doing.
In the filing’s own words
Implementations for capturing user activity as memory for question-answering are provided. One example includes a computing system comprising: processing circuitry and memory storing instructions that, during execution, cause the processing circuitry to: receive a query; perform semantic searching on an L 2 index comprising a database of event summaries based on the query to retrieve an L 2 answer comprising one or more relevant event summaries; generate one or more L 1 answers based on the L 2 answer using an L 1 index comprising a database of screen summaries associated with a plurality of L 0 screenshot images, wherein each of the one or more L 1 answers comprises relevant screen summaries corresponding to L 0 screenshot images associated with a respective relevant event summary; pass the query and the one or more L 1 answers to a large language model to generate an answer to the query; and output the answer to the query.
What exactly they claim (claim 1)
1. A computing system for capturing user activity as memory, the computing system comprising: processing circuitry and memory storing instructions that, during execution, cause the processing circuitry to: capturing a plurality of L 0 screenshot images at intervals during a user session; using an artificial intelligence module, extract information from the plurality of L 0 screenshot images to generate an L 1 index comprising a database of screen summaries, each of the screen summaries corresponding to a respective one of the plurality of L 0 screenshot images; using the artificial intelligence module, generate embedded information for each of the screen summaries; cluster the screen summaries based on semantic similarity of the embedded information of the screen summaries to generate an L 2 index comprising a database of event summaries, each of the event summaries corresponding to a respective cluster of the screen summaries; receive a query; perform semantic searching on the L 2 index based on the query to retrieve an L 2 answer comprising a relevant event summary; generate an L 1 answer based on the L 2 answer using the L 1 index, wherein the L 1 answer comprises a relevant screen summary associated with the relevant event summary; pass the query and 7 to a large language model to generate an answer to the query; and output the answer to the query.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- CAPTURING USER ACTIVITY AS MEMORY FOR QUESTION-ANSWERING
- Publication number
- US20260320095A1
- Application number
- 19178666
- Filed
- Apr 14, 2025
- Published
- Oct 15, 2026
- Inventors
- 3
- Claims
- 20
Classified under
G06F 16/334 Physics & computingG06F 16/3329 Physics & computingG06F 16/345 Physics & computingG06F 16/353 Physics & computingG06F 16/532 Physics & computingG06F 40/30 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents AI tool that writes and runs software tests from text Sep 1, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.