Microsoft patents system that auto-suggests follow-up info for on-screen content
Microsoft describes a system that watches what you are looking at on your screen and automatically suggests related information. It uses an AI model to understand the text or images you see and then pulls up extra details or next steps without you having to ask. The goal is to make AI tools less work by proactively helping you with tasks.
In the filing’s own words
The techniques presented herein provide a system for proactive output of information that is relevant to the context of on-screen visual content such as text and images. Recent developments in artificial intelligence systems have enabled various user productivity tools that streamline aspects of the user experience of a computing device (e.g., a laptop, a smartphone, a tablet). However, some users may not take advantage of artificial intelligence productivity tools despite their utility due to the additional technical burden that users may find overwhelming. As such, the present system proactively extracts on-screen content via a content capture for analysis by a computational model (e.g., a generative language model) in response to a user activity trigger. Accordingly, the computational model identifies the semantic content of the content capture (e.g., the concepts and/or ideas) and follow-up information that is contextually relevant to the content capture without requiring additional input from the user.
What exactly they claim (claim 1)
1. A method for outputting contextual insights for a content capture in a graphical user interface, the method comprising: receiving the content capture, wherein: the content capture contains visual content conveying a semantic content of the content capture; and the content capture is associated with a user entity; identifying the semantic content based on a processing of the visual content by execution of a computational model; determining, by the computational model, a follow-up input based on the semantic content of the content capture; retrieving, by the computational model supplemental information from a supplemental information repository; determining, by the computational model, an output corresponding to the follow-up input, wherein the computational model is configured to determine the output corresponding to the follow-up input based on the supplemental information associated with the user entity; and rendering the follow-up input and the output corresponding to the follow-up input in a section of the graphical user interface.
Facts
- Company
- Microsoft (MICROSOFT TECHNOLOGY LICENSING, LLC)
- Official title
- AUTOMATED CONTEXTUAL INSIGHTS FOR ON-SCREEN VISUAL CONTENT
- Publication number
- US20260299972A1
- Application number
- 19093967
- Filed
- Mar 28, 2025
- Published
- Oct 1, 2026
- Inventors
- 6
- Claims
- 20
Classified under
G06F 9/451 Physics & computingG06F 16/953 Physics & computingG06V 20/50 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.