Microsoft patents method to group screenshots by meaning
Microsoft describes a system that takes multiple screenshots and uses AI to extract text and visual details from each one. It then connects these screenshots into a graph based on how similar they are, grouping them into clusters that represent related user activities.
In the filing’s own words
Techniques and related implementations for analyzing semantic sessions are provided. One implementation includes a computing system comprising: processing circuitry and memory storing instructions that, during execution, cause the processing circuitry to perform the steps of: receiving a plurality of screenshot images; for each of the plurality of screenshot images: identifying one or more image regions and one or more text regions on the screenshot image; and generating features for the screenshot image based on the one or more image regions and the one or more text regions using an artificial intelligence module; generating a user activity graph comprising nodes and edges, wherein the nodes correspond to the plurality of screenshot images, and wherein the edges correspond to commonalities between node pairs based on the generated features of the plurality of screenshot images; and identifying one or more node clusters from the user activity graph based on semantic relationships.
What exactly they claim (claim 1)
1. A computing system for analyzing semantic sessions, the computing system comprising: processing circuitry and memory storing instructions that, during execution, cause the processing circuitry to perform the steps of: receiving a plurality of screenshot images; for each of the plurality of screenshot images: identifying one or more image regions and one or more text regions on the screenshot image, and generating features for the screenshot image based on the one or more image regions and the one or more text regions using an artificial intelligence (AI) module, wherein the features comprise textual embeddings and visual embeddings of the screenshot image; generating a user activity graph comprising nodes and edges, wherein the nodes correspond to the plurality of screenshot images, and wherein the edges correspond to commonalities between node pairs based on the generated features of the plurality of screenshot images; and identifying one or more node clusters from the user activity graph based on semantic relationships.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- ANALYZING SEMANTIC USER SESSIONS USING SCREEN UNDERSTANDING
- Publication number
- US20260311448A1
- Application number
- 19170440
- Filed
- Apr 4, 2025
- Published
- Oct 8, 2026
- Inventors
- 4
- Claims
- 20
Classified under
G06V 10/761 Physics & computingG06F 16/3344 Physics & computingG06F 16/338 Physics & computingG06F 40/295 Physics & computingG06F 40/30 Physics & computingG06V 10/26 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.