Microsoft patents 3D video captions that describe your body movements
Microsoft describes a system for 3D video calls that tracks a person's body position frame-by-frame to create a moving 3D model. It converts the position of joints into text, which a machine learning model uses to write a real-time description of the person's actions. This text caption is then displayed next to the 3D image of the person.
In the filing’s own words
Disclosed are techniques for transcribing motions of a subject participating in a 3 D videoconference. In some configurations, a caption describing the subject’s motion is overlaid proximal to the 3 D representation of the subject. Additionally, or alternatively, physical metrics of the subject may be computed and displayed in real-time. In some configurations, a 4 D mesh of the subject is analyzed to make a per-frame determination of the subject’s pose. Joint coordinates may be obtained from the poses and transformed into text-based representations. The per-frame text-based representations of the joint coordinates may then be provided with a prompt context to a machine learning model to infer a description of the motion and/or physical metrics of the subject.
What exactly they claim (claim 1)
1. A method comprising: receiving a plurality of 3 D representations of a subject taken over a plurality of points in time; obtaining a plurality of poses of the subject from the plurality of 3 D representations; constructing a plurality of parameterized models of the subject with the plurality of poses; generating a plurality of text-based joint representations of the subject from joint coordinates received from the plurality of parameterized models; generating a model input that combines the plurality of text-based joint representations with a prompt context; providing the model input to a machine learning model; receiving, from the machine learning model, a description of a motion of the subject; and displaying the description of the motion of the subject with a rendering of one of the plurality of 3 D representations of the subject.
Facts
- Company
- Microsoft (MICROSOFT TECHNOLOGY LICENSING, LLC)
- Official title
- INTERACTIVE SUPPORT SYSTEM USING MACHINE LEARNING-BASED MOTION TRANSCRIPTION AND CAPTIONING
- Publication number
- US20260301321A1
- Application number
- 19093198
- Filed
- Mar 27, 2025
- Published
- Oct 1, 2026
- Inventors
- 4
- Claims
- 20
Classified under
G06T 17/20 Physics & computingG06T 13/40 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.