Microsoft patents eye-tracking to credit sign language interpreters in meetings


Microsoft describes a system that tracks an interpreter's eye movement to determine which deaf or hard-of-hearing participant they are currently serving. The software then routes the interpreter's spoken voice to that specific participant's audio channel, ensuring meeting transcripts and captions correctly attribute the speech to the signer rather than the interpreter.
In the filing’s own words
The disclosed techniques provide features for automatically tethering an interpreter with multiple signers. When an interpreter is tethered to a first user and a second user, the system also automatically directs the attribution for a voice input to either a first signer or a second signer based on an eye gaze direction of the interpreter providing the voice input. When the interpreter speaks, their audio will be routed through the signer's own audio channel, which will give the appearance that the signing user is speaking, e.g., providing a verbal input to the meeting audio stream. This will ensure that the signers are each given proper attribution in generated meeting content, such as, but not limited to, captions, transcription, and meeting summaries.
What exactly they claim (claim 1)
1. A method for automatically directing attribution for a voice input to a first user or a second user based on an eye gaze direction of a third user that is providing the voice input, the method for execution on a system, the method comprising: causing a display of a user interface comprising a first rendering of a first live video stream depicting the first user and a second rendering of a second live video stream depicting the second user, the user interface displayed on a client computing device of the third user; analyzing an input signal generated by one or more sensors at the client computing device tracking the eye gaze direction of the third user, the eye gaze direction controlling a position of a target region within the user interface; analyzing the first live video stream depicting the first user to determine that the first user is performing one or more predetermined gestures and analyzing the second live video stream depicting the second user to determine that the second user is performing the one or more predetermined gestures; determining that the input signal indicates that the eye gaze direction of the third user causes the target region to be positioned to meet a predetermined condition with respect to the first rendering depicting the first user while the first user is performing the one or more predetermined gestures; in response to determining that the input signal indicates that the eye gaze direction of the third user causes the target region to be positioned to meet the predetermined condition with respect to the first rendering depicting the first user while the first user is performing the one or more predetermined gestures, causing a transition from a first operating mode to a second operating mode, wherein the first operating mode includes generating an interpreter graphical element associating the voice input of the third user with a rendering of the third user, the first operating mode further causing the system to include an identifier of the third user for labeling a transcription of the voice input provided by the third user, wherein the second operating mode includes generating a first graphical element associating the voice input of the third user with the rendering of the first user, the second operating mode further causing the system to include an identifier of the first user for labeling the transcription of the voice input provided by the third user; analyzing the first live video stream depicting the first user to determine that the first user has stopped performing the one or more predetermined gestures; analyzing the input signal to determine that the eye gaze direction of the third user causes the location of the target region to meet the predetermined condition with respect to the second rendering depicting the second user; and in response to determining that the first user has stopped performing the one or more predetermined gestures and determining that the target region meets the predetermined condition with respect to the second rendering depicting the second user while the second user is performing the one or more predetermined gestures, causing a transition from the second operating mode to a third operating mode, wherein the third operating mode causes the system to generate a second graphical element associating the voice input of the third user with the rendering of the second user, the third operating mode further causing the system to include an identifier of the second user for labeling the transcription of the voice input provided by the third user.
Facts
- Company
- Microsoft (MICROSOFT TECHNOLOGY LICENSING, LLC)
- Official title
- DYNAMIC ATTRIBUTION ASSIGNMENTS BASED ON EYE-GAZE DIRECTION OF AN INTERPRETER REPRESENTING MULTIPLE USERS
- Publication number
- US20260349643A1
- Application number
- 19195463
- Filed
- Apr 30, 2025
- Published
- Nov 5, 2026
- Inventors
- 1
- Claims
- 20
Classified under
G06F 3/1454 Physics & computingG06F 3/013 Physics & computingG06F 3/017 Physics & computingG06T 11/00 Physics & computingG06T 13/40 Physics & computingG06T 13/80 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.