/Patent Tracker
GoogleGranted patentAudio & voiceInterest 6/10

Google patents AI that identifies speakers in long audio clips

This patent describes a system that uses a large language model to transcribe long conversations while identifying who is speaking. It breaks the audio into small chunks, processes them one by one, and keeps a running list of known voices to match speakers across the entire recording.

Why it mattersIt offers a more efficient way to automatically label speakers in lengthy meetings or interviews without needing separate, specialized tools for each task.

In the filing’s own words

A method includes receiving a prompt including a textual diarization request and corresponding audio data characterizing a conversation between multiple speakers. The method also includes generating a sequence of audio encoding chunks based on the corresponding data. For each respective audio encoding chunk, the method includes using a trained large language model (LLM) generating corresponding diarization results based on the respective audio encoding chunk and the textual diarization request and generating a new audio cohort for the respective audio encoding chunk based on the corresponding diarization results. The corresponding diarization results include a speech recognition result that has one or more predicted terms. Each respective predicted term is associated with a corresponding speaker token representing a predicted identity of a respective speaker that spoke the respective predicted term. The trained LLM is conditioned on a prior audio cohort generated by the trained LLM for a prior audio encoding chunk.

What exactly they claim (claim 1)

1. A joint speech recognition and speaker diarization model comprising: a tokenizer configured to: receive a prompt comprising a textual diarization request and corresponding audio data characterizing a conversation between multiple speakers; and generate a sequence of audio encoding chunks based on the corresponding audio data; and a trained large language model (LLM) configured to, for each respective audio encoding chunk of the sequence of audio encoding chunks: generate corresponding diarization results based on the respective audio encoding chunk and the textual diarization request, the corresponding diarization results comprising a speech recognition result that comprises one or more predicted terms, each respective predicted term associated with a corresponding speaker token representing a predicted identity of a respective speaker that spoke the respective predicted term, the trained LLM conditioned on a prior audio cohort generated by the trained LLM for a prior audio encoding chunk, the prior audio cohort providing reference audio data of one or more of the multiple speakers that spoke prior to the respective audio encoding chunk, wherein the trained LLM uses the prior audio cohort to determine whether the respective audio encoding chunk includes speech from any of the one or more speakers that spoke prior to the respective audio encoding chunk; and generate, based on the corresponding diarization results, a new audio cohort for the respective audio encoding chunk.

Facts

Company
Google (Google LLC)
Official title
Longform speaker diarization by prompting multimodal LLM with chunk-wise in-context self-enrollment
Patent number
US12743583B2
Application number
18646580
Filed
Apr 25, 2024
Granted
Sep 22, 2026
Inventors
3
Claims
20

Classified under

G06F 40/284 Physics & computingG10L 15/063 Physics & computingG10L 15/07 Physics & computingG10L 17/02 Physics & computing
Read the full filing (official PDF) ↗ Open in the Patent Tracker

More from Google in Audio & voice

Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.