GoogleGranted patentAudio & voiceInterest 8/10

Google patents system to create sound effects from text descriptions

Illustration of the invention
Illustration based on the patent drawing. Not a photo of a real product.
Patent drawing
Patent drawing from the official filing.

This patent describes a system that turns written descriptions into actual audio files. It works by first training a model to understand the link between words and sounds using user comments from video platforms, then using a diffusion model to generate new audio that matches a specific text prompt.

Why it mattersThis technology could allow creators to easily produce custom sound effects or background noise just by typing what they hear, rather than searching for existing files.

In the filing’s own words

A corpus of textual data is generated with a machine-learned text generation model. The corpus of textual data includes a plurality of sentences. Each sentence is descriptive of a type of audio. For each of a plurality of audio recordings, the audio recording is processed with a machine-learned audio classification model to obtain training data including the audio recording and one or more sentences of the plurality of sentences closest to the audio recording within a joint audio-text embedding space of the machine-learned audio classification model. The sentence(s) are processed with a machine-learned generation model to obtain an intermediate representation of the one or more sentences. The intermediate representation is processed with a machine-learned cascaded diffusion model to obtain audio data. The machine-learned cascaded diffusion model is trained based on a difference between the audio data and the audio recording.

What exactly they claim (claim 1)

1. A computing system, comprising: one or more processors; and one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising: obtaining a plurality of audio samples and an associated corpus of descriptive textual data from an audiovisual data hosting entity, wherein the corpus of descriptive textual data comprises, for each of the plurality of audio samples, one or more portions of textual content that are provided by users of the audiovisual data hosting entity responsive to the audio sample, wherein the one or more portions of textual content comprise comments; for each audio sample of the plurality of audio samples: processing the audio sample with an audio embedding portion of a machine-learned audio classification model to obtain an audio embedding; processing the one or more portions of textual content that describe the audio sample with a text embedding portion of the machine-learned audio classification model to obtain a text embedding; and training the machine-learned audio classification model based on a contrastive loss function that evaluates a difference between the audio embedding and the text embedding: processing a query comprising textual content that indicates a desired type of audio content with a machine-learned generator model to generate an intermediate representation of the textual content of the query; and processing the intermediate representation of the textual content with a machine-learned diffusion model to obtain audio data, wherein the audio data comprises audio of the desired type of audio content.

Facts

Company
Google (Google LLC)
Official title
Diffusion models for generation of audio data based on descriptive textual prompts
Patent number
US12784740B2
Application number
18651296
Filed
Apr 30, 2024
Granted
Nov 3, 2026
Inventors
10
Claims
20

Classified under

G10L 15/063 Physics & computingG06F 40/40 Physics & computingG10H 1/0025 Physics & computingG10L 15/16 Physics & computingG10H 2210/111 Physics & computing
Read the full filing (official PDF) ↗ Open in the Patent Tracker

More from Google in Audio & voice

Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.