AmazonGranted patentAI & softwareInterest 6/10

Amazon patents system that mixes audio and text to help AI answer questions

Amazon describes a system that helps large language models find relevant information by searching a database of both spoken words and background sounds. The system picks the right type of data to search based on the user's question, finds the closest matches, and sends the original audio or text to the AI to create an answer.

Why it mattersThis could make AI assistants better at understanding complex audio files, such as podcasts or meetings, by letting them focus on specific sounds or speech patterns rather than just reading transcripts.

In the filing’s own words

A retrieval augmented generation (RAG) system determines, based on a query, an audio embedding type or a text embeddings type to use to filter an embeddings store, wherein the embedding store includes embeddings of the audio embedding type and the text embeddings type in a vector space. The RAG system obtains, from a filtered portion of the embeddings store, embeddings based on similarity scores between the plurality of embeddings and a representation of the query in the vector space. The RAG system obtains embedding source data from which a selected one of the plurality of embeddings of the embeddings store have been generated, wherein the embedding source data is source audio data or transcribed source audio data used to generate the embeddings in the embeddings store. The RAG system provides the query and the embedding source data for the selected embedding to the multimodal LLM.

What exactly they claim (claim 1)

1. A system, comprising: one or more computing devices configured to implement a retrieval augmented generation (RAG) system, configured to: receive a query for a multimodal large language model (LLM) to answer; determine, based on the query, an audio embedding type or a text embeddings type to use to filter an embeddings store storing embeddings of the audio embedding type and the text embeddings type that have been embedded in a vector space; obtain, from a filtered portion of the embeddings store, a plurality of embeddings based on similarity scores between the plurality of embeddings and a representation of the query in the vector space, wherein: the embeddings store comprises: audio embeddings generated from sounds included in non-speech portions of audio data, text embeddings generated from transcribed speech portions of the audio data, audio embedding metadata, and text embedding metadata; obtain embedding source data from which a selected one of the plurality of embeddings of the embeddings store have been generated, wherein the embedding source data is source audio data or transcribed source audio data used to generate the embeddings in the embeddings store; and provide the query and the embedding source data for the selected embedding to the multimodal LLM; and wherein the system further comprises one or more computing devices configured to implement the multimodal LLM, configured to: receive the query and the embedding source data for the selected embedding; and generate a response to the query using the source data.

Facts

Company
Amazon (Amazon Technologies, Inc.)
Official title
Multimodal retrieval augmented generation
Patent number
US12792647B1
Application number
18747774
Filed
Jun 19, 2024
Granted
Nov 10, 2026
Inventors
2
Claims
20

Classified under

G06F 16/3344 Physics & computingG06F 16/3347 Physics & computingG06F 16/335 Physics & computingG10L 15/26 Physics & computing

More from Amazon in AI & software

Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.