AmazonGranted patentAI & softwareInterest 6/10

Amazon patents system to find flaws in AI language models

Patent drawing
Patent drawing from the official filing.

Amazon has patented a system that automatically creates tricky questions to test AI language models. It uses one AI to tweak a question until it forces another AI to give an answer that breaks specific rules or constraints.

Why it mattersThis helps developers find and fix weaknesses in AI systems before they are used by the public.

In the filing’s own words

Synthetic prompts are generated for use with a language model by providing an initial prompt to a first machine learning model that is trained to determine modifications to the prompt having an increased probability of causing the language model to generate a response that violates a constraint. The first machine learning model may use a reward function that determines a reward value based on the text of the initial prompt, the modification, the text of the modified prompt, and one or more intervals of time, the reward value being associated with the probability of a response to the prompt deviating from a constraint. One or more additional machine learning models may determine scores based on characteristics of the prompts and responses generated in this manner, and rationales associated with the scores. The scores and rationales may be stored and used to affect future responses generated by the language model.

What exactly they claim (claim 1)

1. A system comprising: one or more non-transitory memories storing computer-executable instructions; and one or more hardware processors to execute the computer-executable instructions to: access a first prompt comprising first text; use a first machine learning model to determine a modification to the first prompt, wherein the first machine learning model comprises a model-free reinforcement algorithm that includes a reward function for determining a reward value based on: a first state comprising the first text, an action comprising the modification to the first prompt, a second state comprising second text based on the first text and the modification to the first prompt, and one or more intervals of time; wherein the reward value is associated with a probability of a first response associated with the second text deviating from one or more constraints associated with a second machine learning model, and wherein the first machine learning model is trained to maximize the reward value by determining modifications to prompts that are associated with generation, by the second machine learning model, of responses having characteristics associated with an invalid response; determine a second prompt based on the first prompt and the modification to the first prompt determined using first output from the first machine learning model, wherein the second prompt comprises the second text; use the second machine learning model to determine the first response based on the second prompt, wherein the second machine learning model is trained to determine responses based on text and semantic information associated with prompts, wherein the responses are associated with the one or more constraints, and wherein the first response comprises third text that deviates from the one or more constraints; and determine an output based on the first response.

Facts

Company
Amazon (AMAZON TECHNOLOGIES, INC.)
Official title
Systems for generation of prompts for evaluation of language models
Patent number
US12801855B2
Application number
18736763
Filed
Jun 7, 2024
Granted
Nov 17, 2026
Inventors
1
Claims
20

Classified under

G06F 40/30 Physics & computingG10L 15/063 Physics & computingG06F 40/56 Physics & computingG10L 15/16 Physics & computing

More from Amazon in AI & software

Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.