Amazon patents system to find flaws in AI language models
Amazon has patented a system that automatically creates tricky questions to test AI language models. It uses one AI to tweak a question until it forces another AI to give an answer that breaks specific rules or constraints.
In the filing’s own words
Synthetic prompts are generated for use with a language model by providing an initial prompt to a first machine learning model that is trained to determine modifications to the prompt having an increased probability of causing the language model to generate a response that violates a constraint. The first machine learning model may use a reward function that determines a reward value based on the text of the initial prompt, the modification, the text of the modified prompt, and one or more intervals of time, the reward value being associated with the probability of a response to the prompt deviating from a constraint. One or more additional machine learning models may determine scores based on characteristics of the prompts and responses generated in this manner, and rationales associated with the scores. The scores and rationales may be stored and used to affect future responses generated by the language model.
What exactly they claim (claim 1)
1. A system comprising: one or more non-transitory memories storing computer-executable instructions; and one or more hardware processors to execute the computer-executable instructions to: access a first prompt comprising first text; use a first machine learning model to determine a modification to the first prompt, wherein the first machine learning model comprises a model-free reinforcement algorithm that includes a reward function for determining a reward value based on: a first state comprising the first text, an action comprising the modification to the first prompt, a second state comprising second text based on the first text and the modification to the first prompt, and one or more intervals of time; wherein the reward value is associated with a probability of a first response associated with the second text deviating from one or more constraints associated with a second machine learning model, and wherein the first machine learning model is trained to maximize the reward value by determining modifications to prompts that are associated with generation, by the second machine learning model, of responses having characteristics associated with an invalid response; determine a second prompt based on the first prompt and the modification to the first prompt determined using first output from the first machine learning model, wherein the second prompt comprises the second text; use the second machine learning model to determine the first response based on the second prompt, wherein the second machine learning model is trained to determine responses based on text and semantic information associated with prompts, wherein the responses are associated with the one or more constraints, and wherein the first response comprises third text that deviates from the one or more constraints; and determine an output based on the first response.
Facts
- Company
- Amazon (AMAZON TECHNOLOGIES, INC.)
- Official title
- Systems for generation of prompts for evaluation of language models
- Patent number
- US12801855B2
- Application number
- 18736763
- Filed
- Jun 7, 2024
- Granted
- Nov 17, 2026
- Inventors
- 1
- Claims
- 20
Classified under
G06F 40/30 Physics & computingG10L 15/063 Physics & computingG06F 40/56 Physics & computingG10L 15/16 Physics & computingMore from Amazon in AI & software
- Amazon patents system that cites sources for AI answers Sep 29, 2026
- Amazon patents voice assistants that adjust personality based on your mood Nov 3, 2026
- Amazon patents tech to swap words in movies using actor's voice Sep 8, 2026
- Amazon patents ML tool to add graphics to live sports from moving cameras Sep 15, 2026
- Amazon patents shopping assistant that remembers your search filters Oct 6, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.
