Microsoft patents method to refine AI models using human feedback
Microsoft describes a system that uses one AI model to label data and a second AI model to create a 'reward model' that ranks those labels. The system iteratively improves this ranking by identifying difficult examples, sending them to humans for correction, and using the corrected data to fine-tune the original AI model.
In the filing’s own words
A data processing system implements prompting a first GenAI model to label a dataset into preference pairs based on a task and preference judgement criteria; using a second GenAI model to generate an initial reward model (RM) by modeling a reward score difference for each preference pair as a reward distribution curve that ranks each preference pair; training adapters of the second model to iteratively train the initial RM by: identifying on the curve a first portion including first pairs hard for the first model to correctly label, and a second portion including positive pairs ranked higher than the first pairs, and updating the curve for a subsequent iteration RM based on an improved dataset including the positive pairs and human annotated data generated from the first pairs; and fine-tuning the first model based on either a last iteration RM or an improved dataset associated with the last iteration RM.
What exactly they claim (claim 1)
1. A data processing system comprising: a processor, and a machine-readable storage medium storing executable instructions which, when executed by the processor, cause the processor alone or in combination with other processors to perform following operations: prompting a first generative artificial intelligence (AI) model to label an unlabeled dataset into a plurality of preference pairs based on instructions of a task and one or more preference judgement criteria; inputting the preference pairs to a second generative AI model; using the second generative AI model as a base model to generate an initial reward model by modeling reward scores for each of the preference pairs, and generating a reward distribution curve of the initial reward model that ranks each of the preference pairs by a reward score difference of a respective preference pair; training adapters of the second generative AI model to iteratively train the initial reward model into a last iteration reward model by: automatically identifying a first portion and a second portion on the reward distribution curve, wherein the first portion includes first pairs that are hard for the first generative AI model to correctly label, and the second portion includes positive pairs ranked higher as compared to the first pairs, outputting the first pairs in the first portion for human annotations as annotated data, inputting the annotated data into an improved dataset including the positive pairs in the second portion, and updating the reward distribution curve for a subsequent iteration reward model based on the improved dataset; and fine-tuning the first generative AI model by either incorporating the last iteration reward model into a proximal policy optimization (PPO) loop, or inputting an improved dataset associated with the last iteration reward model into a direct preference optimization (DPO) pipeline.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- AI MODEL FINE-TUNING WITH A REWARD MODEL ITERATIVELY TRAINED ON TARGETED HUMAN FEEDBACK
- Publication number
- US20260330192A1
- Application number
- 19182816
- Filed
- Apr 18, 2025
- Published
- Oct 22, 2026
- Inventors
- 4
- Claims
- 20
Classified under
G06N 3/0475 Physics & computingG06F 40/40 Physics & computingG06N 3/0455 Physics & computingG06Q 30/0224 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.