Microsoft patents method to train AI without reference answers
Microsoft has filed a patent for a way to train language models that does not require a separate reference model or 'gold standard' answer. The system compares multiple possible responses to a question, scores them based on quality and length, and uses the differences between good and bad answers to adjust the AI's parameters. This approach aims to make the AI produce richer, more detailed answers without favoring short responses.
In the filing’s own words
The disclosed relates to a method and system for training language models using a reference-free multi-preference alignment approach. It addresses the challenge of aligning model outputs with human preferences by optimizing a contrastive loss function over selected responses. The method involves identifying candidate responses, calculating reward scores, and selecting a subset of responses for optimization. Deviations from mean reward scores are used to assign weights, and a weighted contrastive loss function is generated to update model parameters. This approach enhances model alignment by considering multiple preferences and mitigating biases toward brevity, resulting in richer and more detailed outputs.
What exactly they claim (claim 1)
1. A computer-implemented method for training a language model without a reference model, the method comprising: identifying a dataset comprising queries and associated responses, each response having a reward value; for each query: computing a mean reward by averaging the reward values associated with each of the responses; partitioning the associated responses into a first set containing responses with associated rewards that are above the mean reward and a second set containing responses with associated rewards that are below the mean reward; computing quality scores for each response, the quality score representing an length normalized measure of a certainty of the language model in generating each token in the response; calculating deviation-based weights for each response based on a difference between the response's associated reward and the mean reward; computing a first partition function by applying the deviation-based weights to the quality score, and summing resulting weighted factors for responses in the first set; computing a second partition function by applying the deviation-based weights to the quality score, and summing resulting weighted factors for responses in the second set; computing a loss function based upon the first partition function and second partition function; and computing an overall dataset loss by averaging individual loss functions across the dataset; and updating model parameters using gradient descent based on the overall dataset loss.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- REFERENCE FREE ALIGNMENT FOR MULTI-PREFERENCE OPTIMIZATION
- Publication number
- US20260310954A1
- Application number
- 19092454
- Filed
- Mar 27, 2025
- Published
- Oct 8, 2026
- Inventors
- 4
- Claims
- 20
Classified under
G06N 3/08 Physics & computingG06F 40/284 Physics & computingG06F 40/40 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.