Microsoft patents method to test ranking algorithms without live traffic
Microsoft has filed a patent for a framework that evaluates how well different content ranking models would perform without actually using them in a live system. The method converts raw model scores into standardized, calibrated scores by grouping them into specific bins, allowing companies to compare different algorithms fairly before choosing one to deploy.
In the filing’s own words
Aspects of the disclosure include methods and systems for an off-policy evaluation (OPE) framework for parameter-free, model-free score alignment across content ranking architectures. A method includes receiving one or more candidate models and generating uncalibrated scores for the models. Calibrated scores are assigned to the uncalibrated scores by quantizing the uncalibrated scores into two or more discrete bins, each bin of the two or more discrete bins mapped to a discrete range of uncalibrated scores for the candidate items. The discrete bins are assigned asymmetrical score ranges selected such that a difference between a number of uncalibrated scores assigned to each discrete bin is below a predetermined threshold. Probabilistic scores are assigned to each bin and calibrated scores are assigned to the uncalibrated scores based on the respective bin assignments of the uncalibrated scores. Off-policy estimates are then leveraged to ramp a model of the one or more candidate models.
What exactly they claim (claim 1)
1. A method for universal off-policy estimation, the method comprising: receiving one or more candidate models; generating, for the one or more candidate models, output comprising uncalibrated scores for a plurality of candidate items; generating, based on the uncalibrated scores, calibrated scores, wherein generating the calibrated scores comprises: quantizing the uncalibrated scores into two or more discrete bins, each bin of the two or more discrete bins mapped to a discrete range of uncalibrated scores for the candidate items, the discrete bins having score ranges selected such that a difference between a number of uncalibrated scores assigned to each discrete bin is below a predetermined threshold; assigning a probabilistic score to each bin of the two or more discrete bins; and assigning calibrated scores to the uncalibrated scores based on the probabilistic scores of the respective bin assignments of the uncalibrated scores; generating, from the calibrated scores, off-policy estimates; and ramping a model of the one or more candidate models based on the off-policy estimates.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- OFF-POLICY EVALUATION FRAMEWORK
- Publication number
- US20260300805A1
- Application number
- 19090985
- Filed
- Mar 26, 2025
- Published
- Oct 1, 2026
- Inventors
- 1
- Claims
- 20
Classified under
G06N 20/00 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.