Google patents method to test AI models using data density
Google has been granted a patent for a system that evaluates how well a machine learning model is trained by measuring how close its test questions are to the data it learned from. The method converts both the training data and the test questions into mathematical coordinates, then calculates a quality score based on how crowded the area around each test question is in that space.
In the filing’s own words
Disclosed herein are systems and methods for objectively characterizing machine-learning models including receiving first training data formatted to be used in the training of a machine-learning model; receiving one or more challenge queries formatted to be run on the machine-learning model; generating, for the first training data, a plurality of associated training vectors that embed at least some of the first training data into a vector space; generating, for each of the one or more challenge queries, a plurality of associated challenge vectors that embed at least some of the challenge queries into the vector space; and determining, for each challenge query, a corresponding quality metric for the machine-learning model by determining a neighborhood density for each of the challenge queries in the vector space.
What exactly they claim (claim 1)
1. A system for objective characterization of machine-learning models, the system comprising: one or more processors; and computer memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: receiving first training data formatted to be used in the training of a machine-learning model; receiving one or more challenge queries formatted to be run on the machine-learning model; generating, for the first training data, a plurality of associated training vectors that embed at least some of the first training data into a vector space; generating, for each of the one or more challenge queries, a plurality of associated challenge vectors that embed at least some of the challenge queries into the vector space; determining, for each challenge query, a corresponding quality metric for the machine-learning model by determining a neighborhood density for each of the challenge queries in the vector space, wherein determining the neighborhood density for each of the challenge queries in the vector space comprises determining a count of a number of training vectors within a threshold distance of each of the challenge vectors in the vector space or finding an average distance to N nearest training vectors in the vector space; and responsive to determining, for each challenge query, a corresponding quality metric for the machine-learning model, creating the machine-learning model comprising training the machine-learning model using the first training data.
Facts
- Company
- Google (X Development LLC)
- Official title
- Characterization of machine-learning models
- Patent number
- US12769736B2
- Application number
- 19042774
- Filed
- Jan 31, 2025
- Granted
- Oct 20, 2026
- Inventors
- 3
- Claims
- 18
Classified under
G06F 16/24542 Physics & computingG06F 16/215 Physics & computingMore from Google in AI & software
- Google patents video search that answers questions using AI descriptions Sep 15, 2026
- Google patents photo tool that removes people and their gear Sep 15, 2026
- Google patents AI that writes live meeting notes and summaries Sep 15, 2026
- Google patents AI tool that turns messy emergency notes into clear messages Sep 3, 2026
- Google patents AI that clicks through apps to finish tasks for you Sep 24, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.