Microsoft patents method to pick best answers for training AI models

1. A computer-implemented method for training a language model, comprising: identifying N candidate responses for an input query; identifying reward scores for each of the N candidate responses; selecting K responses from the N candidate responses, where K is less than N by: identifying a highest-reward response as a positive example; and selecting K−1 negative responses from remaining responses by one or more of: selecting remaining responses with the lowest reward scores; applying clustering to group the remaining responses based on corresponding embedding vectors and selecting a response from each cluster with a lowest reward score; or selecting responses by weighting responses and assigning higher weights to responses with lower rewards, and selecting a subset of K−1 responses that minimizes weighted distances between responses while ensuring selected responses are semantically distinct from each other by selecting responses from different regions of an embedding space that clusters vector representations corresponding to the remaining responses based upon the weights; calculating a mean reward score from the reward scores of the K selected responses; computing deviations of each selected response's reward score from the mean reward score; partitioning the K selected responses into positive and negative groups by assigning responses with deviations above the mean reward score to the positive group, and assigning responses with deviations below or equal to the mean reward score to the negative group; assigning weights to each response based on its deviation from the mean reward score, wherein: responses with larger absolute deviations from the mean reward score are assigned higher weights than responses with smaller absolute deviations; computing modified scores for each response by combining a base score with the assigned weight; generating a weighted contrastive loss function based on the modified scores of the responses in the positive and negative groups; and updating parameters of the language model by optimizing the weighted contrastive loss function.