Microsoft patents search training that keeps your queries private

1. A method implemented by a computing system comprising a processor and a hardware storage device storing computer-executable instructions that are executable by the processor for causing the computing system to train a learn-to-rank type model to improve quality and ranking of search results generated by the learn-to-rank type model, the method comprising: the computing system accessing client data, which includes a plurality of user queries; the computing system extracting features and patterns of the client data while keeping values of the user queries private, wherein the extracted features and patterns of the client data include: a first pattern of types of grammar used in the plurality of user queries and a second pattern of formatting of terms used in the plurality of user queries; the computing system inferring a search result quality of search result content based on user behavior data included in the client data, wherein the inferred search result quality is further based on analyzing click signals made by a user at a user system, and wherein the click signals are further used to determine at least one of: a frequency of interacting with the search result content, whether the user navigated to another resource that is referenced in the search result content, whether the user copied or saved any of the content, which one or more search results the user interacted with the most out of the different search results in the search result content, or which search results the user did not interact with at all from the search result content; the computing system generating labelled training data for a learn-to-rank type model by labelling features and patterns of the search result content with corresponding relevance scores based on the inferred search result quality; the computing system identifying conflicting relevance scores for common features and patterns of the search result content included in the labelled training data; the computing system generating filtered and labelled training data by filtering the labelled training data based on omitting or resolving the identified conflicting relevance scores to reduce noise in the labelled training data that is caused by conflicting relevance scores for common features and patterns of the search result content included in the labelled training data; the computing system applying the filtered and labelled training data to the learn-to-rank type model to train the learn-to-rank type model to improve search quality of search results provided by the learn-to-rank type model when new user queries are provided to the learn-to-rank type model; and the computing system accessing analysis data that identifies analyst assessments of quality for the different search result content based on the plurality of user queries, wherein the analysis data includes labels and annotations associated with the values of the user queries.