Google patents method to create training data using language models

1. A method performed by one or more computers, the method comprising: obtaining a plurality of unlabeled text inputs for a text classification task, each unlabeled text input comprising a respective sequence of text tokens, and the text classification task requiring classifying input text inputs into a plurality of categories; generating a plurality of auto-labeled training examples for the text classification task, wherein each auto-labeled training example comprises a respective text input and identifies a respective target category from the plurality of categories for the respective text input, the generating comprising: selecting, as context text inputs, one or more of the unlabeled text inputs; generating an input sequence of text tokens that includes (i) the one or more context text inputs and (ii) a text label that identifies a respective one of the plurality of categories; processing, using a pre-trained language model neural network, the input sequence to generate an output sequence of text token; and generating an auto-labeled training example that (i) includes, as the respective text input in the training example, the output sequence generated by the pre-trained language model neural network by processing the input sequence that includes the text label; and (ii) identifies, as the respective target category for the output sequence, the respective category identified by the text label in the input sequence that was provided as input to the pre-trained language model neural network; and training a task neural network to perform the text classification task on labeled training data that comprises the auto-labeled training examples, wherein the training comprises training the task neural network to predict, for each auto-labeled training example and by processing the output sequence generated as output by the pre-trained language model neural network, the respective category identified by the text label provided as input to the pre-trained language model neural network.