Google patents method to train image AI using reward signals

1. A method performed by one or more computers and for training a computer vision neural network that is configured to process an input image to generate a network output that specifies an output sequence that comprises a plurality of tokens from a vocabulary and that represents a predicted output for the input image for a computer vision task, the method comprising: obtaining a pre-training data set that comprises a plurality of pre-training training examples, each pre-training example comprising (i) a pre-training image and (ii) a target output sequence representing a target output for the pre-training image for the computer vision task; obtaining a fine-tuning data set that includes a plurality of fine-tuning images; training the computer vision neural network on the pre-training data set to optimize a maximum likelihood objective that measures, for each pre-training example, a likelihood assigned to the target output sequence in the pre-training example by a network output generated by the computer vision neural network by processing the pre-training image in the pre-training example, comprising, at each of a plurality of pre-training training steps: obtaining a set of one or more pre-training training examples from the pre-training data set; for each of the pre-training training examples: processing the pre-training image in the pre-training training example using the computer vision neural network to generate a third network output; and determining a likelihood assigned to the target output sequence in the pre-training example by the third network output generated by the computer vision neural network by processing the pre-training image in the pre-training example; and determining a gradient with respect to parameters of the computer vision neural network of the maximum likelihood objective, wherein the maximum likelihood objective measures an average of the likelihoods for the pre-training examples in the set of one or more pre-training examples; and after training the computer vision neural network on the pre-training data set to optimize the maximum likelihood objective: training the computer vision neural network on the fine-tuning data set, the training comprising, at each of a plurality of fine-tuning training steps: obtaining a set of one or more fine-tuning images from the fine-tuning data set; for each of the fine-tuning images: processing the fine-tuning image using the computer vision neural network to generate a first network output that specifies a first output sequence that represents a first predicted output for the computer vision task; and determining a first task reward value for the first predicted output for the computer vision task, wherein the first task reward value measures a task-specific quality of the first predicted output; and training the computer vision neural network using the first task rewards on a reinforcement learning objective that maximizes expected task rewards received for network outputs generated by the computer vision neural network.