Google patents method for training image-text AI models

1. A method of training a neural network for performing one or more multi-modal tasks, the neural network comprising: a visual encoder neural network that is configured to process a visual input that includes one or more images to generate an encoded representation of the visual input; and a decoder-only language model neural network, wherein the decoder-only language model neural network is configured to process a current text sequence to generate an output defining a new token to be appended to the current text sequence, wherein the current text sequence comprises a respective text token at each of one or more input positions, and wherein the training comprises: obtaining a training data set, the training data set comprising a plurality of image-text sequence pairs; and training the neural network on the training data set to minimize an overall loss function that comprises (i) a contrastive loss that measures similarities between image embeddings generated from encoded representations generated by the visual encoder neural network and text embeddings generated from outputs of the language model neural network and (ii) an image captioning loss that measures a quality of text captions generated by the language model neural network given encoded representations generated by the visual encoder neural network.