Google patents smarter way to load AI models onto chips

1. A method performed by a compiler and a hardware accelerator, the method comprising: identifying, by the compiler, a first set of neural network models that have been executed on a hardware accelerator of a computing device more than a threshold number of times in a preset amount of time in the past; identifying, by the compiler, a sequence in which the first set of neural network models are executed on the hardware accelerator; compiling, by the compiler, each neural network model of the first set of neural network models for execution by the hardware accelerator to generate a respective compiled data structure for the neural network model; and outputting, by the compiler, for each neural network model of the first set of neural network models, the respective compiled data structure to the hardware accelerator for storage according to the sequence in one or more memories of the hardware accelerator, wherein the outputting comprises: determining that a portion of a compiled data structure in the respective compiled data structures cannot be allocated in the one or more memories of the hardware accelerator; in response, outputting the compiled data structure for storage in another memory external to the one or more memories of the hardware accelerator; generating instructions that, once executed by the hardware accelerator, cause fetching the compiled data structure stored in the other memory; and storing the rest of the respective compiled data structures other than the compiled data structure stored in the other memory according to the sequence in the one or more memories; executing, by the hardware accelerator, the sequence of the first set of neural network models, wherein the executing comprises executing the generated instructions; determining, by the compiler, a next sequence of neural network models to be executed and if the next sequence of neural network models differs from the sequence of the first set of neural network models, erasing the rest of the compiled data structures stored in the first memory; and if the next sequence of neural network models is the sequence of the first set of neural network models, executing the compiled data structures in the one or more memories again on the hardware accelerator.