Google patents method for generating audio using neural networks

1. A computer-implemented method for generating an acoustic representation of an audio signal, the method comprising: receiving a request to generate an audio signal having a respective audio sample at each of a plurality of output time steps spanning a time window; obtaining a semantic representation of the audio signal that specifies a respective semantic token at each of a plurality of first time steps spanning the time window; and generating, using one or more generative neural networks and conditioned on at least the semantic representation, the acoustic representation of the audio signal, the acoustic representation specifying a set of one or more respective acoustic tokens at each of a plurality of second time steps spanning the time window, wherein the set of one or more respective acoustic tokens at each of the plurality of second time steps comprises a plurality of acoustic tokens that collectively represent a prediction of an output of a residual vector quantization applied to an embedding that represents acoustic properties of the audio signal at the second time step, the residual vector quantization encodes the embedding using a hierarchy of a plurality of vector quantizers that each generate a respective acoustic token from a corresponding vocabulary of acoustic tokens for the vector quantizer, wherein the hierarchy comprises one or more coarse vector quantizers and one or more fine vector quantizers, and the set of acoustic tokens at each second time step comprises, for each vector quantizer, a respective acoustic token selected from the vocabulary for the vector quantizer.