Google patents neural networks that shrink memory to save space

1. A method performed by one or more computers, the method comprising: obtaining a network input; and processing the network input to generate an output sequence of tokens that comprises a respective output token at each of a plurality of output positions, the processing comprising, for each of the output positions: processing the output token at the preceding output position using a generative neural network conditioned on the network input to generate the output token at the output position, wherein the generative neural network comprises a compressive attention layer, and wherein the processing comprises, for the compressive attention layer: identifying a memory state for the compressive attention layer as of the output position; obtaining a layer input token for the compressive attention layer that is generated from the output token at the preceding output position; generating, from the layer input token for the compressive attention layer, a query vector, a key vector, and a value vector; generating, from the layer input token for the compressive attention layer, a target vector; performing a set of one or more optimization steps on a compression objective using the key vector, the value vector, and the target vector to update the memory state for the compressive attention layer; and after updating the memory state for the compressive attention layer, applying the memory state to the query vector to generate a layer output token for the compressive attention layer.