Google patents new neural network structure for machine learning
Google describes a specific architecture for attention neural networks, which are computer programs that mimic how humans focus on relevant parts of data. The design stacks multiple blocks, where each block uses a 'query' mechanism to decide which information from previous steps is most important to keep.
In the filing’s own words
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing a machine learning task on a network input to generate a network output using an attention neural network. The attention neural network includes a plurality of attention blocks arranged in a stack one after the other. One or more of the plurality of attention blocks are generalized residual attention blocks. Each generalized residual attention block includes one or more generalized residual blocks and an attention layer. The one or more generalized residual blocks includes a query generalized residual block.
What exactly they claim (claim 1)
1. A system for performing a machine learning task on a network input to generate a network output, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement: an attention neural network configured to perform the machine learning task, the attention neural network comprising a plurality of attention blocks arranged in a stack one after the other, wherein one or more of the plurality of attention blocks are generalized residual attention blocks, each generalized residual attention block comprising one or more generalized residual blocks and an attention layer, the one or more generalized residual blocks comprising a query generalized residual block, each generalized residual attention block in the plurality of attention blocks configured to: receive, by the generalized residual attention block, a block input that comprises a respective block output generated by each of two or more preceding attention blocks that are arranged preceding to the generalized residual attention block in the stack of the plurality of attention blocks; generate, by using the query generalized residual block, a query vector based on processing the block input in accordance with one or more sets of weights; and generate, by using the attention layer, an attended block input based at least on applying an attention mechanism over the block input using at least the query vector.
Facts
- Company
- Google (Google LLC)
- Official title
- ATTENTION NEURAL NETWORKS WITH GENERALIZED RESIDUAL ATTENTION BLOCKS
- Publication number
- US20260330219A1
- Application number
- 19468910
- Filed
- Feb 3, 2026
- Published
- Oct 22, 2026
- Inventors
- 6
- Claims
- 20
Classified under
G06N 3/08 Physics & computingG06N 3/048 Physics & computingMore from Google in AI & software
- Google patents video search that answers questions using AI descriptions Sep 15, 2026
- Google patents photo tool that removes people and their gear Sep 15, 2026
- Google patents AI that writes live meeting notes and summaries Sep 15, 2026
- Google patents AI tool that turns messy emergency notes into clear messages Sep 3, 2026
- Google patents AI that clicks through apps to finish tasks for you Sep 24, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.