Waymo patents neural network to predict driving scenes from sensor data
This filing describes a system that converts data from various vehicle sensors into a sequence of digital tokens. A neural network processes these tokens to predict how the driving environment will change in the future.
In the filing’s own words
Systems and methods for using multi-modal token processing neural networks for autonomous driving. One of the methods includes obtaining an input sequence of tokens characterizing a driving environment for a vehicle, wherein the input sequence of tokens comprises, for each of one or more time steps, a respective set of tokens for each of a plurality of modalities of data from a set of multiple modalities of data; and processing the input sequence of tokens using a multi-modal token processing neural network to generate an output sequence of tokens, wherein: the output sequence of tokens characterizes a prediction for a prediction task for the vehicle, wherein: the multi-modal token processing neural network has been trained to process input token sequences characterizing current states of example driving environments to generate output token sequences characterizing predicted future states of the example driving environments.
What exactly they claim (claim 1)
1. A method performed by one or more computers, comprising: obtaining an input sequence of tokens characterizing a driving environment for a vehicle, wherein the input sequence of tokens comprises, for each of one or more time steps, a respective set of tokens for each of a plurality of modalities of data from a set of multiple modalities of data; and processing the input sequence of tokens using a multi-modal token processing neural network to generate an output sequence of tokens, wherein: the output sequence of tokens characterizes a prediction for a prediction task for the vehicle, and the multi-modal token processing neural network has been trained to process input token sequences characterizing current states of example driving environments to generate output token sequences characterizing predicted future states of the example driving environments.
Facts
- Company
- Google (Waymo LLC)
- Official title
- WORLD MODELS FOR PROCESSING VEHICLE SENSOR DATA
- Publication number
- US20260296454A1
- Application number
- 19633764
- Filed
- Mar 30, 2026
- Published
- Oct 1, 2026
- Inventors
- 2
- Claims
- 20
Classified under
B60W 50/0097 Operations & transportB60W 2420/403 Operations & transportB60W 2420/408 Operations & transportB60W 2420/54 Operations & transportMore from Google in AI & software
- Google patents video search that answers questions using AI descriptions Sep 15, 2026
- Google patents photo tool that removes people and their gear Sep 15, 2026
- Google patents AI that writes live meeting notes and summaries Sep 15, 2026
- Google patents AI tool that turns messy emergency notes into clear messages Sep 3, 2026
- Google patents AI that clicks through apps to finish tasks for you Sep 24, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.