Waymo patents AI that predicts driving scenes from sensor data
This filing describes a neural network that converts data from car sensors into a sequence of digital tokens. The system is trained to predict what the driving environment will look like in the future based on current conditions.
In the filing’s own words
Systems and methods for using multi-modal token processing neural networks for autonomous driving. One of the methods includes obtaining an input sequence of tokens characterizing a driving environment for a vehicle, wherein the input sequence of tokens comprises, for each of one or more time steps, a respective set of tokens for each of a plurality of modalities of data from a set of multiple modalities of data; and processing the input sequence of tokens using a multi-modal token processing neural network to generate an output sequence of tokens, wherein: the output sequence of tokens characterizes a prediction for a prediction task for the vehicle, wherein: the multi-modal token processing neural network has been trained to process input token sequences characterizing current states of example driving environments to generate output token sequences characterizing predicted future states of the example driving environments.
What exactly they claim (claim 1)
1. A method performed by one or more computers, the method comprising: pre-training a multi-modal token processing neural network on a next frame prediction task that requires processing input token sequences characterizing current states of example driving environments to generate output token sequences characterizing predicted future states of the example driving environments; and after pre-training the multi-modal token processing neural network: obtaining respective training data for each of a plurality of prediction tasks for an autonomous vehicle; and for each of the plurality of prediction tasks, fine-tuning the multi-modal token processing neural network on the respective training data for the prediction task to generate a respective fine-tuned multi-modal token processing neural network for performing the prediction task.
Facts
- Company
- Google (Waymo LLC)
- Official title
- WORLD MODELS FOR PROCESSING VEHICLE SENSOR DATA
- Publication number
- US20260300725A1
- Application number
- 19633813
- Filed
- Mar 30, 2026
- Published
- Oct 1, 2026
- Inventors
- 2
- Claims
- 20
Classified under
G06N 3/08 Physics & computingMore from Google in Cars & transport
- Waymo patents system to detect cyclists losing control before they fall Sep 8, 2026
- Waymo patents self-driving cars that hear sirens and pull over Sep 8, 2026
- Waymo patents AI that moves self-driving cars out of traffic jams Sep 29, 2026
- Waymo patents method for self-driving cars to predict when they can see hidden hazards Sep 8, 2026
- Waymo patents database of past traffic patterns to test self-driving cars Sep 22, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.