/Patent Tracker
GooglePatent applicationRobots & automationInterest 8/10

Google patents robot control via natural language instructions

Patent drawing
Patent drawing from the official filing.

Google has filed a patent for a system that lets robots understand and execute tasks described in plain text. The method converts both the text instructions and visual observations of the environment into a shared digital format, allowing a language model to generate step-by-step commands for the robot.

Why it mattersThis approach could make it easier for people to direct robots using everyday language rather than complex programming codes.

In the filing’s own words

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for controlling an agent interacting with an environment. In one aspect, a method comprises: receiving one or more observations of an environment; receiving an input text sequence that describes a task to be performed by a robot in the environment; generating an encoded representation of the input text sequence in an embedding space; generating a corresponding encoded representation of each of the one or more observations in the embedding space; generating a sequence of input tokens that comprises the encoded representation of the input text sequence and the corresponding encoded representation of each observation; processing the sequence of input tokens using a language model neural network to generate an output text sequence that comprises high-level natural language instructions; and determining, from the high-level natural language instructions, one or more actions to be performed by the robot.

What exactly they claim (claim 1)

1. A computer-implemented method comprising: receiving one or more observations of an environment that comprises one or more objects; receiving an input text sequence that describes a task to be performed by a robot in the environment and that references the one or more objects in the environment; generating an encoded representation of the input text sequence in an embedding space; generating a corresponding encoded representation of each of the one or more observations in the embedding space; generating a sequence of input tokens that comprises the encoded representation of the input text sequence and the corresponding encoded representation of each observation; processing the sequence of input tokens using a language model neural network to autoregressively generate an output text sequence that comprises high-level natural language instructions for the robot; and determining, from the high-level natural language instructions, one or more actions to be performed by the robot.

Facts

Company
Google (Google LLC)
Official title
CONTROLLING ROBOTS USING MULTI-MODAL LANGUAGE MODELS
Publication number
US20260307248A1
Application number
19540472
Filed
Feb 13, 2026
Published
Oct 8, 2026
Inventors
6
Claims
1

Classified under

B25J 9/1658 Operations & transport
Read the full filing (official PDF) ↗ Open in the Patent Tracker

More from Google in Robots & automation

Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.