/Patent Tracker
NvidiaPatent applicationRobots & automationInterest 8/10

Nvidia patents robot control using language models to predict 3D positions

Nvidia describes a system that uses a large language model to help robots understand tasks from text instructions and camera images. The model predicts where the robot's arm should move by generating images that show the likely future position of the robot in 3D space.

Why it mattersThis approach could make robots easier to program by allowing them to follow simple text commands rather than complex code.

In the filing’s own words

Apparatuses, systems, and techniques are disclosed for controlling a robot to execute a task. In at least one embodiment, one or more view images, each corresponding to a predefined view of a set of predefined views, are obtained based on a plurality of input images representing observations of a scene. Based on the encoded input, a large language model (LLM) is utilized to predict a plurality of action tokens corresponding to a future state of the robot. The plurality of action tokens are decoded to generate one or more output images, each corresponding to a predefined view. The one or more output images indicate probability distributions of a location of the robot or an end-effector of the robot in a future state of the robot. A three-dimensional (3D) position corresponding to the future state of the robot is determined based on the probability distributions.

What exactly they claim (claim 1)

1. A computer-implemented method for controlling a robot to execute a task, comprising: generating, based on one or more view images and a text instruction describing the task, a plurality of input tokens, wherein each of the one or more view images corresponds to a predefined view of a set of predefined views; generating, by a large language model (LLM) and based on the plurality of input tokens, a plurality of action tokens; decoding the plurality of action tokens to generate one or more output images each corresponding to a predefined view of a set of predefined views, wherein the one or more output images indicate probability distributions of a location of the robot or an end-effector of the robot in a future state of the robot; and determining, based on the probability distributions of the location of the robot or the end-effector in the future state of the robot, a three-dimensional (3D) position corresponding to the future state of the robot.

Facts

Company
Nvidia (NVIDIA Corporation)
Official title
ROBOTIC-VIEW 3D VISION LANGUAGE ACTION MODELING VIA MULTI-VIEW HEATMAP GENERATION
Publication number
US20260268120A1
Application number
19069662
Filed
Mar 4, 2025
Published
Sep 10, 2026
Inventors
6
Claims
24

Classified under

G06N 3/047 Physics & computingB25J 9/163 Operations & transportB25J 9/1697 Operations & transportG06N 3/096 Physics & computing
Read the full filing (official PDF) ↗ Open in the Patent Tracker

More from Nvidia in Robots & automation

Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.