Google patents robot brain that plans moves from camera views

1. A method implemented by one or more processors, the method comprising: training an environment-conditioned action sequence prediction model to process an instance of vision data captured by a vision component of an agent to generate a set of predicted actions for performing a task and a particular order for performing the predicted actions of the set, wherein training the environment-conditioned action sequence prediction model comprises: selecting a training sequence of task vision data of the agent performing the task, where each instance of task vision data has a corresponding label indicating an action in the training sequence; processing the selected training sequence using a portion of the environment-conditioned action sequence prediction model to generate an output sequence of embeddings; selecting a given embedding in the output sequence of embeddings and a corresponding label from the training sequence; processing the selected given embedding using an additional portion of the environment-conditioned action sequence prediction model to generate predicted action output; updating one or more portions of the environment-conditioned action sequence prediction model based on comparing the predicted action output and the corresponding action label; subsequent to training the environment-conditioned action sequence prediction model: processing a given instance of vision data captured by the vision component of the agent using the environment-conditioned action sequence prediction model to generate given output; and causing the agent to perform one or more actions based on the given output.