Microsoft patents robot hand technique for learning object manipulation
Microsoft describes a method for teaching robots how to handle objects by analyzing video data of a hand or gripper performing an action. The system creates a digital library of movement patterns and contact points, which it then uses to generate specific instructions for manipulating new objects based on their images.
In the filing’s own words
Techniques for generating a latent codebook and for generating instructions indicating how an action is performed against an object using the latent codebook are disclosed. The latent codebook is created by generating interaction trajectories that occur during an action performed by a gripping structure against an object. The vectors in the latent codebook describe actions by mapping trajectories of contact points between gripping structures and objects during performance of actions. A service then accesses an image of a second object that is to be manipulated. The service obtains an instruction indicating that the specific action is to be performed against the second object. In response to accessing a vector in the latent codebook, the service uses the image to localize the vector to the second object. The service then generates a sequence of instructions that represent the action as applied to the second object.
What exactly they claim (claim 1)
1. A method for generating a latent codebook of a vector describing an action performed against an object by translating video data to vector data, resulting in an inference of contact points between a gripping structure and the object, the contact points facilitating performance of the action, said method comprising: accessing a recording in which a gripping structure is performing an action against an object, wherein the recording includes time-series data illustrating performance of the action by the gripping structure over a period of time; from the recording, generating interaction trajectories of the gripping structure performing the action by: generating a sequence of three dimensional ( 3 D) poses of the gripping structure while the gripping structure is performing the action; and for each 3 D pose included in the sequence of 3 D poses, determining a corresponding set of contact points between the gripping structure and the object during performance of the action; generating embeddings from the interaction trajectories; and including the embeddings in a latent codebook.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- GENERATIVE 3D MODEL OF HAND OBJECT INTERACTIONS
- Publication number
- US20260355960A1
- Application number
- 19201327
- Filed
- May 7, 2025
- Published
- Nov 12, 2026
- Inventors
- 4
- Claims
- 20
Classified under
B25J 9/1671 Operations & transportB25J 9/163 Operations & transportG05B 13/0265 Physics & computingMore from Microsoft in Robots & automation
- Microsoft patents robot hands that learn to grip new objects from images Nov 12, 2026
- Microsoft patents robot that hops between rails to sort warehouse items Oct 27, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.
