Microsoft patents robot hands that learn to grip new objects from images


Microsoft describes a system that teaches robotic grippers how to handle objects by analyzing how they touch surfaces. The technology uses a library of movement patterns to adapt a known grip to a new object based on a photo, adjusting for the object's specific shape and size.
In the filing’s own words
Techniques for generating a latent codebook and for generating instructions indicating how an action is performed against an object using the latent codebook are disclosed. The latent codebook is created by generating interaction trajectories that occur during an action performed by a gripping structure against an object. The vectors in the latent codebook describe actions by mapping trajectories of contact points between gripping structures and objects during performance of actions. A service then accesses an image of a second object that is to be manipulated. The service obtains an instruction indicating that the specific action is to be performed against the second object. In response to accessing a vector in the latent codebook, the service uses the image to localize the vector to the second object. The service then generates a sequence of instructions that represent the action as applied to the second object.
What exactly they claim (claim 1)
1. A method for generating instructions indicating how an action is performed against an object that was not previously mapped, said method comprising: accessing a latent codebook comprising a vector, wherein the vector describes a specific action that is performed by a first gripping structure against a first object, wherein the vector describes the specific action by mapping a trajectory of contact points between the first gripping structure and the first object during performance of the specific action; accessing an image of a second object that is to be manipulated, wherein the second object is different from the first object; obtaining an instruction indicating that the specific action is to be performed against the second object; in response to accessing the vector in the latent codebook, using the image to localize the vector to the second object, wherein localizing the vector to the second object includes modifying a first dimensionality associated with the first object, as described in the vector, to account for a second dimensionality of the second object, as identified by the image, such that a new trajectory of new contact points between a second gripping structure and the second object are generated; and generating a sequence of instructions that represent the specific action as applied to the second object, wherein the sequence of instructions identifies the new trajectory of the new contact points between the second gripping structure and the second object, such that the sequence of instructions facilitates performance of the specific action against the second object.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- GENERATIVE 3D MODEL OF HAND OBJECT INTERACTIONS
- Publication number
- US20260355959A1
- Application number
- 19200145
- Filed
- May 6, 2025
- Published
- Nov 12, 2026
- Inventors
- 4
- Claims
- 20
Classified under
B25J 9/1671 Operations & transportB25J 9/163 Operations & transportG05B 13/0265 Physics & computingMore from Microsoft in Robots & automation
- Microsoft patents robot that hops between rails to sort warehouse items Oct 27, 2026
- Microsoft patents robot hand technique for learning object manipulation Nov 12, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.