Microsoft patents method to build 3D models of moving objects from photos
Microsoft has filed a patent for a technique that reconstructs a 3D model of an object with moving parts, like a robot arm or a hand, from a single 2D image. The system uses a vision transformer to identify key points on the object, estimates their 3D positions, and calculates the object's pose to generate a digital mesh.
In the filing’s own words
A method for recovering a three-dimensional mesh model of an articulated object from a two-dimensional image is disclosed. The method includes extracting features from the two-dimensional image by generating image patches and processing them through a vision transformer encoder to produce visual tokens and a classification (CLS) token. Landmarks are predicted by generating landmark heatmaps, determining initial landmark coordinates and confidence scores, and refining these coordinates using the CLS token. The method further involves estimating a three-dimensional pose of the articulated object through landmark-based feature pooling and processing the features through transformer decoder blocks to generate pose parameters. Finally, a three-dimensional mesh of the articulated object is generated using the pose parameters and a parametric model.
What exactly they claim (claim 1)
1. A method for recovering a three-dimensional mesh model of an articulated object from a two-dimensional image, comprising: extracting features from the two-dimensional image by: generating a plurality of image patches from the two-dimensional image; generating a plurality of visual tokens and a classification (CLS) token by processing the plurality of image patches through a vision transformer encoder; predicting landmarks by: generating a plurality of landmark heatmaps using a convolutional decoder applied to the plurality of visual tokens; determining a plurality of initial landmark coordinates and confidence scores from the plurality of landmark heatmaps; refining the plurality of initial landmark coordinates using the CLS token and the plurality of initial landmark coordinates and confidence scores to create a plurality of refined landmark coordinates; estimating a three-dimensional pose of the articulated object by: performing landmark-based feature pooling using the plurality of refined landmark coordinates to generate a plurality of per-landmark features; processing the plurality of per-landmark features and the CLS token through a plurality of transformer decoder blocks; and generating a plurality of pose parameters for the articulated object; and generating a three-dimensional mesh of the articulated object using the pose parameters and a parametric model.
Facts
- Company
- Microsoft (Microsoft Technology Licensing, LLC)
- Official title
- ARTICULATED OBJECT MESH RECOVERY
- Publication number
- US20260260423A1
- Application number
- 19066861
- Filed
- Feb 28, 2025
- Published
- Sep 3, 2026
- Inventors
- 4
- Claims
- 20
Classified under
G06T 17/20 Physics & computingG06T 7/73 Physics & computingG06T 2207/20021 Physics & computingG06T 2207/20084 Physics & computingMore from Microsoft in AI & software
- Microsoft patents method to check AI answers against verified facts Oct 20, 2026
- Microsoft patents AI that presents PowerPoint slides for you Sep 3, 2026
- Microsoft patents method to check if LLM answers are based on real data Sep 10, 2026
- Microsoft patents system to blend multiple AI agents into one chat Oct 1, 2026
- Microsoft patents system that answers questions using your screen history Oct 15, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.