Sony patents method to turn 2D videos into high-detail 3D animations
Sony describes a system that takes a standard 2D video and uses it to create a detailed 3D animation. It works by first building a basic 3D model from the video, then matching features to transfer the movement onto a higher-resolution 3D model.
In the filing’s own words
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for reconstructing high-fidelity three-dimensional (“3D”) animations from two-dimensional (“2D”) videos. Examples of an animation reconstruction system are described herein for performing high-fidelity 2D-to-3D animation reconstruction. The animation reconstruction system includes a shape reconstruction module, a feature matching module, and a deformation transfer module each configured to perform a respective shape reconstruction, feature matching, or deformation transfer task in the 2D-to-3D animation reconstruction process. Some examples of the shape reconstruction, feature matching, and deformation transfer modules utilize machine learning models for automatically performing their respective task. Examples of training systems for training such machine learning models are also described.
What exactly they claim (claim 1)
1. A method performed by one or more computers, the method comprising: receiving a source video comprising a sequence of frames depicting a motion sequence; processing the source video to generate a source animation comprising a sequence of states of a three-dimensional source mesh performing the motion sequence; obtaining a set of three-dimensional paired correspondences between: (i) the three-dimensional source mesh, and (ii) a three-dimensional target mesh, wherein the three-dimensional target mesh has higher resolution than the three-dimensional source mesh; and processing the source animation, in accordance with the set of three-dimensional paired correspondences, to generate a target animation comprising a sequence of states of the three-dimensional target mesh performing the motion sequence.
Facts
- Company
- Sony (Sony Interactive Entertainment Inc.)
- Official title
- RECONSTRUCTING HIGH FIDELITY THREE-DIMENSIONAL ANIMATIONS FROM TWO-DIMENSIONAL VIDEOS
- Publication number
- US20260320891A1
- Application number
- 19180044
- Filed
- Apr 15, 2025
- Published
- Oct 15, 2026
- Inventors
- 1
- Claims
- 20
Classified under
G06T 13/40 Physics & computingG06T 17/20 Physics & computingG06V 10/82 Physics & computingG06V 40/174 Physics & computingMore from Sony in AI & software
- Sony patents AI that plays video games in your style Oct 6, 2026
- Sony patents AI that swaps game visuals for accessibility formats Sep 17, 2026
- Sony patents AI that changes game characters based on your play Oct 1, 2026
- Sony patents AI bots that mimic your friends' gaming styles Sep 8, 2026
- Sony patents eye-tracking controls that link eye shapes to computer actions Sep 8, 2026
Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.