/Patent Tracker
SonyGranted patentCameras & imagingInterest 6/10

Sony patents system to blend live video with 3D models from multiple cameras

Patent drawing
Patent drawing from the official filing.

Sony describes a system that uses multiple cameras to create a 3D model of a subject, allowing users to view it from any angle. It combines this digital model with live footage and sensor data to produce a final image.

Why it mattersThis technology could improve how video games or virtual reality experiences mix real-world footage with digital characters.

In the filing’s own words

The present technology relates to an image production system that includes an estimation information generation unit that generates estimation information regarding a subject on the basis of at least one of a captured image or sensor information, a free viewpoint image generation unit, a three-dimensional image generation unit, and an output image generation unit. The free viewpoint image generation unit generates a three-dimensional model of a subject based on a plurality of pieces of captured image data obtained by simultaneously capturing images from a plurality of viewpoints, and generates a free viewpoint image that is an image of arbitrary viewpoint for the subject using the three-dimensional model. The three-dimensional image generation unit generates a three-dimensional image based on the basis of the estimation information and the three-dimensional model of the subject. The output image generation unit generates an output image on the basis of the free viewpoint image and the three-dimensional image.

What exactly they claim (claim 1)

1. An image production system, comprising: at least one processor configured to: generate estimation information for a subject based on at least one of a captured first image or sensor information; obtain, based on a simultaneous capture of a plurality of second images from a plurality of viewpoints, a plurality of pieces of captured image data; generate a first three-dimensional model of the subject based on the plurality of pieces of captured image data; generate a free viewpoint image based on the first three-dimensional model, wherein the free viewpoint image is an image of an arbitrary viewpoint for the subject; generate a three-dimensional image based on the estimation information and the first three-dimensional model of the subject; and generate an output image by selectively using the three-dimensional image and a live-action image including the free viewpoint image.

Facts

Company
Sony (SONY GROUP CORPORATION)
Official title
Image production system and image production method to generate an output image based on a free viewpoint image
Patent number
US12762786B2
Application number
18713224
Filed
Nov 21, 2022
Granted
Oct 13, 2026
Inventors
2
Claims
15

Classified under

G06T 15/10 Physics & computingG06T 17/20 Physics & computing
Read the full filing (official PDF) ↗ Open in the Patent Tracker

More from Sony in Cameras & imaging

Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.