AmazonGranted patentAI & softwareInterest 7/10

Amazon patents tool to animate specific objects in images into video

Patent drawing
Patent drawing from the official filing.

Amazon has been granted a patent for a method that turns a still image into a short video by animating a specific object selected by the user. The system uses a pre-trained AI model to generate video frames that move the object to a target location defined by a bounding box, while keeping the rest of the image consistent.

Why it mattersThis technology could allow creators to easily animate individual elements within a photo, offering more precise control over video generation than simple text-to-video prompts.

In the filing’s own words

Techniques are generally described for controllable image-to-video generation. In various examples, a first image representing at least a first object may be received. First input data including a selection of the first object in the first image for animation may be received. Second input data including at least a first bounding box indicating a target location of the first object may be received. A latent diffusion text-to-image model and the first image may be used to generate a first plurality of visual tokens. One or more first grounding tokens may be generated representing a location of the first bounding box. The latent diffusion text-to-image model may be used to generate a video animating the first object based on the first plurality of visual tokens and the one or more first grounding tokens.

What exactly they claim (claim 1)

1. A computer-implemented method, comprising: receiving a first image representing at least a first object; receiving first input data comprising a selection of the first object in the first image, the selection of the first object indicating that the first object is to be animated along a motion path; receiving second input data comprising at least a first bounding box indicating a target location of the first object; inputting the first image into a pre-trained latent diffusion text-to-image model; generating, using the pre-trained latent diffusion text-to-image model and the first image, a first plurality of visual tokens; generating one or more first grounding tokens representing a location of the first bounding box; generating, using the first plurality of visual tokens and the one or more first grounding tokens, a spatio-temporal mask comprising a region corresponding to a first location of the first object and a region corresponding to the target location; generating first intermediate visual tokens by combining the one or more first grounding tokens and the first plurality of visual tokens using a gated self-attention layer; generating second intermediate visual tokens by combining the first intermediate visual tokens and text prompt tokens using a cross-attention layer; generating output visual tokens by inputting the second intermediate visual tokens into at least one temporal attention layer; and generating a frame of video by denoising the output visual tokens using a denoising process of the pre-trained latent diffusion text-to-image model, wherein the frame of the video comprises a representation of the first object at the target location, wherein the pre-trained latent diffusion text-to-image model comprises a zero-shot image-to-video model configured to animate the first image using user-provided motion layouts and the spatio-temporal mask.

Facts

Company
Amazon (Amazon Technologies, Inc.)
Official title
Controllable image-to-video generation
Patent number
US12784315B2
Application number
18604917
Filed
Mar 14, 2024
Granted
Nov 3, 2026
Inventors
7
Claims
17

Classified under

G06T 13/00 Physics & computingG06F 40/40 Physics & computingG06T 5/60 Physics & computingG06T 5/70 Physics & computingG06T 5/77 Physics & computingG06T 7/73 Physics & computing

More from Amazon in AI & software

Summaries are simplified from the filing’s own abstract and first claim. A patent filing is not a product announcement.
Patent data and drawings: U.S. Patent and Trademark Office (USPTO). Nebula is not affiliated with or endorsed by the USPTO.