Microsoft patents AI system that layers images and text to auto-design graphics

1. A data processing system comprising: a processor, and a machine-readable storage medium storing executable instructions which, when executed by the processor, cause the processor alone or in combination with other processors to perform operations: receiving, at a client device, a user request to lay out multimodal design elements in a graphic design; applying a first generative model to categorize each of the multimodal design elements based on content attributes of the multimodal design elements in a plurality of layers that are set in a predetermined order, wherein the layers include an initial layer, and at least two pre-defined layers; applying a second generative model to sequentially process the layers in the predetermined order until rendering a final composition image as the graphic design, by: applying a vision encoder to encode content attributes of each design element of the initial layer into initial content embeddings, applying a projector to project the initial content embeddings to match a hidden state dimension required by a backbone of the second generative model, applying the backbone to predict position and size attributes per design element of the initial layer based on the projected content embeddings, rendering an initial image by placing each design element of the initial layer onto the initial layer based on the position and size attributes predicted for each design element of the initial layer, for each of the pre-defined layers: applying the vision encoder to encode content attributes of each design element of the pre-defined layer and each design element of a precedent image into combined content embeddings, applying the projector to project the combined content embeddings to match the hidden state dimension required by the backbone, applying the backbone to predict position and size attributes per design element of the pre-defined layer based on the combined embeddings, and rendering a composition image by placing each design element of the pre-defined layer onto the pre-defined layer based on the position and size attributes predicted for each design element of the pre-defined layer, and overlaying the placed pre-defined layer upon the precedent image as the composition image, wherein the precedent image starts with the initial image; providing the final composition image to the client device; and causing a user interface of the client device to display the final composition image.