file="US12536720-20260127-P00004.TIF" he="3.22mm" id="CUSTOM-CHARACTER-00004" img-content="character" img-format="tif" wi="2.46mm" />_fake (1)

Following the GAN paradigm, the generator 220 is simultaneously and adversarially trained against discriminator D training. Discriminator D^c(e.g., conditional discriminator 230) is also multimodal conditioned, takes either real or generated layout samples as well as multimodal conditions as input, and outputs real (1) or fake (0) binary classification:

D^c( [Image Omitted] ,B,T∪){0,1} (2)

The conditional discriminator 230 targets to adversarially boost the generator 220 to generate reasonable and realistic layouts constrained by background images and driven by multimodal foreground elements. In some embodiments, in order to further enhance the realism of layout structures regardless of conditions, an unconditional discriminator D^umay also be trained:

D^u( [Image Omitted] ){0,1} (3)

There is an empirical observation that the discriminators tend to be non-sensitive to the positional trends of bounding boxes in a layout. For example, the discriminators lean to overlook the unusual layout where a header texts is placed at the bottom. As a result, position-aware regularization may be added to the conditional and unconditional discriminators. Specifically, an auxiliary decoder may be included for each discriminator to reconstruct its input. For example, auxiliary decoder 232 may be used in conjunction with conditional discriminator 230. The decoders F^c/F^utake the output features f^c/f^uof their discriminators D^c/D^u, add them with learnable positional embeddings

$ε^{c} = {e_{i}^{c}} \begin{matrix} N \\ i = 1 \end{matrix} / ε^{u} = {ε^{u}} \begin{matrix} N \\ i = 1 \end{matrix},$

and reconstruct the bounding box parameters and multimodal conditions that are the input of the discriminators:

F^c(f^c,ε^c) [Image Omitted] _dec^c,B_dec,T_dec∪_dec (4)

F^u(f^u,ε^u)_dec^u (5)

The decoders are jointly trained with the discriminators to minimize the reconstruction.

All together the adversarial learning objective is formulated as:

$\begin{matrix} \begin{matrix} \min \\ G \end{matrix} \begin{matrix} \max \\ D^{c} F^{c} ε^{c} D^{u} F^{u} ε^{u} \end{matrix} \begin{matrix} L_{GAN} \end{matrix} ≐ 𝔼_{z ~ 𝒩 (0, I), {B, T ⋃ 𝒫} ~ P_{data}} & (6) \end{matrix}$

$[\log (1 - D^{c} (ℒ_{fake}, B, T ⋃ 𝒫)) + \log (1 - D^{u} (ℒ_{fake}))] +$

$𝔼_{{ℒ_{real}, B, T ⋃ 𝒫} ~ P_{data}} [\log (D^{c} (ℒ_{real}, B, T ⋃ 𝒫)) + \log (D^{u} (ℒ_{real})) -$

$L_{dec} (F^{c} (f^{c}, ε^{c}), (ℒ_{real}, B, T ⋃ 𝒫)) - L_{layout} (F^{u} (f^{u}, ε^{u}, ℒ_{real}))]$

$\begin{matrix} ℒ_{fake} = G (z, B, T ⋃ 𝒫) & (7) \end{matrix}$

$where$

$\begin{matrix} L_{dec} ((ℒ_{1}, B_{1}, T_{1}, ⋃ 𝒫_{1}), (ℒ_{2}, B_{2}, T_{2}, ⋃ 𝒫_{2})) = λ_{layout} L_{layout} (ℒ_{1}, ℒ_{2}) + λ_{im} ({ B_{1} - B_{2} }_{2} + L_{im} (𝒫_{1,} 𝒫_{2})) + λ_{text} L_{text} (T_{1}, T_{2}) & (8) \end{matrix}$

$\begin{matrix} L_{layout} (ℒ_{1}, ℒ_{2}) = \frac{1}{N} \sum_{i = 1}^{N}  {\hat{b}}_{1}^{i} - {\hat{b}}_{2}^{i}  2 & (9) \end{matrix}$

$\begin{matrix} L_{im} (𝒫_{1,} 𝒫_{2}) = \frac{1}{N} \sum_{i = 1}^{N}  p_{i}^{1} - p_{2}^{i}  2 & (10) \end{matrix}$

$\begin{matrix} L_{text} (T_{1}, T_{2}) = \frac{1}{N} \sum_{i = 1}^{N} (λ_{str} L_{str} (s_{1}, s_{2}) + λ_{cls} L_{cls} (c_{1}, c_{2}) + λ_{len} L_{len} (l_{1}, l_{2})) & (11) \end{matrix}$

L_decis the loss associated with auxiliary decoder 232. L_strL_clsL_lenare the reconstruction losses for foreground text strings, text classes, and text lengths respectively. L_stris formulated as the auto-regressive loss according to BERT language modeling. L_clsis the standard cross-entropy loss for classification. L_lenis also the standard cross-entropy loss for classification, a string length integer scalar is quantized (in the range of [0, 255], 0 indicating a non-text input) into 256 levels. λ_layout, λ_im, λ_text, λ_str, λ_clsand λ_lenare hyper-parameters used to balance each loss term in the same order of magnitude.

For the VAE paradigm, the generator 220 is jointly trained with an encoder 208 that maps from the layout space to the latent noise distribution space. The output of E stands for the mean μ and covariance matrix Σ of a multivariate Gaussian distribution, the sampling of which is used to trigger the generation of G:

$\begin{matrix} E (ℒ) \mapsto μ, Σ & (12) \end{matrix}$

$\begin{matrix} sample (μ, Σ) = z_{0}^{T} \sum^{\frac{1}{2}} z_{0} + μ, z_{0} \sim 𝒩 (0, I) & (13) \end{matrix}$

where Eq. 13 represents the differentiable reparameterization trick in standard VAE pipeline as described in Kingma and Welling, Auto-encoding variational bayes, ICLR, 2013. The conditional VAE objective is to maximize the evidence lower bound (ELBO) of real sample distribution, which is equivalent to minimize the reconstruction loss:

$\begin{matrix} \begin{matrix} \min \\ E, G \end{matrix} L_{VAE} ≐ 𝔼_{{ℒ_{real}, B, T ⋃ 𝒫} ~ P_{data}} [λ_{layout} L_{layout} (ℒ_{fake}, ℒ_{real}) + λ_{kL} KL (E (ℒ_{real})  𝒩 (0, I))] & (14) \end{matrix}$

$\begin{matrix} ℒ_{fake} = G (sample (E (ℒ_{real})) B, T ⋃ 𝒫) & (15) \end{matrix}$

λ_KLis a hyper-parameter used to balance the loss term in the same order of magnitude as other losses.

KL(⋅∥⋅) is the Kullback-Leibler (KL) Divergence used in standard VAE to regularize the encoded latent noise distribution close to a standard multivariate Gaussian. Without losing representativeness, the covariance matrix Σ may be simplified as a diagonal matrix, meaning each of the d dimensions of multivariate Gaussian are non-correlated to each other. As a result, Σ may be rewritten as its diagonal vector σ². Then the closed-form KL divergence is formulated as:

$\begin{matrix} KL (N (μ, σ^{2})  𝒩 (0, I)) = - \frac{1}{2} \sum_{j = 1}^{D} (1 + \log σ_{j}^{2} - σ_{j}^{2} - μ_{j}^{2}) & (16) \end{matrix}$

The VAE and GAN training frameworks are also compatible to each other. In this embodiment the encoder E from VAE, as well as the discriminators D^cand D^ufrom GAN, are trained together along with the training of generator G. Eq. 6 and Eq. 14 can be combined then to obtain:

$\begin{matrix} \begin{matrix} \min \\ E, G \end{matrix} \begin{matrix} \max \\ D^{c} F^{c} ε^{c} D^{u} F^{u} ε^{u} \end{matrix} \begin{matrix} L_{GAN} + L_{VAE} \end{matrix} & (17) \end{matrix}$

Other losses and regularization terms may also contribute to the generated layout quality. First, bounding box supervision may be added. The generalized intersection over union loss gIoU (⋅,⋅) may be used between generated layout and its ground truth L_gIoU( [Image Omitted] _fake, _real) where:

$\begin{matrix} L_{gIoU} (ℒ_{1}, ℒ_{2}) = λ_{gIoU} \frac{1}{N} \sum_{i = 1}^{N} gIoU ({\hat{b}}_{1}^{i} - {\hat{b}}_{2}^{i}) & (18) \end{matrix}$

where λ_gIoUis a hyper-parameter used to balance the loss term in the same order of magnitude as other losses. IoU is invariant to the sizes of bounding boxes while the generalized version is differentiable for training.

Second, reconstructor R (e.g., conditional reconstructor 250) may be included. R takes the last features of G (e.g., generator 220) as input tokens before outputting box parameters,

$ℱ = {f_{i}^{g}} \begin{matrix} N \\ i = 1 \end{matrix}$

and learns to reconstruct [Image Omitted] and :

R()→_rec,T_rec (19)

Jointly with G training, the model may be trained to minimize:

L_rec=λ_imL_im(_rec,_real)+λ_textL_text(T_rec,T_real) (20)

where L_imrefers to Eq. 10 and L_textrefers to Eq. 11.

Third, reasonable layouts typically avoid overlapping with foreground elements. An overlap loss L_overlap=λ_overlapL_overlap( [Image Omitted] _fake) may be computed that discourages overlapping between any pair of bounding boxes in a generated layout. λ_overlapis a hyper-parameter used to balance the loss term in the same order of magnitude as other losses.

Fourth, aesthetically appealing layouts usually maintain one of the six alignment between a pair of adjacent bounding boxes: left, horizontal-center, right, top, vertical-center, and bottom aligned. The misalignment loss (e.g., Eq. 10) L_misalign≐λ_misalignL_misalign( [Image Omitted] _fake) may be leveraged that discourages such misalignment. λ_misalignis a hyper-parameter used to balance the loss term in the same order of magnitude as other losses.

Altogether the most complete combination of training objectives is formulated as follows:

$\begin{matrix} \begin{matrix} \min \\ E, G, R \end{matrix} \begin{matrix} \max \\ D^{c} F^{c} ε^{c} D^{u} F^{u} ε^{u} \end{matrix} \begin{matrix} L_{GAN} + L_{VAE} + L_{gIoU} + L_{rec} + L_{overlap} + L_{misalign} \end{matrix} & (21) \end{matrix}$

The losses (or a subset of the losses) may be used to update parameters of the model including the layout generator 220 via back-propagation.

FIG. 3 illustrates an exemplary layout 300. This exemplary layout is illustrative of different foreground element categories, which may be labeled in a training dataset. Bounding box 302 contains a “header” type of foreground text. Generally a header has larger font size, capital letters, and accent to attract attention. [Inventors: What is meant here by “accent”?]. Bounding box 304 contains a “body” type of foreground text. Generally body text has smaller font size, and detailed information. Bounding box 306 includes a button type of foreground text. Button bounding boxes need to completely and tightly include the entire button. Bounding box 308 contains a foreground product image. The bounding box of an image needs to completely and tightly include all the foreground objects, and cannot go outside the banner canvas. In some embodiments, training layouts are automatically labeled by detecting the foreground elements.

FIG. 4 illustrates examples of separating foreground elements from background images, according to some embodiments described herein. In some embodiments, a training layout (e.g., layout 402a, 402b, or 402c) includes a background and overlaid foreground elements. Using an object detection model, bounding boxes may be generated around each of the foreground objects. Using optical character recognition (OCR) techniques, text elements may be identified and removed from the image. The area behind the removed elements may be filled using inpainting techniques, as shown in inpainting results 404a, 404b, or 404c. The inpainted image may be used as the background image for training purposes. Results of the OCR (e.g., OCR results 406a, 406b, or 406c) may be used to generate training data containing the text information used in training as described in FIGS. 1-2C.

FIGS. 5-6 illustrates exemplary views of a user interface, according to some embodiments described herein. FIG. 5 illustrates a user interface view 500 that allows users to customize their background images and optionally foreground elements: header text, body text, footnote/disclaimer text, button text, as well as button border radius (zero radius means a rectangular button). Text colors, button pad color, and text fonts can be customized. Once users update their background and foreground elements, they are previewed on the right part of the same page, as illustrated. The locations and sizes of foreground elements in the preview are exemplary, and conceptually show what contents are going to be rendered on the background. Clicking the “Next” button advances to the next user interface view in FIG. 6.

FIG. 6 illustrates a user interface view that presents a number of variations of a generated layout. The layout generation model described herein generates one layout design in the backend. That generated layout is further post-processed with certain variations in the following three steps: (1) Randomly and independently jitter the generated box parameters by 20%. (2) By chance either center-align or left-align the boxes horizontally, or skip this step. (3) De-overlap two overlapping boxes by vertically pushing away their centers. These post-processing steps are exemplary, and a number of different combinations or other methods may be used to generate variants.

Afterwards, the system renders foreground elements given the layout bounding boxes. The text font sizes and line breakers are adaptively determined so as to tightly squeeze into the boxes. Considering header texts usually have short strings yet are assigned with large boxes, their font sizes are naturally large enough. In the example of FIG. 6, the user interface showcases six of the rendered results on this page, and allow users to select one or more satisfactory designs. Once a user make their selection(s) and click “Save Selection”, the user interface allows the user to manually customize the size and location of each rendered foreground elements. Once they finish, they click “Save” to save the layout and exit the system.

Computer and Network Environment

FIG. 7 is a simplified diagram illustrating a computing device implementing the layout generation framework, according to one embodiment described herein. As shown in FIG. 7, computing device 700 includes a processor 710 coupled to memory 720. Operation of computing device 700 is controlled by processor 710. And although computing device 700 is shown with only one processor 710, it is understood that processor 710 may be representative of one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs) and/or the like in computing device 700. Computing device 700 may be implemented as a stand-alone subsystem, as a board added to a computing device, and/or as a virtual machine.

Memory 720 may be used to store software executed by computing device 700 and/or one or more data structures used during operation of computing device 700. Memory 720 may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any other medium from which a processor or computer is adapted to read.

Processor 710 and/or memory 720 may be arranged in any suitable physical arrangement. In some embodiments, processor 710 and/or memory 720 may be implemented on a same board, in a same package (e.g., system-in-package), on a same chip (e.g., system-on-chip), and/or the like. In some embodiments, processor 710 and/or memory 720 may include distributed, virtualized, and/or containerized computing resources. Consistent with such embodiments, processor 710 and/or memory 720 may be located in one or more data centers and/or cloud computing facilities.

In some examples, memory 720 may include non-transitory, tangible, machine readable media that includes executable code that when run by one or more processors (e.g., processor 710) may cause the one or more processors to perform the methods described in further detail herein. For example, as shown, memory 720 includes instructions for layout generation module 730 that may be used to implement and/or emulate the systems and models, and/or to implement any of the methods described further herein. A layout generation module 730 may receive input 740 such as an input training data (e.g., annotated layouts with background and foreground elements) or user inputs via the data interface 715 and generate an output 750 which may be a layout (e.g., bounding boxes for each element or a rendered layout) for an advertisement or other media.

The data interface 715 may comprise a communication interface, a user interface (such as a voice input interface, a graphical user interface, and/or the like). For example, the computing device 700 may receive the input 740 (such as a training dataset) from a networked database via a communication interface. Or the computing device 700 may receive the input 740, such as background images, foreground images, text inputs etc., from a user via the user interface.

In some embodiments, the layout generation module 730 is configured to generate a layout based on input elements. The layout generation module 730 may further include a foreground element submodule 731, a background element submodule 732, a composition submodule 733, and/or a training submodule 734. In one embodiment, the layout generation module 730 and its submodules 731 may be implemented by hardware, software and/or a combination thereof. Foreground element submodule 731 may be configured to extract foreground elements from training images, encoder foreground element information, concatenate the encodings, and further encode the concatenated encodings as described herein. Background element submodule 732 may be configured to perform object detection on the background image as described herein. Composition submodule 733 may be configured to compose a generated layout based on the inputs and generated bounding boxes as described herein. Training submodule 734 may be configured to train a layout generator as described herein.

Some examples of computing devices, such as computing device 700 may include non-transitory, tangible, machine readable media that include executable code that when run by one or more processors (e.g., processor 710) may cause the one or more processors to perform the processes of method. Some common forms of machine-readable media that may include the processes of method are, for example, floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any other medium from which a processor or computer is adapted to read.

FIG. 8 is a simplified block diagram of a networked system suitable for implementing the layout generation framework described herein. In one embodiment, block diagram 800 shows a system including the user device 810 which may be operated by user 840, data vendor servers 845, 870 and 880, server 830, and other forms of devices, servers, and/or software components that operate to perform various methodologies in accordance with the described embodiments. Exemplary devices and servers may include device, stand-alone, and enterprise-class servers which may be similar to the computing device 700 described in FIG. 7, operating an OS such as a MICROSOFT® OS, a UNIX® OS, a LINUX® OS, or other suitable device and/or server-based OS. It can be appreciated that the devices and/or servers illustrated in FIG. 8 may be deployed in other ways and that the operations performed, and/or the services provided by such devices and/or servers may be combined or separated for a given embodiment and may be performed by a greater number or fewer number of devices and/or servers. One or more devices and/or servers may be operated and/or maintained by the same or different entities.

The user device 810, data vendor servers 845, 870 and 880, and the server 830 may communicate with each other over a network 860. User device 810 may be utilized by a user 840 (e.g., a driver, a system admin, etc.) to access the various features available for user device 810, which may include processes and/or applications associated with the server 830 to receive an output data anomaly report.

User device 810, data vendor server 845, and the server 830 may each include one or more processors, memories, and other appropriate components for executing instructions such as program code and/or data stored on one or more computer readable mediums to implement the various applications, data, and steps described herein. For example, such instructions may be stored in one or more computer readable media such as memories or data storage devices internal and/or external to various components of system 800, and/or accessible over network 860.

User device 810 may be implemented as a communication device that may utilize appropriate hardware and software configured for wired and/or wireless communication with data vendor server 845 and/or the server 830. For example, in one embodiment, user device 810 may be implemented as an autonomous driving vehicle, a personal computer (PC), a smart phone, laptop/tablet computer, wristwatch with appropriate computer hardware resources, eyeglasses with appropriate computer hardware (e.g., GOOGLE GLASS®), other type of wearable computing device, implantable communication devices, and/or other types of computing devices capable of transmitting and/or receiving data, such as an IPAD® from APPLE®. Although only one communication device is shown, a plurality of communication devices may function similarly.

User device 810 of FIG. 8 contains a user interface (UI) application 812, and/or other applications 816, which may correspond to executable processes, procedures, and/or applications with associated hardware. For example, the user device 810 may receive a message indicating a generated layout such as an ad banner from the server 830 and display the message via the UI application 812. In other embodiments, user device 810 may include additional or different modules having specialized hardware and/or software as required.

In various embodiments, user device 810 includes other applications 816 as may be desired in particular embodiments to provide features to user device 810. For example, other applications 816 may include security applications for implementing client-side security features, programmatic client applications for interfacing with appropriate application programming interfaces (APIs) over network 860, or other types of applications. Other applications 816 may also include communication applications, such as email, texting, voice, social networking, and IM applications that allow a user to send and receive emails, calls, texts, and other notifications through network 860. For example, the other application 816 may be an email or instant messaging application that receives a prediction result message from the server 830. Other applications 816 may include device interfaces and other display modules that may receive input and/or output information. For example, other applications 816 may contain software programs for asset management, executable by a processor, including a graphical user interface (GUI) configured to provide an interface to the user 840 to view a generated layout such as an ad banner.

User device 810 may further include database 818 stored in a transitory and/or non-transitory memory of user device 810, which may store various applications and data and be utilized during execution of various modules of user device 810. Database 818 may store user profile relating to the user 840, predictions previously viewed or saved by the user 840, historical data received from the server 830, and/or the like. In some embodiments, database 818 may be local to user device 810. However, in other embodiments, database 818 may be external to user device 810 and accessible by user device 810, including cloud storage systems and/or databases that are accessible over network 860.

User device 810 includes at least one network interface component 817 adapted to communicate with data vendor server 845 and/or the server 830. In various embodiments, network interface component 817 may include a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, a broadband device, a satellite device and/or various other types of wired and/or wireless network communication devices including microwave, radio frequency, infrared, Bluetooth, and near field communication devices.

Data vendor server 845 may correspond to a server that hosts database 819 to provide training datasets including sample layout designs to the server 830. The database 819 may be implemented by one or more relational database, distributed databases, cloud databases, and/or the like.

The data vendor server 845 includes at least one network interface component 826 adapted to communicate with user device 810 and/or the server 830. In various embodiments, network interface component 826 may include a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, a broadband device, a satellite device and/or various other types of wired and/or wireless network communication devices including microwave, radio frequency, infrared, Bluetooth, and near field communication devices. For example, in one implementation, the data vendor server 845 may send asset information from the database 819, via the network interface 826, to the server 830.

The server 830 may be housed with the layout generation module 730 and its submodules described in FIG. 1. In some implementations, layout generation module 730 may receive data from database 819 at the data vendor server 845 via the network 860 to generate a layout. The generated layout may also be sent to the user device 810 for review by the user 840 via the network 860.

The database 832 may be stored in a transitory and/or non-transitory memory of the server 830. In one implementation, the database 832 may store data obtained from the data vendor server 845. In one implementation, the database 832 may store parameters of the layout generation module 730. In one implementation, the database 832 may store previously generated layout, and the corresponding input feature vectors.

In some embodiments, database 832 may be local to the server 830. However, in other embodiments, database 832 may be external to the server 830 and accessible by the server 830, including cloud storage systems and/or databases that are accessible over network 860.

The server 830 includes at least one network interface component 833 adapted to communicate with user device 810 and/or data vendor servers 845, 870 or 880 over network 860. In various embodiments, network interface component 833 may comprise a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, a broadband device, a satellite device and/or various other types of wired and/or wireless network communication devices including microwave, radio frequency (RF), and infrared (IR) communication devices.

Network 860 may be implemented as a single network or a combination of multiple networks. For example, in various embodiments, network 860 may include the Internet or one or more intranets, landline networks, wireless networks, and/or other appropriate types of networks. Thus, network 860 may correspond to small scale communication networks, such as a private or local area network, or a larger scale network, such as a wide area network or the Internet, accessible by the various components of system 800.

Example Work Flows

FIG. 9 is an example logic flow diagram illustrating a method of layout generation based on the framework shown in FIGS. 1-8, according to some embodiments described herein. One or more of the processes of method 900 may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors may cause the one or more processors to perform one or more of the processes. In some embodiments, method 900 corresponds to the operation of the layout generation module 730 (e.g., FIGS. 7-8) that generates layouts based on input elements.

As illustrated, the method 900 includes a number of enumerated steps, but aspects of the method 900 may include additional steps before, after, and in between the enumerated steps. In some aspects, one or more of the enumerated steps may be omitted or performed in a different order.

At step 901, a system receives, via a data interface, a background image and a plurality of multimodal foreground elements including at least an image and a text. During training the information may be from a training dataset. At inference, the data may be inputs from a user interface.