by combining the driving latent code z_dwith the audio latent code z_a, and mapping the combined code to a multimodal motion space by a multimodal motion encoder E_m. In some embodiments, the multimodal motion encoder E_mmay include three multilayer perceptrons.

In some embodiments, the latent fusion network 130 may acquire the motion code z_c→daccording to Equation 1 below in order for the combined code to include only a motion feature when combining the driving latent code z_dwith the audio latent code z_a.

z

c
→
d

=

E
m

(

E
a

(

x
a

)

⊕

E
inv

(

x
d

)

)

(

Equation
⁢

1

)

Here, z_c→dis the motion code, E_mis the multimodal motion encoder, E_ais the audio encoder, E_invis the visual encoder, x_ais the input audio, x_drepresents the series of driving images, and ⊕ represents channel-wise concatenation.

Meanwhile, the latent fusion network 130 may acquire the multimodal fused latent code z_s→dby combining the canonical code z_s→cwith the motion code z_c→d. In some embodiments, the latent fusion network 130 may acquire the multimodal fused latent code z_s→dby combining the canonical code z_s→cwith the motion code z_c→dby a linear operation.

Alternatively, in some embodiments, the latent fusion network 130 may acquire the multimodal fused latent code z_s→dby 2-stage motion transfer according to Equation 2 below. Directly exploring the multimodal fused latent code z_s→din the latent space is very difficult because the direct exploration is only possible when a model represents a detailed facial attribute while capturing a complex distribution of the motion. To solve these problems, it is possible to acquire the multimodal fused latent code z_s→dthrough the 2-stage motion transfer by assuming the existence of a canonical image xc in the canonical space, which is the latent space that has unified motions related to the face and an individual identity.

z

s
→
d

=

z

s
→
c

+

z

c
→
d

(

Equation
⁢

2

)

Here, z_s→dis the multimodal fused latent code, z_s→cis the canonical code and represents conversion from the source image x_sto the canonical image, and z_c→dis the motion code and represents conversion from the canonical image to the series of driving images x_d.

The latent fusion network 130 may impose the constraint to ensure the orthogonality in which an identity feature and the motion feature are distinguished from each other between the canonical code z_s→cand the motion code z_c→d. Imposing the constraint in this way may be important for generating the natural-looking talking face through the fully-controllable facial attributes and the accurate lip synchronization. The face image generation device 10 may impose the orthogonality constraint on a correlation between the canonical space that only includes a person-specific characteristic of a style latent space and the multimodal motion space that includes person-agnostic motion feature encoded from a driving pose video and an audio source to explicitly separate the canonical space and the multimodal motion space from each other, thereby generating a detailed and controllable facial animation without mixing the motion and audio characteristics.

In some embodiments, the latent fusion network 130 may introduce orthogonality loss L_orthoaccording to Equation 3 below to ensure the orthogonality.

L
ortho

=

1
N

⁢

∑

(

z

s
→
c

⊙

z

c
→
d

)

(

Equation
⁢

3

)

Here, N is the number of channels of the latent code, z_s→cis the canonical code, z_c→dis the motion code, and ⊙ represents Hadamard product.

In some embodiments, the orthogonality loss L_orthomay be introduced by being included in an overall loss in the form of combining other losses with the orthogonality loss L_ortho, as shown in Equation 4 below.

L
total

=

λ
1

⁢

L
ortho

+

λ
2

⁢

L
sync

+

λ
3

⁢

L
id

+

λ
4

⁢

L

L
⁢
1

+

λ
5

⁢

L
LPIPS

+

λ
6

⁢

L
GAN

(

Equation
⁢

4

)

Here, L_totalis the overall loss, L_syncis synchronization loss, L_idis identity loss, L_L1is reconstruction loss, L_LPIPSis perceptual loss, and L_GANis adversarial loss, λ may be a hyperparameter introduced to balance scales of the respective losses. Each λ may control relative importance of its corresponding loss term. In some embodiments, λ₁, λ₂, λ₃, λ₄, λ₅, and λ₆may respectively be set to 1, 0.1, 0.5, 1, 1, and 0.1.

The multimodal fused latent code z_s→dmay be transferred to the GAN 140 to thus generate the talking face image. The GAN 140 may include a generator G, a discriminator D and a pre-trained encoder, and the generator G may generate generated data based on the multimodal fused latent code z_s→d, and perform learning until the generated data is identified as real data by the discriminator D. The discriminator D may receive original data and the generated data provided by the generator G, and discriminate whether the generated data is the real data or fake data. The discriminator D may perform the learning until the discriminator D has higher authenticity discrimination accuracy of the generated data. After completing its learning, the generator G may generate a video x_gincluding the talking face image.

In some embodiments, the multimodal fused latent code z_s→dmay be refined through a temporal fusion layer and then transferred to the GAN 140. Refining here may indicate fitting a dimension or extracting a useful feature. In some embodiments, the temporal fusion layer may include a one-dimensional convolution layer.

The device 10 according to this embodiment is designed to pursue the canonical space for separating a visual feature from the person-specific characteristic through latent space exploration to thus generate all facial expressions while synchronizing a mouth movement with an input audio source, and may generate more detailed and controllable facial animation while preventing unwanted mixing of different types of conditions. In addition, the device 10 may be particularly useful for providing personalized digital avatar or virtual assistant that may communicate and interact with a user in a more natural and realistic way.

FIGS. 2, 3, 4, and 5 are diagrams for explaining the face image generation device according to an embodiment.

Referring to FIG. 2, the face image generation device according to an embodiment may accurately reflect all the facial expressions of a motion source and synchronize a lip shape with the input audio source. To this end, the face image generation device may find a standard space where all faces have the same motion pattern and different identities.

Referring to FIG. 3, the face image generation device according to an embodiment may transfer an individual portrait movement through the latent space exploration. An entire pipeline may largely include two stages, and a first stage may be a stage of transmitting a style latent space code to a canonical latent space. Here, all the faces may have the same facial attributes such as the same lip shape, head posture, and eye blinking. A second stage may be a stage of exploring the multimodal motion space where each code includes only motion information and excludes identity information. A target motion code may be acquired by fusing an image latent feature and an audio latent feature. The canonical code and the motion code may then be combined with each other by the linear operation to ultimately generate the multimodal fused latent code. The fused latent code may be input to a decoder and used to synthesize an attribute-controllable talking face video.

Referring to FIG. 4, the face image generation device according to an embodiment may include the visual encoder E_inv, the audio encoder E_a, the generator G, the discriminator D, and the latent fusion network. In this embodiment, for brevity, one visual encoder E_invmay be used to extract the source latent code z_sand the driving latent code z_d. The audio encoder E_amay extract the audio latent code z_a.

The canonical encoder E_canmay map the source latent code z_sto the canonical space and generate the canonical code z_s→c, and the canonical code z_s→cmay be linearly combined with the motion code z_c→d. The motion code z_c→dmay be generated by the multimodal motion encoder E_m, which combines the driving latent code z_dwith the audio latent code z_a.

To generate a natural movement, the multimodal fused latent code z_s→dmay pass through the temporal fusion layer before being transferred to the generator G. The generated video x_gmay be compared with the driving latent code z_din terms of visual and synchronization quality. In this embodiment, a StyleGAN2 generator is used as the decoder to control coarse-fine motion transfer, and L represents the number of modulation layers in the generator. Accordingly, the canonical encoder E_canand the multimodal motion encoder E_mof the latent fusion network may each have L independent weights.

Referring to FIG. 5, it may be seen that the orthogonality is ensured in the combination of the canonical code z_s→cand the motion code z_c→d. Accordingly, the canonical code, which includes the person-specific characteristics such as a face shape and texture, may be independent of a multimodal movement code, which encodes the target movements such as the head pose, the facial expression, and gaze. As described above, it is possible to apply the orthogonality between the two codes to thus generate the more controllable facial animation while preventing the unwanted mixing of the different types of conditions.

FIG. 6 is a flowchart showing a face image generation method according to an embodiment.

Referring to FIG. 6, the face image generation method according to an embodiment may include: receiving a source image and a series of driving images, sampled from the same video, and an input audio (S601); acquiring a source latent code and a driving latent code by encoding the source image and the series of driving images into a visual space (S