The low resolution can be, for example, 16×16 pixels, and the high resolution can be, for example, 256×256 pixels.

By using the model architecture, detailed attributes of the face in the high-resolution face-shifted image can be kept consistent with detailed attributes of the face in the low-resolution source image, thereby improving a degree of similarity between the detailed attributes of the high-resolution face-shifted image and the detailed attributes of the low-resolution source image. In addition, there is no obvious abnormal texture on a boundary between a face area and a non-face area in the high-resolution face-shifted image. As such, it cannot be identified by naked eyes that the high-resolution face-shifted image is forged.

The first face can be a victim, and the source image can be a face image of the victim. The second face can be an attacker, and the target image can be a face image of the attacker. The detailed attributes of the face in the high-resolution face-shifted image are kept consistent with the detailed attributes of the face in the low-resolution source image, so that the high-resolution face-shifted image has a relatively strong attack force. The high-resolution face-shifted image can be used to defend against attacks on the face recognition model. For example, by using the high-resolution face-shifted image, adversarial training can be performed on the face recognition model to improve robustness of the face recognition model. The face recognition model can include a model used to identify an identity in an application scenario such as face-scanning payment or access control recognition.

In this scenario example, the image fusion model can include the identity recognition model, the attribute recognition model, the context recognition model, and the feature fusion model. The high-resolution source image can be input into the identity recognition model to obtain the identity feature, and the high-resolution source image can be input into the attribute recognition model to obtain the attribute feature. The attribute feature is used to represent the attribute information of the first face. The attribute feature can include the beard feature, the pouch feature, the gender feature, and the age feature.

The high-resolution target image can be input into the context recognition model to obtain the context feature. The context feature can include the posture feature and the background feature. The posture feature is used to represent a posture of the second face, for example, a deflection angle of the second face. The background feature is used to represent a background of the second face, for example, a scene used to bring out the second face.

The identity feature, the attribute feature, the context feature, and the intermediate feature can be input into the feature fusion model to obtain the high-resolution face-shifted image. The intermediate feature can be an intermediate feature of the resolution enhancement model in a process of performing resolution enhancement processing on the low-resolution source image. The intermediate feature can be used to generate the high-resolution source image.

Some embodiments of this specification provide an image processing method. The image processing method can be applied to a computer device. The computer device includes but is not limited to a personal computer, a server, a server cluster including multiple servers, and the like.

Refer to FIG. 2. The image processing method can include the following steps.

Step S11: Obtain an identity feature and an attribute feature based on a source image.

In some embodiments, the source image can include a first biological object. The identity feature can be used to represent identity information of the first biological object. The attribute feature can be used to represent attribute information of the first biological object. In actual applications, the source image can be input into an identity recognition model to obtain the identity feature; and the source image can be input into an attribute recognition model to obtain the attribute feature. For the identity recognition model and the attribute recognition model, reference can be made to the above-mentioned embodiments, and details are not described herein again.

In some embodiments, the identity feature and the attribute feature can be extracted directly from the source image. Alternatively, resolution enhancement processing can be further performed on the source image, and the identity feature and the attribute feature can be extracted from a resolution-enhanced source image. Specifically, resolution enhancement processing can be performed based on a super-resolution algorithm. In actual applications, the source image can be input into a resolution enhancement model to obtain the resolution-enhanced source image. Such practice helps generate a high-resolution fused image based on a low-resolution source image.

Step S13: Obtain a context feature based on a target image.

In some embodiments, the target image can include a second biological object. The context feature is used to represent context information of the second biological object. The context feature includes at least one of the following: a posture feature used to represent posture information and a background feature used to represent background information. In actual applications, the target image can be input into a context recognition model to obtain the context feature. For the context recognition model, reference can be made to the above-mentioned embodiments, and details are not described herein again.

In some embodiments, the context feature can be extracted directly from the target image. Alternatively, resolution enhancement processing can be further performed on the target image, and the context feature can be extracted from a resolution-enhanced target image. Specifically, resolution enhancement processing can be performed based on a super-resolution algorithm. In actual applications, the target image can be input into the resolution enhancement model to obtain the resolution-enhanced target image. Such practice helps generate a high-resolution fused image based on a low-resolution target image.

Step S15: Generate a fused image of the source image and the target image based on the identity feature, the attribute feature, and the context feature.

In some embodiments, the fused image is an image obtained by fusing the source image and the target image. The “fusing” can be fusing information in the source image and information in the target image into the fused image. The fused image can be a biological feature image.

The fused image can include a third biological object. The third biological object each can be a face, an iris, a sclera, a fingerprint, a palmprint, a teeth bite, or the like. The third biological object can be a fused biological object of the first biological object and the second biological object. The “fusing” can be fusing information about the first biological object and information about the second biological object into the third biological object. Specifically, the identity information and the attribute information of the first biological object are migrated to the third biological object, so that identity information and attribute information of the third biological object are the same as the identity information and the attribute information of the first biological object. The context information of the second biological object is migrated to the third biological object, so that context information of the third biological object is the same as the context information of the second biological object.

In some embodiments, the identity feature is used to make the identity information of the third biological object the same as the identity information of the first biological object. The attribute feature is used to make the attribute information of the third biological object the same as the attribute information of the first biological object. The context feature is used to make the context information of the third biological object the same as the context information of the second biological object. The identity feature, the attribute feature, and the context feature can be fused to obtain the fused image of the source image and the target image. Specifically, the identity feature, the attribute feature, and the context feature can be input into a feature fusion model to obtain the fused image of the source image and the target image. In actual applications, the identity feature, the attribute feature, and the context feature can be directly input into the feature fusion model. Alternatively, dimension expansion processing can be performed on the identity feature, the attribute feature, and the context feature, and a dimension-expanded identity feature, a dimension-expanded attribute feature, and a dimension-expanded context feature can be input into the feature fusion model. The dimension expansion processing is used to map a low-dimensional feature to a high-dimensional feature, for example, map a feature vector to a feature map.

In some embodiments, an intermediate feature can be obtained to further improve a fusion effect and make the identity information and the attribute information of the third biological object more consistent with the identity information and the attribute information of the first biological object; and the fused image can be generated based on the identity feature, the attribute feature, the context feature, and the intermediate feature. The intermediate feature can be a feature extracted based on the source image in the resolution enhancement processing step, and is used to generate the resolution-enhanced source image.

For a specific process of generating the fused image based on the identity feature, the attribute feature, the context feature, and the intermediate feature, reference can be made to the specific process of generating the fused image based on the identity feature, the attribute feature, and the context feature, and details are not described herein again.

According to the image processing methods in the embodiments of this specification, the identity feature and the attribute feature can be obtained based on the source image; the context feature can be obtained based on the target image; and the fused image of the source image and the target image can be generated based on the identity feature, the attribute feature, and the context feature. As such, detailed attributes of the biological object in the source image can be migrated to the biological object in the fused image in an attribute editing way, so that the detailed attributes of the biological object in the fused image are consistent with the detailed attributes of the biological object in the source image, thereby improving a degree of similarity between the detailed attributes of the fused image and the detailed attributes of the source image.

Some embodiments of this specification provide a model training method. The model training method can be applied to a computer device. The computer device includes but is not limited to a personal computer, a server, a server cluster including multiple servers, and the like.

Refer to FIG. 3 and FIG. 4. The model training method can include the following steps.

Step S21: Input a source image sample and a target image sample into a generator in an adversarial network to obtain a fused image sample of the source image sample and the target image sample.

In some embodiments, the source image sample, the target image sample, and the fused image sample can be biological feature images. For example, the source image sample and the target image sample can be face images, and the fused image sample can be a face-shifted image. Specifically, the source image sample can include a first biological object, and the first biological object can be a face, an iris, a sclera, a fingerprint, a palmprint, a teeth bite, or the like. The target image sample can include a second biological object, and the second biological object can be a face, an iris, a sclera, a fingerprint, a palmprint, a teeth bite, or the like. The fused image sample can include a third biological object, and the third biological object can be a face, an iris, a sclera, a fingerprint, a palmprint, a teeth bite, or the like. The third biological object can be a fused biological object of the first biological object and the second biological object.

In some embodiments, the adversarial network (GAN, Generative Adversarial Networks) is a deep learning model that can include a generator (Generative Model) and a discriminator (Discriminative Model). The generator can be obtained based on the model architecture shown in FIG. 1. Specifically, the generator can be the image fusion model in the model architecture shown in FIG. 1. Alternatively, the generator can be the model architecture shown in FIG. 1. The generator is configured to generate the fused image sample based on the source image sample and the target image sample. The discriminator can include a classifier, such as a binary classifier or a multi-classifier. The discriminator can be a fully-connected neural network, a convolutional neural network model, a cyclic neural network model, a long-short term memory network model, or the like. The discriminator can be configured to distinguish between the biological object in the fused image sample and the biological object in the source image sample. For example, if identity information and/or attribute information of the third biological object in the fused image sample are/is the same as identity information and/or attribute information of the first biological object in the source image, the discriminator can consider that the biological objects in the fused image sample and the source image sample are the same biological object. If the identity information and/or the attribute information of the third biological object in the fused image sample are/is different from the identity information and/or the attribute information of the first biological object in the source image, the discriminator can consider that the biological objects in the fused image sample and the source image sample are different biological objects.

In some embodiments, the generator can include an identity recognition model, an attribute recognition model, and a context recognition model. As such, the source image sample can be input into the identity recognition model and the attribute recognition model to obtain an identity feature and an attribute feature; the target image sample can be input into the context recognition model to obtain a context feature; and the identity feature, the attribute feature, and the context feature can be input into a feature fusion model to obtain the fused image sample. The identity feature is used to represent the identity information of the first biological object. The attribute feature is used to represent the attribute information of the first biological object. The context feature is used to represent context information of the second biological object. For the identity feature, the attribute feature, and the context feature, reference can be made to the above-mentioned description.

In some embodiments, the generator can further include a resolution enhancement model. As such, the source image sample can be input into the resolution enhancement model to obtain a resolution-enhanced source image sample; the resolution-enhanced source image sample can be input into the identity recognition model and the attribute recognition model to obtain the identity feature and the attribute feature; the target image can be input into the resolution enhancement model to obtain a resolution-enhanced target image sample; and the resolution-enhanced target image sample can be input into the context recognition model to obtain the context feature.

In some embodiments, an intermediate feature of the resolution enhancement model in a process of performing resolution enhancement processing on the source image sample can be further obtained, to further improve a fusion effect and make the identity information and the attribute information of the third biological object more consistent with the identity information and the attribute information of the first biological object. The intermediate feature can be used to generate the resolution-enhanced source image sample. As such, the identity feature, the attribute feature, the context feature, and the intermediate feature can be input into the feature fusion model to obtain the fused image sample.

In some embodiments, one or more sample sets can be collected. The sample set includes the source image sample and the target image sample. The source image sample and the target image sample in each sample set can be input into the generator to obtain the fused image sample.

Step S23: Input the source image sample and the fused image sample into the discriminator in the adversarial network to obtain a discrimination result.

In some embodiments, the source image sample in the sample set and a fused image sample corresponding to the sample set can be input into the discriminator to obtain a discrimination result. The fused image sample corresponding to the sample set can be the fused image sample generated based on the source image sample and the target image sample in the sample set. The discrimination result is used to indicate whether the biological objects in the fused image sample and the source image sample are the same biological object. Specifically, the discrimination result can include a probability value. The probability value is used to indicate a probability that the biological objects in the fused image sample and the source image sample are the same biological object. For example, the probability value can be a real number in an interval [0, 1]. Alternatively, the discrimination result can further include an identifier. The identifier is used to indicate whether the biological objects in the fused image sample and the source image sample are the same biological object. For example, the discrimination result can be 0 or 1.0 is used to indicate that the biological objects in the fused image sample and the source image sample are the same biological object, and 1 is used to indicate that the biological objects in the fused image sample and the source image sample are different biological objects.

Step S25: Optimize a model parameter of the adversarial network based on the discrimination result by using a loss function.

In some embodiments, the loss function includes at least a first term. The first term is used to constrain an attribute information deviation between the third biological object and the first biological object. Specifically, the first term can include a first sub-term and/or a second sub-term. The first sub-term is used to constrain an attribute migration loss between the third biological object and the first biological object. For example, the second term can be represented as L_at−∥p(I_source)−p(I_swapped)∥. I_sourcerepresents the source image sample. I_swappedrepresents the fused image sample. p(I_source) represents an output of the attribute recognition model after the source image sample is input into the attribute recognition model. p(I_swapped) represents an output of the attribute recognition model after the fused image sample is input into the attribute recognition model. ∥ ∥ represents a 2-norm. The second sub-term is used to constrain an attribute consistency loss between the third biological object and the first biological object. The second sub-term can include a cross-entropy loss function, a mean square error loss function, or the like. For example, the second sub-term can be a cross-entropy loss function L_ce=1/NΣ_n−(y_nlog x_n+(1−y_n)log(1−x_n)). x_nrepresents an output of the attribute recognition model after the source image sample is input into the attribute recognition model. y_nrepresents an attribute label of the n^thsource image sample. The attribute label is used to represent the attribute information (for example, an attribute category) of the first biological object in the source image sample. N represents a quantity of source image samples.

Certainly, the loss function can further include at least one of the following: a second term, a third term, and a fourth term.

The second term is used to constrain an identity information loss between the third biological object and the first biological object. For example, the second term can be represented as L_COS=1−cos(q(I_source), q(I_swapped)). I_sourcerepresents the source image sample. I_swappedrepresents the fused image sample. q(I_source) represents an output of the identity recognition model after the source image sample is input into the identity recognition model. q(I_swapped) represents an output of the identity recognition model after the fused image sample is input into the identity recognition model. cos indicates a degree of cosine similarity. The third term is used to constrain a pixel loss between the fused image sample and the source image sample. For example, the third term can be represented as L_rec=∥I_source−I_swapped∥. I_sourcerepresents the source image sample. I_swappedrepresents the fused image sample. ∥ ∥ represents a 2-norm. The fourth term is used to constrain a loss of the adversarial network. For example, the fourth term can be represented as

L

g
⁢
a
⁢
n

=

min
G

max
D

E

x
~

P
r

[

log
⁢

D
⁡
(
x
)

]

+

E

G
⁡
(
z
)

~

P
g

[

1
-

log
⁢

D
⁡
(

G
⁡
(
z
)

)

]

.

x represents the source image sample, P_rrepresents a set formed by source image samples, and P_grepresents a set formed by fused image samples. D(x) represents the discrimination result. G (z) represents the fused image sample. z represents the sample set.

In some scenario examples, the loss function can be represented as a weighted sum of the first, second, third, and fourth terms. For example, the loss function can be represented as L=γ₁L_gan+γ₂L_ce+γ₃L_at+γ₄L_cos+γ₅L_rec. γ₁, γ₂, γ₃, γ₄, and γ₅are weight coefficients. γ₁, γ₂, γ₃, γ₄, and γ₅can be super parameters of the adversarial network.

It should be noted that the above-mentioned equations (for example, the calculation equation of the first sub-term, the calculation equation of the second sub-term, the calculation equation of the second term, the calculation equation of the third term, the calculation equation of the fourth term, and the calculation equation of the loss function) are merely examples. Actual applications are not limited thereto, and can further have other variants or changes.

In some embodiments, a training process of the adversarial network can be a game process of the generator and the discriminator. The generator is configured to generate a fused image sample as similar to the source image sample as possible to deceive the discriminator. The discriminator is configured to try to distinguish between the biological objects in the source image sample and the fused image sample. Loss information can be calculated based on the discrimination result by using the loss function, and the model parameter of the adversarial network can be optimized based on the loss information. For example, the model parameter of the adversarial network is optimized by using a back propagation algorithm. The model parameter of the adversarial network can specifically include a model parameter of the generator and a model parameter of the discriminator.

The model parameter of the generator and the model parameter of the discriminator in the adversarial network can be alternately optimized. A training process of the adversarial network can include one or more iteration processes. Each iteration process can include a first stage and a second stage. In the first stage, the model parameter of the generator can be kept unchanged, while the model parameter of the discriminator can be optimized. Specifically, the source image sample and the target image sample can be input into the generator to obtain a fused image sample of the source image sample and the target image sample; the fused image sample and the source image sample can be input into the discriminator to obtain a discrimination result; and the model parameter of the discriminator can be optimized based on the discrimination result. In the second stage, the model parameter of the discriminator can be kept unchanged, while the model parameter of the generator can be optimized. The source image sample and the target image sample can be input into the generator to obtain a fused image sample of the source image sample and the target image sample; the fused image sample and the source image sample can be input into the discriminator to obtain a discrimination result; and the model parameter of the generator can be optimized based on the discrimination result.

An iteration termination condition can be as follows: A quantity of iteration times reaches a predetermined quantity of times. Certainly, the iteration termination condition can alternatively be in another form. For example, the discriminator cannot distinguish between the biological objects in the fused image sample and the source image sample.

According to the model training method in the embodiments of this specification, the loss function of the adversarial network can include the first term, and the first term is used to constrain the attribute information deviation between the third biological object and the first biological object. Therefore, the generator obtained by training by using the loss function can generate a fused image that has detailed attributes more similar with the detailed attributes of the source image.

Refer to FIG. 5. Some embodiments of this specification further provide an image processing apparatus, which specifically includes the following units:

- a first acquisition unit 31, configured to obtain an identity feature and an attribute feature based on a source image, where the source image includes a first biological object, the identity feature is used to represent identity information of the first biological object, and the attribute feature is used to represent attribute information of the first biological object;
- a second acquisition unit 33, configured to obtain a context feature based on a target image, where the target image includes a second biological object, and the context feature is used to represent context information of the second biological object; and
- a generation unit 35, configured to generate a fused image of the source image and the target image based on the identity feature, the attribute feature, and the context feature, where the fused image includes a third biological object, identity information and attribute information of the third biological object are the same as the identity information and the attribute information of the first biological object, and context information of the third biological object is the same as the context information of the second biological object.

Refer to FIG. 6. Some embodiments of this specification further provide a model training apparatus, which specifically includes the following units:

- a first input unit 41, configured to input a source image sample and a target image sample into a generator in an adversarial network to obtain a fused image sample of the source image sample and the target image sample, where the source image sample includes a first biological object, the target image sample includes a second biological object, and the fused image sample includes a third biological object;
- a second input unit 43, configured to input the source image sample and the fused image sample into a discriminator in the adversarial network to obtain a discrimination result; and
- an optimization unit 45, configured to optimize a model parameter of the adversarial network based on the discrimination result by using a loss function, where the loss function includes at least a first term, and the first term is used to constrain an attribute information deviation between the third biological object and the first biological object.

The following describes an embodiment of a computer device in this specification. FIG. 7 is a schematic diagram of a hardware structure of a computer device according to this embodiment. As shown in FIG. 7, the computer device can include one or more processors (only one is shown in the figure), a memory, and a transmission module. Certainly, a person of ordinary skill in the art can understand that the hardware structure shown in FIG. 7 is merely an example, and does not impose a limitation on the hardware structure of the computer device described above. In practice, the computer device can further include more or fewer component units than those shown in