idref="DRAWINGS">FIG. 1 may also be implemented in any one of the following ways:

A: screening the identification of the target reference model matched with the object model from the reference model library based on attribute information of the object model and attribute information of each reference model in the reference model library; or

In practical application, when each digital human model is established, corresponding attribute information may also be configured to describe or define the model. For example, the attribute information may be a middle-aged female, a child, an elderly male, or the like. At this point, according to the attribute information of the object model, the identification of the reference model with the same attribute information can be screened from the model reference library as the identification of the target reference model.

B: showing the attribute information of each reference model in the reference model library to a user; and receiving a user selected identification of the target reference model matched with the object model.

In this mode, the identification of the target reference model matched with the object model is artificially selected by the user.

In a word, no matter which way is adopted, the identification of the target reference model matched with the object model can be efficiently and accurately acquired.

S205: acquiring an expression library of the target reference model based on the identification of the target reference model.

S206: integrally registering the target reference model with the object model.

In the present embodiment, the target reference model may be adjusted to realize the integral registration of the object model and the target reference model, so as to align head positions thereof, and the target reference model and the object model have consistent coordinate system directions and sizes by shifting, rotating and/or scaling. The step is used to roughly determine positions and sizes corresponding to the five sense organs of the model to facilitate matching of key features in a next step.

S207: registering a key feature corresponding to an expression of the target reference model with a key feature of the object model.

In the present embodiment, as an example, only one key feature is involved in the expression. For example, the key feature may include the left eye, the right eye, the mouth, or the like. In order to realize accurate expression migration, in implementation of the registration of the key features in the present embodiment, operations, such as shifting, rotating and/or scaling, may be performed on the target reference model, such that key points of the key feature corresponding to the expression and the key feature of the object model have consistent coordinates. For example, the registration of the left eye can be shown as the same coordinates of the two canthi of the left eye. The registration of the mouth can be shown as the same coordinates of the two mouth corners of the mouth.

By adopting the step, any key feature of any expression can be registered.

S208: acquiring a first size ratio of the expression based on point cloud data of a key feature corresponding to a last frame of the expression of the target reference model and point cloud data of the key feature of the target reference model in a natural state.

S209: acquiring point cloud data of a key feature corresponding to a last frame of an expression to be transferred of the object model based on the first size ratio of the expression and point cloud data of the key feature of the object model in a natural state, so as to obtain the last frame of the expression of the object model.

In the present embodiment, when the last frame of the expression is transferred, direct migration may have mismatching, or the like. For example, when normal lips are transferred to a small cherrylike mouth, European-style large lips, or the like, obvious contradictions can be generated by a width change of the lips during smiling and a stretching change of the lower jaw during mouth opening. In order to solve such problems, in the present embodiment, a length, a width, and a height of a point cloud of the key feature corresponding to the last frame of the expression may be calculated based on the point cloud data of the key feature corresponding to the last frame of the expression of the target reference model. Then, a length, a width and a height of a point cloud of the key feature in the natural state are calculated based on the point cloud data of the key feature of the target reference model in the natural state, and a ratio of the length of the point cloud of the key feature corresponding to the last frame of the expression to the corresponding length of the point cloud of the key feature in the natural state, a ratio of the width of the point cloud of the key feature corresponding to the last frame of the expression to the corresponding width of the point cloud of the key feature in the natural state, and a ratio of the height of the point cloud of the key feature corresponding to the last frame of the expression to the corresponding height of the point cloud of the key feature in the natural state are calculated as the first size ratio of the expression.

For example, for an open-mouth expression, coordinates of a highest point of an upper lip line and coordinates of a lowest point of a lower lip line of the mouth can be obtained based on point cloud data corresponding to a last frame of the open-mouth expression of the target reference model, so as to obtain a width of a point cloud of the last frame of the open-mouth expression of the target reference model; or the point cloud data of the target reference model in the natural state can be obtained, coordinates of a highest point of the lower lip line and coordinates of the lowest point of the lower lip line in the natural state are obtained, and then, the width of the point cloud of the target reference model in the natural state is obtained; and then, the width of the point cloud of the last frame of the open-mouth expression of the target reference model is divided by the width of the point cloud in the natural state to obtain a width ratio of the open-mouth expression. In the same way, an abscissa distance between the two mouth corners can be taken as the length of the point cloud, and according to the above mode, a length ratio of the open-mouth expression is correspondingly obtained; a distance between an outermost point of the lip and a height of the mouth corner at an innermost layer is taken as the height of the point cloud, and a height ratio of the open-mouth expression is correspondingly obtained according to the above mode; and therefore, the ratios of three coordinate axis directions in the first size ratio are obtained.

Next, according to the first size ratio of the expression and the point cloud data of the key feature of the object model in the natural state, the point cloud data of the key feature corresponding to the last frame when the object model makes the expression, i.e., the coordinates corresponding to each point in the point cloud of the key feature corresponding to the last frame, can be obtained, and scaling in different ratios can be performed on three coordinate axis directions in an updated migration result, such that after the transferring to the object model, the point cloud of the key feature corresponding to the last frame of the expression is better matched with the object model.

Experiments prove that after the processing in the steps S208 and S209, a negative influence of volume deformation during the expression migration is reduced, and amplitudes of unreasonable expressions, such as mouth opening, smiling, pouting, or the like, of a small cherrylike mouth are obviously inhibited.

It should be noted that, unlike global scaling of linear migration, in this step, transformation of each point is scaled differently in the three axis directions, thus ensuring that an excessive change is suppressed and similar parts are maintained.

The above-mentioned steps S206-S209 are an implementation of the step S103 in the embodiment shown in FIG. 1.

S210: transferring the point cloud data of the key feature corresponding to the last frame of the expression to be transferred of the object model onto the object model.

S211: acquiring a third feature curve corresponding to the last frame of the expression and a fourth feature curve corresponding to the last frame of the expression on the target reference model after the transferring to the object model.

S212: adjusting the third feature curve by taking the fourth feature curve as a constraint.

For example, the third feature curve and the fourth feature curve of the present embodiment may be feature curves presented in a state where the digital human model makes an expression. For example, for a closed-eye expression, a coincidence curve of the upper eyelid and the lower eyelid may be used as the third feature curve and the fourth feature curve; for a pouting expression, a curve of the closed and forwards-protruding mouth may be taken as the third feature curve and the fourth feature curve; for a puckering expression, a curve formed by the closed and inwards-rolling lips can be used as the third feature curve and the fourth feature curve. When the expression is transferred on different models, the third feature curve and the fourth feature curve are quite prone to show inconsistency, and based on this, in the present embodiment, the fourth feature curve corresponding to the last frame of the expression on the target reference model can be used as a constraint to adjust the third feature curve corresponding to the last frame of the expression after the transferring to the object model, such that the expression can be more accurately fused onto the object model, the expression can be more completely expressed on the object model, and meanwhile, topology around the feature of the expression transferred on the object model is ensured to evenly change along with the expression, and an overall fusion degree is quite high.

Specifically, in the present embodiment, the third feature curve is adjusted with the fourth feature curve as the constraint, the third feature curve and the fourth feature curve may also be referred to as dynamic curves, and this adjustment manner may also be referred to as adjustment based on a dynamic curve constraint. The dynamic curve may be a mathematical relationship used in a technical pipeline of the figure of the digital human. When a group of points are connected in series to form a space curve in the dynamic curve, a change of any point on the curve causes changes of other points on the curve. On a surface of the digital human model, such as the reference model and the object model, all the points are connected with surrounding adjacent points through straight lines to form a grid structure. The third and fourth feature curves are also formed by sides of the grid structure. The grid structure includes transverse sides and longitudinal sides, and the third and fourth feature curves in the present embodiment are generally transverse sides. When the third feature curve is adjusted with the fourth feature curve as the constraint, a position of each point in the third feature curve is adjusted, such that the third feature curve is completely consistent with the fourth feature curve, or a distance between the third feature curve and the fourth feature curve is smaller than a quite small preset threshold.

FIG. 3 is a schematic diagram of an eye structure of an object model in the present disclosure. As shown in FIG. 3, the feature curve indicated by the arrow may be one third feature curve. Due to the features of the sides and surfaces existing in the grid structure of the object model, a topological structure of the feature curve in the grid structure cannot be changed by the adjusted third feature curve. Therefore, the change of the third feature curve is inevitably transmitted to adjacent surface and adjacent points in sequence. In marks of the five sense organs, this transmission may be attenuated outwards by outer contour feature curves of the five sense organs. In other words, in order to enable smoother adjustment, in practical application, the change of the third feature curve may be attenuated outwards by a preset number of curves. The preset number may be set according to actual requirements, and may be 6, 8, 10 or other values, for example. The third feature curve is adjusted by a maximum magnitude to be matched with the corresponding fourth feature curve in the target reference model. Compared with the adjustment magnitude of the third feature curve, an adjustment magnitude of each of a preset number of feature curves adjacent to the third feature curve is reduced gradually with an increase of a distance from the third feature curve until a last step of the current adjustment of the last curve of the preset number of adjacent feature curves is completed.

Based on the above manner, under the effect of the current migration expression result of the object model, the third feature curve is restricted by the constraint of the dynamic curve, and a feature line and topological lines around a feature successively get close to the fourth feature curve. Therefore, topological features under the expression can be completely transferred into the object model.

S213: smoothing a joint between the point cloud of the key feature corresponding to the last frame of the expression after the transferring to the object model and the original point cloud of the object model.

In order to enable the point cloud of the key feature corresponding to the last frame of the transferred expression and the original point cloud of the object model to be more closely fused together, in the present embodiment, whether the joint of the point cloud and the original point cloud is smooth is further required to be detected. An unsmooth position is required to be smoothed correspondingly. For example, the side AB shown in FIG. 3 may be a joint between the point cloud of the key feature corresponding to the last frame of the transferred expression and the original point cloud of the object model, and the side AB is located in the point cloud of the key feature corresponding to the last frame of the transferred expression. During specific processing, whether an included angle between two adjacent transverse sides of the point B is greater than a preset threshold included angle may be detected, and the preset threshold included angle may be set according to an actual scenario, for example, may be 15 degrees, 20 degrees, 30 degrees, or the like, and is not limited herein. If yes, the joint is considered to be unsmooth, and a position of the point B is required to be adjusted to allow the included angle between the two adjacent transverse sides of the point B to be smaller than or equal to the preset threshold included angle. Similarly, an included angle between two adjacent longitudinal sides of the point B can be detected in a similar manner, and if the joint is unsmooth, the position of the point B is also adjusted. In a word, with the adjustment according to the above manner, each joint in the point cloud of the key feature corresponding to the last frame of the expression with the original point cloud of the object model is quite smooth, such that the point cloud of the key feature corresponding to the last frame of the expression can be perfectly fused with the original point cloud of the object model.

For example, when the transferring of the last frame of the expression is completed, a change of the joint between the point cloud of the last frame of the transferred expression and the original point cloud has a slight problem; for example, at an intersection of a plurality of five sense organ parts, such as two sides of the wing of the nose, a middle of the root of the nose, or the like, a line layout may be unsmooth in some cases. By adopting the smoothing solution of the present embodiment, such problems can be fully solved, and line smoothness is ensured.

The steps S210-S213 are complementary to the steps S206-S209, and can further enhance the transferring effect of the last frame of the expression, thus improving the accuracy and the migration efficiency of the expression migration.

S214: acquiring a second size ratio of an intermediate frame of the expression relative to the last frame of the expression based on point cloud data of a key feature corresponding to the intermediate frame of the expression in the expression library of the target reference model, the point cloud data of the key feature of the target reference model in the natural state and the point cloud data of the key feature corresponding to the last frame of the expression of the target reference model.

Specifically, the fourth feature curve corresponding to the last frame of the expression of the target reference model, a fifth feature curve corresponding to the intermediate frame of the expression, and a sixth feature curve in the natural state are obtained based on the point cloud data of the key feature corresponding to the intermediate frame of the expression in the expression library of the target reference model, the point cloud data of the key feature of the target reference model in the natural state, and the point cloud data of the key feature corresponding to the last frame of the expression of the target reference model. The fourth, fifth and sixth feature curves are feature curves of the key features in the target reference model.

Similarly, for the open-mouth expression, the fourth feature curve may correspond to the upper lip line corresponding to maximum mouth opening, the fifth feature curve may be the upper lip line of a certain intermediate process in a mouth opening process, and the sixth feature curve is a closing line of the closed upper and lower lips when the mouth is not opened. During specific calculation, the highest point on the upper lip line may be used as the key point, a distance difference between the key point on the fifth feature curve and the key point on the upper lip line on the fourth feature curve is taken and divided by a distance difference between the key point on the fourth feature curve and the key point on the upper lip line on the sixth feature curve, so as to obtain a ratio of a height of the intermediate frame of the open-mouth expression relative to the width of the last frame as a width ratio in the second size ratio. Similarly, a length ratio and a height ratio in the second size ratio can also be obtained in a similar manner in combination with the determination of the length and height in the first size ratio.

S215: obtaining an intermediate frame of the expression of the object model based on the second size ratio of the intermediate frame of the expression relative to the last frame and the last frame of the expression of the object model.

The step may include the following steps during implementation:

- (a3) obtaining a corresponding reference feature curve of the intermediate frame of the expression of the object model based on the second size ratio of the intermediate frame of the expression relative to the last frame and the corresponding third feature curve of the last frame of the expression of the object model;
- (b3) transferring the point cloud data of the key feature corresponding to the intermediate frame of the expression of the target reference model onto the object model;
- (c3) acquiring a seventh feature curve corresponding to the intermediate frame of the expression after the transferring to the object model; and
- (d3) adjusting the seventh feature curve with the reference feature curve as a constraint.

In the present embodiment, the reference feature curve is a reference feature curve which theoretically should correspond to the intermediate frame after the expression migration. The seventh feature curve is the actual reference feature curve after the intermediate frame of the expression is transferred, and in order to improve an intermediate frame transferring effect, dynamic curve adjustment is required to be performed using the step (d3).

Specifically, as with the implementation of the step S212 in the above embodiment, implementation of the step (d3) can realize outward-attenuation-type feature curve adjustment, and reference may be made to the related description of the above embodiment for details, which are not repeated herein. Further, the transferred intermediate frame may be further smoothed using the above manner of the step S213, which is not repeated herein.

The steps S214 and S215 mean that if the expression in the expression library includes the intermediate frame, the intermediate frame is also required to be transferred.

In traditional expression binding, only two pieces of data of a natural state and an expression frame exist, and an amplitude change of the expression is an interpolation of the natural state and a target frame. This method cannot show details of a real character in an expression change process, such as the situation that when a person smiles, the dimple is partly hidden and partly visible.

From the expression migration perspective, the intermediate frame, a first frame and the last frame of the expression have a certain nonlinear relationship.

In the present embodiment, the change constraint of each intermediate frame relative the last frame is established by sampling the feature curve in the intermediate frame and the last frame, and when the intermediate frame is transferred, the feature curve of the intermediate frame is retained, and the change constraint relative to the last frame is also required to be added to the feature curve, thus effectively improving the migration efficiency of the intermediate frame.

In the present embodiment, the transferred intermediate frame means that a plurality of intermediate expressions of the intermediate frame changing nonlinearly are added between the natural state and the last frame. Therefore, the change of one expression from 0 to 1 is evolved from a linear change to a nonlinear change, and detail changes, such as sequential appearance and disappearance of the dimple, sequential protrusion and straightening of a lip mark, or the like, can be realized, such that the transferred expression is more natural, reasonable and efficient.

By continuously repeating the transferring step using the above steps, all the expressions in the target reference model can be transferred to the object model one by one, and the transferred expressions of the object model can reach a better level based on the series of constraints.

With the method for transferring facial expression of digital human according to the present embodiment, the expressions in the expression library of the target reference model can be quickly and efficiently transferred to the object model in the above manner. Moreover, the expression migration process has a high speed and low computing power, and is run in real time in the background in a figure production project, minute-level automatic binding can be completed, and face capture driving can be further directly performed to preview an effect. The method does not depend on support of a device with large computing power, and does not need a long generation waiting time, and the expression migration is quite efficient and accurate.

Further, for the scenario that the effect is difficult to completely ensure by other means, such as eye closing, pouting, puckering, or the like, the method for transferring facial expression of digital human according to the present embodiment can completely transfer the expression and effectively guarantee the expression transferring effect.

Moreover, the method for transferring facial expression of digital human according to the present embodiment supports 4D nonlinear transition. The expression transition means of the nonlinear interpolation of the intermediate frame can also be supported. Under the constraint action, the effect of the transferred 4D nonlinear expression can be more effectively guaranteed.

FIG. 4 is a schematic diagram according to a third embodiment of the present disclosure. As shown in FIG. 4, the present embodiment provides an apparatus 400 for transferring facial expression of digital human, including:

- a screening module 401 configured to screen an identification of a target reference model matched with an object model from a preset reference model library, the reference model library including a plurality of reference models;
- an acquiring module 402 configured to acquire an expression library of the target reference model based on the identification of the target reference model; and
- a last frame transferring module 403 configured to transfer a last frame of an expression in the expression library of the target reference model into the object model to obtain a last frame of an expression of the object model.

The apparatus 400 for transferring facial expression of digital human according to the present embodiment has the same implementation as the above-mentioned relevant method embodiment by adopting the above-mentioned modules to implement the implementation principle and the technical effects of the digital-human expression migration, and for details, reference may be made to the description of the above-mentioned relevant method embodiment, and details are not repeated herein.

FIG. 5 is a schematic diagram according to a fourth embodiment of the present disclosure. The technical solution of an apparatus 500 for transferring facial expression of digital human according to the present embodiment of the present disclosure is further described in more detail based on the technical solution of the above-mentioned embodiment shown in FIG. 4. As shown in FIG. 5, the 500 for transferring facial expression of digital human according to the present embodiment includes modules with the same names and functions as those in FIG. 4: a screening module 501, an acquiring module 502 and a last frame transferring module 503.

In the present embodiment, the screening module 501 is configured to:

- screen the identification of the target reference model matched with the object model from the reference model library based on attribute information of the object model and attribute information of each reference model in the reference model library.

Or optionally, in one embodiment of the present disclosure, the screening module 501 is configured to:

- show the attribute information of each reference model in the reference model library to a user; and receive the user selected target reference model matched with the object model.

Or optionally, in one embodiment of the present disclosure, the screening module 501 is configured to:

- acquire a first feature curve of a preset part from the object model;
- acquire a second feature curve of the preset part from each reference model;
- calculate an offset distance of the preset part between the object model and each reference model based on the first feature curve and each second feature curve; and
- screen the identification of the target reference model matched with the object model from the reference model library based on the offset distance of the preset part between the object model and each reference model.

Further optionally, in one embodiment of the present disclosure, the screening module 501 is configured to: acquire coordinates of each point on the first feature curve and each second feature curve;

- calculate a distance of points with same point identifications on the first feature curve and each second feature curve based on the coordinates of the points on the first feature curve and each second feature curve;
- add the distances of the points with the point identifications on the first feature curve and each second feature curve to obtain a point distance sum; and
- obtain the offset distance of the preset part between the object model and each reference model based on the point distance sum and a number of the points included on the first feature curve.

Further optionally, in one embodiment of the present disclosure, the screening module 501 is configured to:

- screen an identification of the reference model with the minimum offset distance of the preset part from the multiple reference models in the reference model library as the identification of the target reference model matched with the object model.

Further optionally, in one embodiment of the present disclosure, the screening module 501 is configured to: if two or more preset parts are included, calculate a comprehensive offset distance between the object model and the reference models based on pre-configured weights of the preset parts and the offset distances corresponding to the preset parts; and

- screen the identification of the target reference model matched with the object model from the reference model library based on the comprehensive offset distance between the object model and each reference model.

Further optionally, in one embodiment of the present disclosure, the last frame transferring module 503 is configured to:

- integrally register the target reference model with the object model;
- register a key feature corresponding to an expression of the target reference model with a key feature of the object model;
- acquire a first size ratio of the expression based on point cloud data of a key feature corresponding to a last frame of the expression of the target reference model and point cloud data of the key feature of the target reference model in a natural state; and
- acquire point cloud data of a key feature corresponding to a last frame of the expression to be transferred of the object model based on the first size ratio of the expression and point cloud data of the key feature of the object model in a natural state, so as to obtain the last frame of the expression of the object model.

Further optionally, in one embodiment of the present disclosure, the last frame transferring module 503 is configured to:

- transfer the point cloud data of the key feature corresponding to the last frame of the expression to be transferred of the object model onto the object model;
- acquire a third feature curve corresponding to the last frame of the expression and a fourth feature curve corresponding to the last frame of the expression on the target reference model after the transferring to the object model;
- adjust the third feature curve by taking the fourth feature curve as a constraint; and
- smooth a joint between the point cloud of the key feature corresponding to the last frame of the expression after the transferring to the object model and the original point cloud of the object model.

Further optionally, as shown in FIG. 5, in one embodiment of the present disclosure, the apparatus 500 for transferring facial expression of digital human further includes:

- an intermediate frame transferring module 504 configured to, if the expression library further includes an intermediate frame of the expression, transfer the intermediate frame of the expression in the expression library of the target reference model into the object model to obtain an intermediate frame of the expression of the object model.

Further optionally, in one embodiment of the present disclosure, the intermediate frame transferring module 504 is configured to:

- acquire a second size ratio of the intermediate frame of the expression relative to the last frame of the expression based on point cloud data of a key feature corresponding to the intermediate frame of the expression in the expression library of the target reference model, the point cloud data of the key feature of the target reference model in the natural state and the point cloud data of the key feature corresponding to the last frame of the expression of the target reference model; and
- obtain an intermediate frame of the expression of the object model based on the second size ratio of the intermediate frame of the expression relative to the last frame and the last frame of the expression of the object model.

Further optionally, in one embodiment of the present disclosure, the intermediate frame transferring module 504 is configured to:

- obtain the fourth feature curve corresponding to the last frame of the expression of the target reference model, a fifth feature curve corresponding to the intermediate frame of the expression, and a sixth feature curve in the natural state based on the point cloud data of the key feature corresponding to the intermediate frame of the expression in the expression library of the target reference model, the point cloud data of the key feature of the target reference model in the natural state, and the point cloud data of the key feature corresponding to the last frame of the expression of the target reference model; and
- acquire the second size ratio of the intermediate frame of the expression relative to the last frame based on coordinates of key points on the fourth feature curve, the fifth feature curve and the sixth feature curve.

Further optionally, in one embodiment of the present disclosure, the intermediate frame transferring module 504 is configured to:

- obtain a corresponding reference feature curve of the intermediate frame of the expression of the object model based on the second size ratio of the intermediate frame of the expression relative to the last frame and the corresponding third feature curve of the last frame of the expression of the object model;
- transfer the point cloud data of the key feature corresponding to the intermediate frame of the expression of the target reference model onto the object model;
- acquire a seventh feature curve corresponding to the intermediate frame of the expression after the transferring to the object model; and
- adjust the seventh feature curve with the reference feature curve as a constraint.

The apparatus 500 for transferring facial expression of digital human according to the present embodiment has the same implementation as the above-mentioned relevant method embodiment by adopting the above-mentioned modules to implement the implementation principle and the technical effects of the digital-human expression migration, and for details, reference may be made to the description of the above-mentioned relevant method embodiment, and details are not repeated herein.

In the technical solution of the present disclosure, the acquisition, storage and application of involved user personal information are in compliance with relevant laws and regulations, and do not violate public order and good customs.

According to the embodiment of the present disclosure, there are also provided an electronic device, a readable storage medium and a computer program product.

FIG. 6 shows a schematic block diagram of an exemplary electronic device 600 which may be configured to implement the embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other appropriate computers. The electronic device may also represent various forms of mobile apparatuses, such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other similar computing apparatuses. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementation of the present disclosure described and/or claimed herein.