The embodiments of the present disclosure provide a video processing method, a video processing apparatus, an electronic device, and a computer readable storage medium, where the method may be executed by an electronic device such as a terminal device or a server, and the terminal device may include but is not limited to a personal computer, a smart phone, a tablet computer, a Personal Digital Assistant (PDA), a server, or the like. The video processing method may be realized by calling computer readable program instructions stored in a memory by a processor. Alternatively, the method may be performed by a server.

For a better understanding of the present disclosure, the video processing method according to the embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be noted that these embodiments are not intended to limit the scope of the present disclosure.

FIG. 1 is a flowchart of a video processing method in some embodiments of the present disclosure. As shown in FIG. 1, the video processing method includes the following steps S110 and S120.

- S110, acquiring a video frame of a video to be processed.

In this step, an execution device may receive the video to be processed through a communication network, read the video to be processed from an internal storage in the execution device or an external storage, obtain the video to be processed by downloading from the internet through the search engine in the execution device, obtain the video to be processed by shooting through a shooter in the execution device, such as a camera, a video camera, of the like, or acquire, from a video processing instruction, the video frame of the video to be processed carried by the instruction. It should be understood that the video to be processed in the embodiment of the present disclosure may alternatively be acquired in other manners, and the specific contents are not limited herein.

- S120, performing color adjustment on the video frame using at least one color enhancement model in a color enhancement model pool, to obtain an video output frame subjected to the color adjustment.

The color enhancement model pool includes a set of models for performing color adjustment on videos of various video content types with different requirements on styles thereof.

In some embodiments, the color enhancement model pool may include at least one color enhancement model, and each color enhancement model in the color enhancement model pool may be applicable to color adjustment on the video frame of a video content type with requirements on a style thereof.

As an example, the video content types may include at least one of the following types: television series, documentary, movie, animation, sports, news, drama, entertainment, and education. In practical application scenes, the video content types may alternatively be other types, and are not limited herein.

As an example, the requirements on styles refer to requirement on style characteristics of the video to be processed. For a video of a certain video content type, a video content of the video generally has more uniform style characteristics, and the style characteristics may include at least one of the following characteristic items: painting style, image quality, color tone, and brightness.

The painting style is an integral audio-visual feeling and visual style formed for the viewer watching the video. The image quality, i.e. the quality of image, includes the indexes of definition, sharpness, lens distortion, dispersion, resolution, color gamut range, color purity (or color brilliance), color balance, etc. The color tone refers to the relative brightness and darkness of color in the image. The brightness refers to the brightness degree of the image. In practical application scenes, the style characteristics of the video content may further include other characteristic items, which are not specifically limited herein.

According to the video processing method in the embodiment of the present disclosure, the color enhancement model pool includes a set of models for adjusting videos of various video types with different requirements on styles thereof, the color adjustment is performed on the acquired video frame of the video to be processed according to at least one color enhancement model in the model pool, a color toning effect corresponding to the video content type and the requirement on style of the video frame can be automatically generated, and the video processing process can be automatically realized through calling the model by the execution device, so that video color creation can be automatically performed, ultra-high definition reproduction of low-definition film sources can be automatically completed, the film source production period can be shortened, the labor cost can be saved, and the video processing efficiency can be improved.

FIG. 2 is a flowchart of a color adjustment process in some embodiments of the present disclosure. As shown in FIG. 2, in some embodiments, the step S120 may specifically include the following steps S21 to S23.

- S21, receiving information including the video content type and the requirement on style of the video frame.

In some embodiments, in the case that the video frame of the video to be processed is received, the information including the video content type and the requirement on style of the video frame may also be received, so as to select the corresponding model from the color enhancement model pool according to the video content type and the requirement on style. For example, the step S21 may be executed prior or subsequent to the step S110. Alternatively, the step S21 may be executed synchronously with the step S110, that is, the execution device may synchronously receive the information including the video content type and the requirement on style of the video frame, when receiving the video frame of the video to be processed. In actual application scenes, as long as the video frame of the video to be processed and the video content type and the requirement on style of the video are acquired prior to the color enhancement model is selected, the execution sequence of the steps S110 and S21 may be flexibly set according to actual needs.

- S22, selecting the color enhancement model corresponding to the video content type and the requirement on style from the color enhancement model pool.

In some embodiments, different video content types may have different requirements on styles, as may requirements on styles for the same video content type. Therefore, according to the received information including the video content type and the requirement on style of the video frame, a color enhancement model matched with the video content type and the requirement on style can be selected from the color enhancement model pool.

- S23, processing the video frame using the selected color enhancement model, to obtain the video output frame subjected to color adjustment.

In some embodiments, according to the received information including the video content type and the requirement on style of the video frame, the color enhancement model matched with the video content type and the requirement on style may be selected from the color enhancement model pool, and then the video frame may be processed according to the matched color enhancement model, to obtain the video output frame subjected to color adjustment.

In other embodiments, if at least two color enhancement models are selected from the color enhancement model pool, and one of the at least two color enhancement models is a model matched with the video content type and the requirement on style, then each of the at least two color enhancement models may be used to process the video frame, to obtain an output result of each selected color enhancement model, and the output results of the selected color enhancement models may be performed a weighted fusion processing according to preset weights of the color enhancement models corresponding to the video frame, to generate the video output frame.

In the embodiment of the present disclosure, when the color adjustment is performed on the video frame using at least one color enhancement model in the color enhancement model pool, for the acquired video frame of the video to be processed, the matched color enhancement model may be firstly selected from the color enhancement model pool according to the received information including the video content type and the requirement on style of the video frame, and then the video frame of the video to be processed is processed using the selected color enhancement model, so as to generate a color toning effect corresponding to the received video content type and the requirement on style, thereby realizing automatic adjustment of the video to be processed, meeting the requirement on color style of the corresponding video content type, and improving the data processing efficiency of the video color adjustment.

FIG. 3 illustrates a flowchart of a color adjustment process in some embodiments of the present disclosure. As shown in FIG. 3, in some embodiments, the step S120 may specifically include the following steps S31 to S33.

- S31, processing the video frame using each color enhancement model in the color enhancement model pool, to obtain the video frame subjected to color adjustment output by each color enhancement model.

In this step, each color enhancement model in the color enhancement model pool processes the video frame of the video to be processed, and a result of color adjustment of each color enhancement model on the video frame according to the requirement on style of the corresponding video content type is obtained.

- S32, processing the video frame based on a content analysis model, to generate a weight of each color enhancement model corresponding to the video frame: where the content analysis model is configured to determine the weight of each color enhancement model in the color enhancement model pool according to the video frame.

In this step, the content analysis model may assign a weight (may also be referred to as a weighing coeffient) to each color enhancement model in the color enhancement model pool through analyzing the video content type of the video frame, so that the output result of each model in the model pool is assigned a weight, according to the weight of each color enhancement model corresponding to the video frame output by the content analysis model.

- S33, performing weighted fusion processing on the video frames subjected to color adjustment output by the color enhancement models, using the weights of the color enhancement models corresponding to the video frame, to generate the video output frame.

In this step, the weighted fusion process includes: performing matrix multiplication operation on the weight of each color enhancement model corresponding to the video frame and the video frame subjected to color adjustment output by the corresponding color enhancement model; and performing matrix addition on the multiplication operation results, to obtain the video output frame in a matrix form.

In the video processing method according to the embodiment of the present disclosure, each color enhancement model in the color enhancement model pool may be assigned a weight, according to the weight of each color enhancement model corresponding to the video frame, output by the content analysis model. Finally, a color adjustment result obtained by fusing the processing results of all the color enhancement models in the model pool is output, and the color adjustment result is based on the analysis on the video content type of the video to be processed, so that a video color adjustment result better conforming to the video content type is obtained.

The video processing system and the corresponding video processing method in the exemplary embodiments of the present disclosure are described below with reference to FIGS. 4a and 4b. FIG. 4a illustrates a schematic diagram of a structure of a video processing system in some embodiments of the present disclosure; and FIG. 4b illustrates a schematic diagram of a structure of a video processing system in some embodiments of the present disclosure. The same reference characters in FIGS. 4a and 4b may refer to the same structures.

As shown in FIG. 4a, in some embodiments, the video processing system includes: a video frame input module 41, a color enhancement model pool processing module 42, a content analysis model processing module 43, a weight output module 44, a fusion processing module 45, and a video frame output module 46.

In some embodiments, the video frame input module 41 may be used to acquire an input video frame.

In some embodiments, the execution device may acquire, from the received video processing instruction, the video frame carried in the instruction.

The color enhancement model pool processing module 42 is used to perform color adjustment on the video frame using at least one color enhancement model in the color enhancement model pool, to obtain a video output frame subjected to the color adjustment.

In some embodiments, the color enhancement model pool processing module may be referred to simply as the color enhancement model pool.

In some embodiments, in the case of acquiring an input video frame, if information including a video content type and a requirement on style of the video frame is further received, then a color enhancement model corresponding to the video content type and the requirement on style may be selected from a color enhancement model pool, and the video frame is processed using the selected color enhancement model, to obtain the video output frame subjected to the color adjustment.

In some embodiments, no matter whether the information including the video content type and requirement on style of the video frame is received, each color enhancement model in the color enhancement model pool may be used to process the video frame, to obtain a video frame subjected to the color adjustment output by each color enhancement model.

The content analysis model processing module 43 is used to process the video frames based on the content analysis model, to generate a weight of each color enhancement model corresponding to the video frame.

In some embodiments, the weight of each color enhancement model in the color enhancement model pool corresponding to the video frame may be preset. For example, when a large batch of videos to be processed of the same video content type with the same requirement on style thereof are processed, a weight may be pre-assigned to each color enhancement model in the color enhancement model pool, according to a plurality of test results or actual experiences.

In other embodiments, the video frame is processed based on the content analysis model, and the weight of each color enhancement model corresponding to the video frame can be automatically generated, so that the participation of manual processing is reduced, which is favorable to realize the automation of color creation on the video, the video reproduction cost is greatly reduced, and the video processing efficiency is improved.

The weight output module 44 is used to output a weight of each color enhancement model corresponding to the video frame.

The fusion processing module 45 is used to perform a weighted fusion processing on the video frames subjected to color adjustment output by the color enhancement models, using the weights of the color enhancement models corresponding to the video frame, to obtain a fusion result.

The video frame output module 46 is used to generate the video output frame subjected to color adjustment, according to the fusion result.

In the embodiment of the present disclosure, color adjustment may be performed on an acquired video frame according to at least one color enhancement model in the color enhancement model pool, and a color toning effect corresponding to the video content type and the requirement on style of the video frame is automatically generated. Further, each color enhancement model in the color enhancement model pool is assigned a weight, according to the weight of each color enhancement model corresponding to the video frame output by the content analysis model, and the color adjustment result obtained by fusing the processing results of all the color enhancement models is output, so that a video color adjustment result better meeting the requirement on style of the video content type is obtained, the automatic adjustment of the video color is realized, the video reproduction cost can be reduced, and the video processing efficiency can be improved.

As shown in FIG. 4b, the video processing system is substantially the same as the video processing system in FIG. 4a, except that a plurality of color enhancement models in the color enhancement model pool are shown in FIG. 4b.

In FIG. 4b, the color enhancement model pool includes at least one group of color enhancement models, and each group of color enhancement models corresponds to one video content type. Each group of color enhancement models includes at least one color enhancement model, and each color enhancement model corresponds to one requirement on style.

Illustratively, a first group of color enhancement models is a group of models corresponding to a video of television series, and the group of models include N1 number of models, such as television series style model 1, television series style model 2 . . . , television series style model N1. Each model in the group of models may correspond to a different requirement on style of the video of television series. A second group of color enhancement models is a group of models corresponding to a documentary video, and the group of models include N2 number of models, such as documentary style model 1, documentary style model 2 . . . documentary style model N2. Each model in the group of models corresponds to a different requirement on style of the documentary video. A third group of color enhancement models is a group of models corresponding to a movie video, and the group of models include N3 number of models, such as movie style model 1, movie style model 2 . . . , movie style model N3. Each model in the group of models may correspond to a different requirement on style of the movie video. N1, N2, and N3 are each an integer greater than or equal to 1.

It should be understood that other groups of models corresponding to at least one requirement on style of other video content types may also be included in the color enhancement model pool, and may be specifically set according to an actual situation, which is not specifically limited by the embodiment of the present disclosure.

With continued reference to FIG. 4b, the particular processing units in the fusion module are shown in FIG. 4b, where the symbol ⊗ represents a matrix multiplication operation and the symbol Σ represents a matrix addition operation. In some embodiments, the fusion module may be specifically used for: performing matrix multiplication operation on the weight of each color enhancement model corresponding to the video frame output by the content analysis model and the video frame subjected to color adjustment output by the corresponding color enhancement model; and performing matrix addition on the matrix multiplication operation results, to obtain a matrix addition result, to be used for generating the video output frame according to the matrix addition result.

In the embodiment of the present disclosure, the video processing method may be automatically implemented by the execution device using an Artificial Intelligence (AI) model. The AI model may be a model implemented based on a convolutional neural network, and may be used in the image processing fields of image detection, image classification, image content analysis, and the like. Both of the color enhancement model and the content analysis model in the embodiments of the present disclosure are AI models.

In some embodiments, the models in the color enhancement model pool have a same model structure. The specific structure and operation principle of the color enhancement model are described in detail below with reference to FIGS. 5 to 8.

FIG. 5 illustrates a schematic diagram of a specific architecture of a color enhancement model in an embodiment of the present disclosure.

In FIG. 5, the color enhancement model includes 2N number of Residual Dense Blocks (RDBs) 51 and N number of fully connected layers 52, where N is an integer greater than or equal to 1. Illustratively, the color enhancement model includes at least two RDBs 51 and at least one fully connected layer 52.

In the network structure of the color enhancement model shown in FIG. 5, the above 2N number of RDBs may be set as two task branch models: an Image Reconstruction (IR) branch model and a color enhancement branch model, where each task branch model includes N number of RDBs 51.

With continued reference to FIG. 5, the color enhancement model may further include a first convolutional layer 53, a filter unit 54, a second convolutional layer 55, a third convolutional layer 56, and a fourth convolutional layer 57.

How the color adjustment of the video frame is performed by the color enhancement model in FIG. 5 is described below with reference to FIGS. 6 and 7. FIG. 6 illustrates a specific flowchart of color adjustment on a video frame using a color enhancement model in an embodiment of the present disclosure; and FIG. 7 illustrates a specific flowchart of image reconstruction and color adjustment in the embodiment of the present disclosure.

As shown in FIG. 6, the processing the video frame using the selected color enhancement model, to obtain the video output frame subjected to color adjustment, in step S23, may specifically include the following steps S601 to S605.

- S601, performing feature extraction on the video frame, to obtain a first image feature.

In this step, the feature extraction may be performed on the video frame input to the color enhancement model, through the first convolutional layer 53 in the color enhancement model, to obtain the first image feature.

- S602, filtering the video frame, and performing feature extraction on the filtered video frame, to obtain a second image feature.

In this step, the filter unit 54 may perform filtering processing on the video frame input to the color enhancement model, and the second convolutional layer 55 may perform the feature extraction on the filtered video frame, to obtain the second image feature.

In some embodiments, the filter unit 54 may be one of a Gaussian filter unit, a mean filter unit, and a median filter unit. The Gaussian filter unit is used to perform Gaussian filtering on the video frame: the mean filter unit is used to perform a mean filtering on the video frames; and the median filter unit is used to perform a median filtering on the video frame.

Specifically, the Gaussian filtering is a linear smoothing filtering, and is suitable for eliminating Gaussian noise. In some scenes, the Gaussian smoothing filter is very effective in suppressing noise that follows normal distribution. The mean filtering may take the pixel mean of local image information of image pixel points in a video frame as the pixel value subjected to the processing of the image pixel points, and has a simple filtering algorithm and a high calculation speed. The median filtering may sequentially arrange the gray values of the image pixel points in the video frame from high to low, and take the median value of gray values as the gray value of the image pixel points. In some scenes, median filtering may be beneficial to better preserve the image boundaries when reducing noise in an image.

In the embodiment of the present disclosure, the filter unit is used to perform the filtering processing on the video frame, which is not only beneficial to removing noise in the video frame and realizing smooth transition of an image in the video frame, but also beneficial to enhancing low frequency characteristics of the image in the video frame, to obtain low frequency information of the image in the video frame.

In some embodiments, if a difference between the pixel values of adjacent pixels in a region in an image is small (less than or equal to a pixel value threshold), the information of the region is low frequency information: if the difference between pixel values of the adjacent pixels in a region in an image is large (exceeds the pixel value threshold), the information of the region is high frequency information.

In some embodiments, the low frequency information may be color information, grayscale information, etc. of the image, and the high frequency information may refer to texture information, edge information, etc. of the image. Subjected to the filtering processing of the filter unit, the image feature extracted subsequently is facilitated to be less affected by the influence of high frequency information such as image noise, image texture and the like, so that more low frequency color feature (also referred to as color feature with low frequency) is reserved.

It should be understood that the filter unit may alternatively be other types of filter units for performing other types of filtering processing on the video frame. Specifically, the type of the filter unit may be selected according to actual needs, which is not specifically limited by the embodiment of the present disclosure.

- S603, inputting the first image feature into the image reconstruction branch model, and inputting the second image feature into the color enhancement branch model, to perform image reconstruction and color adjustment according to the first image feature and the second image feature.

Image Reconstruction (IR) is a technique of processing an existing Low-resolution (LR) image and converting the LR image into a High-resolution (HR) image.

In this step, the image reconstruction branch model performs feature extraction on the first image feature of the video frame through RDB, and performs rough reconstruction on the output image. In the rough reconstruction, the video frame of the video to be processed is taken as an initial image, the initial image is subjected to feature extraction by a convolutional layer to obtain feature information of the initial image, the feature information of the initial image is extracted using the RDB, and an image obtained through reconstruction according to the feature information of the initial image may be called a roughly reconstructed image.

In this step, the second image feature (i.e., a low frequency color feature) obtained through the filtering processing of the filter unit is analyzed and extracted, to be used for subsequently performing color adjustment according to the analyzed and extracted feature.

- S604, performing feature fusion processing on an output result of the image reconstruction branch model and an output result of the color enhancement branch model, to obtain a fusion result.

In this step, the output result of the image reconstruction branch model and the output result of the color enhancement branch model are multiplied in a corresponding feature channel, and results of the multiplication operation of the feature channels are subjected to the feature addition throughout the feature channels, to obtain a fusion result.

- S605, generating the video output frame subjected to color adjustment, according to the fusion result.

In this step, an output result of a global color adjustment on the input video frame is obtained, according to the fusion result.

Illustratively, a feature extraction is performed on the output result of the image reconstruction branch model using the third convolutional layer 56, and a feature extraction is performed on the output result of the color enhancement branch model using the fourth convolutional layer 57. The result of the feature extraction of the third convolutional layer 56 and the result of the feature extraction of the fourth convolutional layer 57 are multiplied in a corresponding feature channel, and the feature of the two task branch models are fused, to obtain the output result of color adjustment of the color enhancement model.

In the embodiment of the present disclosure, when the color enhancement model is used to process a video frame, the image reconstruction and the color adjustment may be performed on the input video frame through the image reconstruction branch model and the color enhancement branch model, and the feature fusion processing may be performed on the output result of the image reconstruction branch model and the output result of the color enhancement branch model, to obtain the output result of color adjustment of the color enhancement model.

In FIG. 5, a size (i.e., a width and a height) of a convolutional kernel in each RDB 51 is the same. For example, the size of the convolutional kernel in RDB 51 is 3×3, and the number of output feature channels is 64. A size of the filter kernel in the filter unit 54 is 21×21. A size of the first convolutional layer 53 is the same as a size of the second convolutional layer 55, for example, both of the two sizes are 3×64. A size of the third convolutional layer 56 is the same as a size of the fourth convolutional layer 57, for example, both of the two sizes are 64×3.

It should be noted that, in the embodiment of the present disclosure, the number and size of convolutional kernels for feature extraction, the number and size of convolutional kernels in each RDB 51, and the number and size of filter kernels in the filter unit 54 may be set as needed, which are not particularly limited by the embodiment of the present disclosure.

In some embodiments, for two task branch models (i.e., the image reconstruction branch model and the color enhancement branch model) in the color enhancement model shown in FIG. 5, each branch model may include N number of residual dense blocks RDBs; and the color enhancement model further includes N number of fully connected layers: where N is an integer greater than or equal to 1. The following describes the processes of image reconstruction and color adjustment based on the model architecture of the two task branch models with reference to a specific embodiment.

As shown in FIG. 7, the inputting the first image feature into the image reconstruction branch model and inputting the second image feature into the color enhancement branch model, to perform image reconstruction and color adjustment according to the first image feature and the second image feature, in step S603, may specifically include the following steps S701 to S705.

- S701, acquiring a first RDB in the image reconstruction branch model and a first RDB in the color enhancement branch model, as a current RDB of a first branch model and a current RDB of a second branch model, respectively.
- S702, processing the first image feature through the current RDB of the first branch model, and processing the second image feature through the current RDB of the second branch model, to obtain a feature map output by the current RDB of the first branch model and a low frequency color feature output by the current RDB of the second branch model.
- S703, processing the low frequency color feature output by the current RDB of the second branch model through a fully connected layer, to obtain a feature vector, and correspondingly multiplying the feature vector by the feature map output by the current RDB of the first branch model, to obtain a feature map subjected to local color feature adjustment.

In this step, image feature extraction may be performed through the current RDB (e.g., the convolutional kernel has a size of 3×3, and the number of the output feature channels is 64) in the image reconstruction branch model. The low frequency color feature of the image is analyzed and extracted through the current RDB (e.g., the convolutional kernel has a size of 3×3, and the number of the output feature channels is 64) in the color enhancement branch model, each layer of features in the extracted low frequency color features is converted into a feature vector with a width of 1 (such as a feature vector with the size of 1×64) through a fully connected layer, and each element in the feature vector output by the fully connected layer is multiplied by the feature map output by the image reconstruction branch model in a corresponding feature channel of the feature map output by the current RDB, so that a local fine adjustment on the color feature can be realized.

- S704, taking the feature map subjected to the local adjustment on the color feature as a new first image feature, and taking the low frequency color feature output by the current RDB of the second branch model as a new second image feature.
- S705, acquiring a next RDB in the image reconstruction branch model and a next RDB in the color enhancement branch model as a new current RDB of the first branch model and a new current RDB of the second branch model, respectively, until the number of acquiring times is equal to N, taking the newly obtained feature map subjected to local adjustment on the color feature as an output result of the image reconstruction branch model, and taking the newly obtained low frequency color feature the an output result of the color enhancement branch model.

Through the steps S701 to S705, the low frequency color feature extracted by the last RDB in the color enhancement branch model is obtained, the low frequency color feature is converted into a feature vector with a width of 1 through the last fully connected layer, and each element in the feature vector output by the last fully connected layer is multiplied by the feature map output by the last RDB in the image reconstruction branch model in a corresponding feature channel, to obtain an output result of the image reconstruction branch model in the color enhancement model. The low frequency color feature extracted by the last RDB in the color enhancement branch model is taking as the output result of the color enhancement branch model in the color enhancement model. The feature fusion processing is performed, based on the output result of the image reconstruction branch model and the output result of the color enhancement branch model, to obtain an output result of color adjustment of the color enhancement model, thereby realizing a global color style enhancement.

FIG. 8 illustrates a diagram of a specific architecture of a residual dense block in an embodiment of the present disclosure. As shown in FIG. 8, in some embodiments, each RDB includes a densely connected convolutional network (DenseNet) 511 and a local feature fusion layer 512.

In this embodiment, the local feature fusion layer 512 is used for performing feature fusion processing on the input feature to the RDB and the output feature of the DenseNet 511 using local residual learning, to obtain the output feature of the RDB.

In some embodiments, the DenseNet 511 includes M number of dense blocks 5111, where M is an integer greater than or equal to 1. An input to a first dense block 5111 is the input feature to the RDB; and an input feature to the i^thdense block 5111 is the input feature to the RDB and the output features of all dense blocks 5111 before the i^thdense block 5111, where i is an integer greater than 1 and less than or equal to M.

In FIG. 8, three dense blocks 5111 included in the DenseNet 511 in the case of M=3 is shown. In actual application scenes, the number of the dense blocks 5111 may be set according to actual needs.

With continued reference to FIG. 8, an input to the local feature fusion layer 512 includes the input feature to the RDB (e.g., which may be denoted as F_d-1) and output feature of the dense blocks 5111 (e.g., which may be denoted as F_d,1, F_d,C, etc.). The local feature fusion layer 512 may include a concat layer and a convolutional layer with a size of 1×1. The concat layer fuses feature input to the concat layer in a concatenation manner, and the 1×1 convolutional layer is used for adaptively fusing a series of feature of different layers, to generate the output feature (e.g., which may be denoted as F_d,LF) of local feature fusion. The output feature of the local feature fusion and the input feature to the RDB are subjected to the feature addition in the corresponding channel, to generate a global feature fusion result (e.g., which may be denoted as F_d), and the global feature fusion result is taken as the output feature of the RDB.

In the embodiment of the present disclosure, the RDB is of a network structure combining a residual network and a dense network, and realizes information sharing by retaining low-level information and adaptively acquires information acquired by different convolutional layers through feature fusion, under the combined action of dense connection, feature fusion and residual learning, so that better performance is realized in the image processing direction. Specifically, the middle layer of the RDB may use the dense connection mode of the dense block 5121, and the connection block 5122 of the last layer outputs the processed result of the RDB in a manner of residual skip connection.

In this embodiment, each dense block in the RDB may transmit feature information to be stored to a subsequent dense block. The network structure may fully ensure transmission and circulation of the feature information between every two adjacent layers, reduce parameter amount while implementing feature reuse, and ensure a better processing speed in a case where an amount of the network parameters is large.

In some embodiments, each color enhancement model in the color enhancement model pool has the same model structure. The color enhancement model in the color enhancement model pool is a trained color enhancement model obtained through pre-training an initial color enhancement model by adopting different color enhancement data pairs for different color styles of different video content types. The color enhancement data pair includes an un-toned image block and a toned image block with a same image size, which are selected from an un-toned image and a first toned image, respectively, at a same position in image, where the first toned image is an image pre-acquired by toning the un-toned image based on the color style characteristics of the image.

In this embodiment, the initial color enhancement model has the same model structure as each color enhancement model in the color enhancement model pool. Initial values of the model parameters of the initial color enhancement model may be random values of parameters of the model or empirical values of parameters of the model. The first toned image may be a pre-acquired image subjected to color adjustment performed by a colorist. The training data of the color enhancement network model includes image blocks with the same image size selected from the un-toned image and the first toned image at the same positions in image. Illustratively, an image block with a size of 256×256 at a certain position in an image may be randomly extracted and input to the network for training, during the training process of each color enhancement model.

It should be understood that the size of the image block required in the training process may be selected according to actual needs, which is not particularly limited in the embodiments of the present disclosure.

In some embodiments, training an initial color enhancement model, i.e., a color enhancement model, may include: processing an un-toned image block selected from the un-toned image using an initial color enhancement model, to obtain a color adjustment result of the un-toned image block, and acquiring a toned image block corresponding to the un-toned image block from the color enhancement data pair, as a color adjustment reference result: constructing a loss function of the initial color enhancement model based on the color adjustment result and the color adjustment reference result: adjusting model parameters of the initial color enhancement model using the loss function of the initial color enhancement model, to obtain an updated color enhancement model; and taking the updated color enhancement model as a new initial color enhancement model, and processing the un-toned image block selected from the un-toned image using the new initial color enhancement model, until the training process of the initial color enhancement model meets a first training end condition, to obtain the trained color enhancement model.

The first training end condition includes at least one of the following conditions that a training loss calculated according to the loss function of the initial color enhancement model is less than or equal to a set first loss threshold, or the number of training times for the initial color enhancement model reaches a set first threshold of number of times.

In the embodiment of the present disclosure, the loss function may be used to estimate a gap between the result of each model training and the training target. For the initial color enhancement model, the result of each model training is the color adjustment result of the un-toned image block output by the initial color enhancement model, and the training target is the corresponding color adjustment reference result. The first loss threshold and the first threshold of number of times may be customized according to the actual training requirements.

In some embodiments, in the training process of the color enhancement model, the step of constructing the loss function of the initial color enhancement model based on the color adjustment result and the color adjustment reference result may specifically include: calculating a minimum square error between the color adjustment result and the color adjustment reference result, to obtain a minimum square error loss; calculating a Gaussian filter output result of the color adjustment result and a Gaussian filter output result of the color adjustment reference result, to obtain a color loss; and performing weighted summation on the minimized square error loss and the color loss, to generate the loss function of the initial color enhancement model.

Illustratively, the loss function of the initial color enhancement model may be expressed as the following expression (1):

Loss
⁢
1

=

a
*



I
^

1

-

I

gt
⁢
11



2

+

b
*



Gauss
(

I
^

1

)

-

Gauss
(

I

gt
⁢
1

)



2

.

(
1
)

In the above expression (1), Loss1 is the Loss function of the initial color enhancement model, Î₁is the color adjustment result, I_gt1is the color adjustment reference result, Gauss is a function of calculating a Gaussian filter output result, where a size of the filter kernel is a size of a filter kernel of a filter unit in the trained color enhancement model, ∥Î₁−I_gt1∥₂, is an L2 norm of a difference between the color adjustment result and the color adjustment reference result, i.e., an L2 Loss of image feature of each model training of the initial color enhancement model, ∥Gauss(Î)−Gauss(I_gt1)∥₂, is an L2 norm of a difference between the Gaussian filter output result of the color adjustment result of each model training of the initial color enhancement model and the Gaussian filter output result of the color adjustment reference result, i.e., a color Loss of each model training: a is a weight of the L2 loss, b is a weight of the color loss; and both of a and b may be preset according to actual training needs or empirical values.

In some embodiments, a network optimizer, such as Adam optimizer, may be used in the training process of the model, to ensure that the output result (color adjustment result) of each model training is closer to the target (color adjustment reference result) of the model training.

In the embodiment of the present disclosure, the color enhancement model may be trained, by adopting different color enhancement data pairs for different requirements on styles of different video content types, to obtain a set of models, which perform color adjustment on the video frame according to different requirements on styles of different video content types to generate different color toning effects, so that a color enhancement model pool is obtained, to be used for performing automatic color adjustment on video frames of a video to be processed according to the color enhancement model pool, thereby improving the efficiency of color adjustment.

FIG. 9 illustrates a diagram of a specific architecture of a content analysis model in an embodiment of the present disclosure. As shown in FIG. 9, in some embodiments, the content analysis model includes a convolutional network of K number of stacked convolutional structures 91 connected in sequence and two fully connected layers.

Each of the preceding K−1 number of stacked convolutional structures 91 includes a convolutional layer 911, an activation layer (Rectified Linear Unit, ReLU) 912 and a pooling layer (Max Pooling) 913. The K^thstacked convolutional structure 92 includes a convolutional layer 911 and an activation layer 912. The two fully connected layers are a fully connected layer 93 and a fully connected layer 94.

In the embodiment of the present disclosure, the activation layer 912 may be used to perform nonlinear mapping on an output result of the corresponding convolutional layer, to accelerate the convergence speed of the model. The pooling layer 913 may be used to perform under sampling on the image, and dimensions of the image feature may be reduced through the under sampling, so that accuracy may be improved, and over-fitting may be avoided.

In FIG. 9, as an example, a size of the input image is 256×256×3 (length, width, and height), and a size of the convolutional layer in each of the stacked convolutional structures 91 in the content analysis model may be 3×3. Taking K=5 as an example, for the five stacked convolutional structures 91 connected in sequence, a size of the input image may be scaled to 128×128×32 in the 1^ststacked convolutional structure 91, to 64×64×64 in the 2^ndstacked convolutional structure 91, to 32×32×128 in the 3^rdstacked convolutional structure 91, to 16×16×256 in the 4^thstacked convolutional structure 91, and to 8×8×512 in the 5^thstacked convolutional structure 91.

With continued reference to FIG. 9, the fully-connected layer 93 converts the output of the last stacked convolutional structure 91 into a one-dimensional vector with a specified dimension (e.g., 1×1024), and the fully-connected layer 94 is used to convert the one-dimensional vector into a one-dimensional vector with a dimension of L, where L is a total number of models in the color enhancement model pool.

In some embodiments, the color enhancement model pool in FIG. 9 has the same model structure as the color enhancement model pool described in the above embodiments with reference to FIGS. 4a and 4b.

In the embodiment of the present disclosure, the stacked convolutional network structure in the content analysis model may analyze and extract image feature in a manner of stacking convolutional layers. Feature extraction is performed by the convolutional layers, and an output of the final convolutional layer is converted into L number of weights through the two fully connected layers, where L is an integer greater than or equal to 1. Each weight is multiplied by the output result of a corresponding color enhancement model, and the multiplication results are subjected to the feature addition in the corresponding feature channel, to generate a final color enhancement result, so that different weights are assigned to different color enhancement models by analyzing the content of the video frame, thereby achieving the purpose of performing an adaptive color adjustment according to the content.

In some embodiments, the content analysis model is a trained content analysis model obtained by training an initial content analysis model based on each color enhancement model in a pre-trained color enhancement model pool and using image data pairs of different styles. The image data pair includes an un-toned image block and a toned image block with a same image size, which are selected from an un-toned image and a second toned image, respectively, at a same position in image, where the second toned image is an image pre-acquired by toning the un-toned image based on different video content types.

In this embodiment, the model structure of the initial content analysis model is the same as the model structure of the content analysis model. Initial values of the model parameters of the initial content analysis model may be random values of parameters of the model or empirical values of parameters of the model. The second toned image may be a pre-acquired image subjected to color adjustment performed by a colorist. The second toned image is different from the first toned image in the above-described embodiments.

The training data of the content analysis model includes image blocks of the same image size selected from the un-toned image and the second toned image at the same position in image. Illustratively, an image block with a size of 256×256 at a certain position in an image may be randomly extracted and input to the network for training, during the training process of content analysis model. The manner of random extraction may ensure that the model does not over-fit a certain style.

It should be understood that the size of the image block required in the training process may be selected according to actual needs, which is not particularly limited in the embodiments of the present disclosure.

In some embodiments, training an initial content analysis model includes: processing the un-toned image block selected from the un-toned images using each color enhancement model in a pre-trained color enhancement model pool, to obtain the color adjustment result of each color enhancement model on the un-toned image block; processing the un-toned image blocks using the initial content analysis model, to obtain a predetermined number of weights, where the predetermined number is the same as the number of color enhancement models in the color enhancement model pool: performing weighted fusion processing on the color adjustment results of all the color enhancement models on the un-toned image block, according to the predetermined number of weights, to obtain a color enhancement result of the un-toned image block, and acquiring a toned image block corresponding to the un-toned image block from the image data pair, as the color enhancement reference result: constructing a loss function of the initial content analysis model based on the color enhancement result and the color enhancement reference result of the un-toned image block: adjusting model parameters of the initial content analysis model using the loss function of the initial content analysis model, to obtain an updated content analysis model; and taking the updated content analysis model as a new initial content analysis model, and processing the un-toned image block using the new initial content analysis model, until the training process of the initial content analysis model meets a second training end condition, to obtain the trained content analysis model.

The second training end condition includes at least one of the following conditions that a training loss calculated according to the loss function of the initial content analysis model is less than or equal to a set second loss threshold, or the number of training times of the initial content analysis model reaches a set second threshold of number of times.

In this embodiment, for the initial content analysis model, the result of each model training is the color enhancement result of the un-toned image block output by the content analysis model, and the training target is the corresponding color enhancement reference result. The loss function of the initial content analysis model is constructed according to the result of each model training of the initial content analysis model and the training target. It should be understood that the second loss threshold and the second threshold of number of times may be customized according to actual training requirements.

In some embodiments, in the training process of the content analysis model, the loss function of the initial content analysis model is a loss function constructed from the minimized square error of the color enhancement result of the un-toned image block and the color enhancement reference result.

Illustratively, the loss function of the initial content analysis model may be expressed as the following expression (2):

Loss
⁢
2

=



I
^

2

-

I

gt
⁢
2



2

.

(
2
)

In the above expression (2), Loss2 is the Loss function of the initial content analysis model, Î₂is the color enhancement result, I_gt1is the color enhancement reference result, and ∥Î₂−I_gt2∥₂, is an L2 norm of a difference between the color enhancement result and the color enhancement reference result, i.e., an L2 loss of image feature of each model training of the initial content analysis model.

In some embodiments, a network optimizer, such as Adam optimizer, also may be used in the training process of the model, to ensure that the output result (color enhancement result) of each model training is closer to the target (color enhancement reference result) of the model training.

In the training process of the color enhancement model and the content analysis model, a learning rate of model training may be set. The learning rate determines whether and when the loss function can converge to a local minimum. As an example, the learning rates may be the same, e.g., both 1e-4, for the training process of the color enhancement model and the training process of the content analysis model. In other examples, the learning rates in the model training processes of the two models may alternatively be different, and the specific values of the learning rates may be set according to actual situations.

In the embodiment of the present disclosure, the initial content analysis model may be trained, by adopting different color enhancement data pairs for different requirements on styles of different video content types. The trained content analysis model may output a weight of each color enhancement model in the color enhancement model pool corresponding to the video frame. A weighting fusion processing is performed according to the weight of each color enhancement model in the color enhancement model pool corresponding to the video frame and an output result of the corresponding color enhancement model, so that an adaptive color style enhancement processing on the video frame is realized, and the efficiency of video color enhancement is improved.

It is understood that the above-described embodiments of the method of the present disclosure may be combined with each other to form a combined embodiment without departing from the principle logic, which is not repeated herein, limited by the length of the present disclosure. One of ordinary skill in the art will appreciate that in the above methods of the specific embodiments, the specific execution order of the steps should be determined by their function and their possible inherent logic.

In addition, the present disclosure further provides a video processing apparatus, an electronic device, and a computer readable storage medium, which can be used to implement any video processing method provided by the present disclosure, and the corresponding technical solutions and descriptions thereof refers to the corresponding descriptions in the methods section, and are not repeated herein.

An embodiments of the present disclosure provide a video processing apparatus, which is a corresponding apparatus for implementing the video processing method provided by the above embodiments of the present disclosure. The apparatus may be implemented in a software and/or hardware manner, and generally may be integrated in an electronic device.

FIG. 10 is a schematic diagram illustrating a structure of a video processing apparatus in an embodiment of the present disclosure.

Referring to FIG. 10, the video processing apparatus 1000 provided in an embodiment of the present disclosure includes the following devices.

An acquisition device 1010 is used to acquire a video frame to be processed.

A processing device 1020 is used to perform color adjustment on the video frame using at least one color enhancement model in a color enhancement model pool, to obtain a video output frame subjected to color adjustment.

The color enhancement model pool includes a set of models for performing color adjustment on videos of various video content types with different requirements on styles thereof.

According to the video processing device provided by the embodiment of the present disclosure, the color adjustment is performed on the acquired video frame of the video to be processed according to at least one color enhancement model in the model pool, a color toning effect corresponding to the video content type and the requirement on style of the video frame can be automatically generated, and the video processing process can be automatically realized through calling the model by the execution device, so that video color creation can be automatically performed, the labor cost can be saved, and the video processing efficiency can be improved.

In some embodiments, the processing device 1020 may specifically include: a receiving unit for receiving information including the video content type and the requirement on style of the video frame: a selection unit for selecting the color enhancement model corresponding to the video content type and the requirements on style from the color enhancement model pool; and a model processing unit for processing the video frame using the selected color enhancement model, to obtain the video output frame subjected to color adjustment.

In some embodiments, the processing device 1020 may specifically include: a color adjusting unit for processing the video frame using each color enhancement model in the color enhancement model pool, to obtain the video frame subjected to color adjustment output by each color enhancement model: a content analysis unit for processing the video frames based on a content analysis model, to generate a weight of each color enhancement model corresponding to the video frame, where the content analysis model is configured to determine a weight of each color enhancement model in the color enhancement model pool according to the video frame; and a fusion processing unit for performing weighted fusion processing on the video frame subjected to the color adjustment output by each color enhancement model, using the weight of each color enhancement model corresponding to the video frame, to generate the video output frame.

In some embodiments, the color enhancement model includes an image reconstruction branch model and a color enhancement branch model. The model processing unit may specifically include: an extraction sub-unit for performing feature extraction on the video frame, to obtain a first image feature: a filter sub-unit for filtering the video frame and performing feature extraction on the filtered video frame, to obtain a second image feature: a branch model processing sub-unit for inputting the first image feature into the image reconstruction branch model, inputting the second image feature into the color enhancement branch model, to perform image reconstruction and color adjustment according to the first image feature and the second image feature: a result fusion sub-unit for performing feature fusion processing on the output result of the image reconstruction branch model and the output result of the color enhancement branch model, to obtain a fusion result; and a result generating sub-unit for generating the video output frame subjected to color adjustment, according to the fusion result.

In some embodiments, the image reconstruction branch model and the color enhancement branch model each include N number of residual dense blocks RDBs; and the color enhancement model further includes N number of fully connected layers: where N is an integer greater than or equal to 1. The branch model processing sub-unit is specifically used to: acquiring a first RDB in the image reconstruction branch model and a first RDB in the color enhancement branch model as a current RDB of a first branch model and a current RDB of a second branch model, respectively: processing the first image feature through the current RDB of the first branch model, and processing the second image feature through the current RDB of a second branch model, to obtain a feature map output by the current RDB of the first branch model and a low frequency color feature output by the current RDB of the second branch model: processing the low frequency color feature output by the current RDB of the second branch model through a fully connected layer, to obtain a feature vector, and correspondingly multiplying the feature vector by the feature map output by the current RDB of the first branch model, to obtain a feature map subjected to local color feature adjustment: taking the feature map subjected to the local color feature adjustment as a new first image feature, and taking the low frequency color feature output by the current RDB of the second branch model as a new second image feature; and acquiring a next RDB in the image reconstruction branch model and a next RDB in the color enhancement branch model as a new current RDB of the first branch model and a new current RDB of the second branch model, respectively, until the number of acquiring times is equal to N, taking the newly acquired feature map subjected to local adjustment on the color feature as an output result of the image reconstruction branch model, and taking the newly acquired low frequency color feature as an output result of the color enhancement branch model.

In some embodiments, the RDB includes a DenseNet and a local feature fusion layer. The local feature fusion layer is used for performing feature addition on the input feature to the RDB and the output feature of the DenseNet using local residual learning, to obtain the output feature of the RDB. The DenseNet includes M number of dense blocks and one connection block, where M is an integer greater than or equal to 1. An input to the first dense block is the input feature to the RDB; and an input feature to the i^thdense block is the input feature to the RDB and the output features of all dense blocks before the i^thdense block, where i is an integer greater than 1 and less than or equal to M. An input to the connection block is the input feature to the RDB and the output features of the dense blocks.

In some embodiments, each color enhancement model in the color enhancement model pool has the same model structure. The color enhancement model in the color enhancement model pool is a trained color enhancement model obtained through pre-training an initial color enhancement model by adopting different color enhancement data pairs for different color styles of different video content types. The color enhancement data pair includes an un-toned image block and a toned image block with a same image size, which are selected from an un-toned image and a first toned image, respectively, at a same position in image, where the first toned image is an image pre-acquired by toning the un-toned image based on the color style characteristics of the image.

In some embodiments, the video processing apparatus 1000 further includes a color enhancement model training module for training an initial color enhancement model. The color enhancement training module includes: a first adjustment result obtaining unit for processing an un-toned image block selected from the un-toned image using an initial color enhancement model, to obtain a color adjustment result of the un-toned image block: a first reference result acquiring unit for acquiring a toned image block corresponding to the un-toned image block from the color enhancement data pair, as a color adjustment reference result: a first loss function construction unit for constructing a loss function of the initial color enhancement model based on the color adjustment result and the color adjustment reference result: a first parameter adjusting unit for adjusting the model parameters of the initial color enhancement model using the loss function of the initial color enhancement model, to obtain an updated color enhancement model; and a first iterative training unit for taking the updated color enhancement model as a new initial color enhancement model, and processing the un-toned image block selected from the un-toned image using the new initial color enhancement model, until the training process of the initial color enhancement model meets a first training end condition, to obtain the trained color enhancement model, where the first training end condition includes at least one of the following conditions that a training loss calculated according to the loss function of the initial color enhancement model is less than or equal to a set first loss threshold, or the number of training times of the initial color enhancement model reaches a set first threshold of number of times.

In some embodiments, when the color enhancement training module is used for constructing the loss function of the initial color enhancement model based on the color adjustment result and the color adjustment reference result, the color enhancement training module is specifically used for calculating a minimum square error between the color adjustment result and the color adjustment reference result, to obtain a minimum square error loss; calculating a Gaussian filter output result of the color adjustment result and a Gaussian filter output result of the color adjustment reference result, to obtain the color loss; and performing weighted summation on the minimized square error loss and the color loss, to generate the loss function of the initial color enhancement model.

In some embodiments, the content analysis model includes a convolutional network of K number of stacked convolutional structures connected in sequence and two fully connected layers. Each of the preceding K−1 number of stacked convolutional structures includes a convolutional layer, an activation layer and a pooling layer. The K^thstacked convolutional structure includes a convolutional layer and an activation layer.

In some embodiments, the content analysis model is a trained content analysis model obtained by training an initial content analysis model based on each color enhancement model in a pre-trained color enhancement model pool and using image data pairs of different styles. The image data pair includes an un-toned image block and a toned image block with a same image size, which are selected from an un-toned image and a second toned image, respectively, at a same position in image, where the second toned image is an image pre-acquired by toning the un-toned image based on different video content types.

In some embodiments, the video processing apparatus 1000 further includes a content analysis model training module for training an initial content analysis model. The content analysis model training module includes: a second adjustment result obtaining unit for processing the un-toned image block selected from the un-toned image using each color enhancement model in a pre-trained color enhancement model pool, to obtain the color adjustment result of each color enhancement model on the un-toned image block; a weight obtaining unit for processing the un-toned image block using the initial content analysis model, to obtain a predetermined number of weights, where the predetermined number is the same as the number of color enhancement models in the color enhancement model pool; an enhancement result obtaining unit for performing weighting fusion processing on the color adjustment result of each color enhancement model on the un-toned image block, according to the predetermined number of weights, to obtain the color enhancement result of the un-toned image block; a second reference result acquiring unit for acquiring a toned image block corresponding to the un-toned image block from the image data pair, as the color enhancement reference result; a second loss function construction unit for constructing a loss function of the initial content analysis model based on the color enhancement result and the color enhancement reference result of the un-toned image block; a second parameter adjusting unit for adjusting the model parameters of the initial content analysis model using the loss function of the initial content analysis model, to obtain an updated content analysis model; and a second iterative training unit for taking the updated content analysis model as a new initial content analysis model, and processing the un-toned image block using the new initial content analysis model, until the training process of the initial content analysis model meets a second training end condition, to obtain the trained content analysis model, where the second training end condition includes at least one of the following conditions that a training loss calculated according to the loss function of the initial content analysis model is less than or equal to a set second loss threshold, or the number of training times of the initial content analysis model reaches a set second threshold of number of times.

In some embodiments, the loss function of the initial content analysis model is a loss function constructed from the minimized square error of the color enhancement result of the un-toned image block and the color enhancement reference result.

According to the video processing apparatus in the embodiment of the present disclosure, the input video frame may be processed using the color enhancement model and the content analysis model in the color enhancement model pool, so that each color enhancement model in the color enhancement model pool is assigned a weight, according to the weight corresponding to each color enhancement model in the color enhancement model pool output by the content analysis model. Finally, a color adjustment result obtained by performing weighted fusion on the processing results of all the color enhancement models in the color enhancement model pool is output, and the color adjustment result is based on the analysis on the video content type of the video to be processed, so that a video color adjustment result better conforming to the video content type is obtained, thereby automatically adjusting the video to be processed based on the weights assigned by the content analysis model, meeting the requirement on color style of the corresponding video content type, and improving the data processing efficiency of the video color adjustment.