100 or the images that it is restoring). In other words, the filter kernel(s) 121 may be kernels typically used in digital signal processing of images and may be convolved over padded input data so that a filter kernel's output data has a same dimension as its input data.

The filter kernels 121 may be identified through respective filter parameter. For example, a filter parameter may include the type and characteristics of an associated filter. The identified characteristics of the filter may include a filtering factor, the size of the filter, and/or the like. For example, the filter type may be a Gaussian filter, the filtering factor may be σ and μ, and the size of the filter may be w*h. The σ factor may be a standard deviation, the μ factor may be an average, w may be a width, and h may be a height. As noted, the image processing apparatus 100 may use a plurality of filter kernels, including the filter kernel 121, for image restoration. The number of filter kernels may be the same as the number of layers of the image restoration model 110. Alternatively, the number of filter kernels may be greater or less than the number of layers of the image restoration model 110.

The image processing apparatus 100 may substantially reduce an operation quantity and memory needed for image restoration by decreasing the number of layers of the image restoration model 110 and by supplementing the image restoration model 110 with the filter kernel 121 that is based on signal processing. The image processing apparatus 100, despite using a DNN with less layers than a typical DNN (which is implemented without a limit on the number of layers) may provide better or at least similar image restoration performance than a typical DNN. In addition, the DNN with less layers may enable faster image restoration with less memory than the typical DNN. Accordingly, the image processing apparatus 100 may be applied to low-power or real-time applications, for example.

FIG. 2 illustrates an example of an extracting operation based on a filter kernel and a network operation of an image restoration model. Referring to FIG. 2, an image restoration model 210 may include first, second, and nth layers 211, 212, and 213 (i.e., two or more layers). An image processing apparatus may execute the image restoration model 210 and generate an output image 202 corresponding to an input image 201. The image processing apparatus may extract first and second auxiliary data 2213 and 2223 respectively from first and second target data 2211 and 2221 by using first and second filter kernels 2212 and 2222 and may transform at least some of first and second input data 2111 and 2121 of the first, second, and nth layers 211, 212, and 213 by respectively using the first and second auxiliary data 2213 and 2223. The first and second target data 2211 and 2221 may include at least some of the input image 201, the first and second input data 2111 and 2121, and new first input data 2112. Although not illustrated in FIG. 2, the image restoration model 210 may further include a connection (e.g., a skip connection) for summing the input image 201 and the output image 202. In this case, a summing result may correspond to a final output of the image restoration model 210.

More specifically, the image processing apparatus may perform an extracting operation 221, based on the first target data 2211 and the first filter kernel 2212. The first target data 2211 may include at least some of the input image 201 and the first input data 2111. The extracting operation 221 may include a filtering operation by using the first filter kernel 2212. For example, the filtering operation may include convolving the first filter kernel 2212 over first target data 2211. The image processing apparatus may filter the first target data 2211 with the first filter kernel 2212 and generate a first filtering result. If a second (or more) extracting operation 222 is included, the second extracting operation 222 may be performed in a like manner.

The first auxiliary data 2213 may be the first filtering result or a result of an additional operation between the first target data 2211 and the first filtering result. For example, the first auxiliary data 2213 may be a difference between the first target data 2211 and the first filtering result. For example, when the first filter kernel 2212 is a low pass filter (LPF) (e.g., a Gaussian LPF), the first filtering result may correspond to a low-frequency element of the first target data 2211 and the additional operation result (the first auxiliary data 2213) may correspond to a high-frequency element of the first target data 2211.

The image processing apparatus may fuse (e.g., combine) the first auxiliary data 2213 with the first input data 2111 to determine the new first input data 2112. For example, fusion may be performed through concatenation or other methods of combining (concatenation may involve adding the auxiliary data as an additional feature map or image to be inputted to a network layer of an image, e.g., as an additional part of an input volume). The first auxiliary data 2213 may be merged into an additional channel of the first input data 2111 through concatenation in a channel dimension and may serve as a basis for determining the new first input data 2112. Without the fusion, the same as the first input data 2111 serves as the new first input data 2112 (i.e., the first input data 2111 is input to the first layer 211).

The image processing apparatus may execute the first layer 211 to generate the second input data 2121 as an output of the first layer 211. Executing the image restoration model 210 or the first, second, and/or nth layers 211, 212, and 213 may be involve processing a network operation related to the image restoration model 210 or the first, second, and/or nth layers 211, 212, and 213. For example, any layer of the first, second, and/or nth layers 211, 212, and 213 may be a convolutional layer, and the image processing apparatus may perform a convolution operation between the layer and a weight kernel of the layer and determine an output of the layer. Alternatively, when the layer further requires an activation operation (e.g., a rectified linear unit (ReLu)) in addition to the convolution operation, the image processing apparatus may perform the activation operation on a convolution operation result to generate an output of the layer.

The image processing apparatus may process the second layer 212 responsive to an operation on the first layer 211. The image processing apparatus may determine the second auxiliary data 2223 through an extracting operation 222 based on the second target data 2221 and the second filter kernel 2222 and determine new second input data 2122 by fusing the second auxiliary data 2223 with the second input data 2121. For example, the extracting operation 222 may use input data of the first layer 211 and/or output data of the first layer 211 as the second target data 2221. In this case, the input data of the first layer 211 may be the new first input data 2112 and the output data of the first layer 211 may be the second input data 2121. The image processing apparatus may execute the second layer 212 with the new second input data 2122. The extracting and fusing of the first and second auxiliary data 2213 and 2223 may be further performed on several subsequent layers, such as the nth layer 213, and accordingly, the output image 202 may be determined.

When the first and second target data 2211 and 2221 correspond to the input image 201, the first and second auxiliary data 2213 and 2223 may be generated in parallel. In this case, a hardware configuration for generating the first and second auxiliary data 2213 and 2223 in parallel may be provided. The hardware configuration may serve as an accelerator for generating the first and second auxiliary data 2213 and 2223 in parallel, and accordingly, an image restoration time may decrease. A structure of deriving auxiliary data by using the first and second filter kernels 2212 and 2222 may not correspond to a DNN's cascade structure. Therefore, when the input image 201 is the first and second target data 2211 and 2221, a process of generating the first and second auxiliary data 2213 and 2223 may be parallelized (the input image 201 being available at the onset). For example, the first and second auxiliary data 2213 and 2223 may be completely prepared before or at the beginning of executing the image restoration model 210. Because the generation of the first and second auxiliary data 2213 and 2223 may not delay the execution of the image restoration model 210, a time of deriving the output image 202 may decrease.

FIG. 3 illustrates examples of a fusion pattern of auxiliary data. FIG. 2 illustrates an example of applying the first and second auxiliary data 2213 and 2223 respectively to the first and second layers 211 and 212. Referring to FIG. 3, auxiliary data 300 may fuse (e.g., join, combine), in various patterns, with various positions of an image restoration model 310. The auxiliary data 300 may include a plurality of first, second, third, and fourth auxiliary data 301, 302, 303, and 304. The first, second, third, and fourth auxiliary data 301, 302, 303, and 304 may be generated by using respective different filter kernels, or at least some of the first, second, third, and fourth auxiliary data 301, 302, 303, and 304 may be generated by using the same filter kernel. When using different filter kernels, the number of filter kernels may be the same as the number of pieces of the first, second, third, and fourth auxiliary data 301, 302, 303, and 304. The first, second, third, and fourth auxiliary data 301, 302, 303, and 304 may be respectively input to at least some of inputs of initial, intermediate, intermediate, and final layers 311, 312, 313, and 314. Accordingly, there may be various fusion patterns. For example, each of the first, second, third, and fourth auxiliary data 301, 302, 303, and 304 may be evenly distributed and fused (combined, joined, etc.) into each input of the initial, intermediate, intermediate, and final layers 311, 312, 313, and 314, or two or more of first, second, third, and fourth auxiliary data 301, 302, 303, and 304 may be distributed and fused into a certain input thereof.

In an example, the auxiliary data 300 may include a same number of the auxiliary datum as the number of layers in the image restoration model 310. For example the auxiliary data 300 may include first, second, third, and fourth auxiliary data 301, 302, 303, and 304 and the image restoration model 310 may include initial, intermediate, intermediate, and final layers 311, 312, 313, and 314. In such example, the first, second, third, and fourth auxiliary data 301, 302, 303, and 304 may respectively fuse (e.g. join) with the initial, intermediate, intermediate, and final layers 311, 312, 313, and 314. An intermediate layer may correspond to a hidden layer, and a final layer may correspond to an output layer.

In another example, two or more of the first, second, third, and fourth auxiliary data 301, 302, 303, and 304, that is, a plurality of pieces of auxiliary data, may fuse with at least one input of any layer of the image restoration model 310. For example, the first auxiliary data 301 and the second auxiliary data 302 may fuse (combine or join) with an input of the initial layer 311, and the third auxiliary data 303 and the fourth auxiliary data 304 may fuse with an input of the final layer 314. In another example, the first, second, third, and fourth auxiliary data 301, 302, 303, and 304 may fuse with an input of the initial layer 311. There may be various other examples of a fusion point, other than the foregoing examples. A fusion pattern which is suitable for a current restoration task (e.g., noise removal, blur removal, image enhancement, etc.), may be adopted for a restoration process of the current restoration task. In other words, a fusion pattern may be any combination(s) of pieces of auxiliary data with fused with any input(s) of restoration model layers.

FIG. 4A illustrates an example of a network structure in which auxiliary data fuses with input data of an initial layer, an intermediate layer, and a final layer. Referring to FIG. 4A, an image processing apparatus may determine first auxiliary data, based on an extracting operation 431 between an input image 401 and a first filter kernel of filter kernels 420. For example, the image processing apparatus may determine a first filtering result by filtering the input image 401 through the first filter kernel and determine the first auxiliary data by subtracting the first filtering result from the input image 401. For example, the first filter kernel may correspond to an LPF (e.g., a Gaussian LPF), the first filtering result may correspond to a low-frequency element, and the first auxiliary data may correspond to a high-frequency element.

The image processing apparatus may determine new first input data by fusing the first auxiliary data with first input data of a first layer 411. When the first layer 411 is an initial layer, the first input data of the first layer 411 may be the input image 401. The image processing apparatus may concatenate the first auxiliary data with the first input data in a channel dimension. When the number of channels of the first input data is c1, the number of channels of the new first input data may be c1+1. The image processing apparatus may execute the first layer 411 by inputting the new first input data into the first layer 411. For example, the first layer 411 may be a convolutional layer.

The image processing apparatus may determine second auxiliary data by using the input image 401 and a second filter kernel of the filter kernels 420 (to generate second auxiliary data) and determine new second input data by fusing the second auxiliary data with second input data of a second layer 412; the new second input data is input to the second layer 412. The image processing apparatus may execute a third layer 413, based on third auxiliary data, which is determined through an extracting operation 433 and may additionally perform the foregoing operation on other filter kernels and other layers. The second layer 412 and the third layer 413 may each include a respective convolutional layer and an activation layer (as used herein “activation layer” may refer to an activation function). A final layer of an image restoration model 410 may also be a convolutional layer. Accordingly, an output image 402 may correspond to a degradation element (e.g., a high-frequency element) of the input image 401. The image processing apparatus may generate a restored image of the input image 401 by subtracting the output image 402 from the input image 401.

Different filter parameters may be stored and respectively associated with the filter kernels 420 that they represent, which may facilitate selection of different filter kernels for different tasks. For example, a filter parameter may include the type and characteristics of a corresponding filter. The characteristics of the filter may include a filtering factor, the size of the filter, and/or the like. The filter parameters of filter kernels may be set based on the operational characteristics of the first, second, and third layers 411, 412, and 413. For example, when each of the first, second, and third layers 411, 412, and 413 performs a convolution operation, a receptive field may gradually increase toward a following layer of the first, second, and third layers 411, 412, and 413. Accordingly, the filter parameter may be set such that a filter size may increase toward a following layer of the first, second, and third layers 411, 412, and 413. For example, when defining that the filter size (filter parameter) of the first filter kernel is k1, the filter size of the second filter kernel is k2, and the filter size of the third filter kernel is k3, the filter size may be k1<k2<k3. In this case, the filter size of filter parameters of the filter kernels 420 may affect another element (e.g., a filtering factor). For example, when the filter kernels 420 corresponds to a Gaussian filter, the filter size may be set to increase toward a following layer, and accordingly, a filtering factor (e.g., σ) of the filter kernels 420 may be changed (e.g., an increase of σ). As a result, a receptive field may gradually increase toward a following layer.

Although the examples of FIGS. 4A, 5, and 6 may use the input image 401 as filter kernel target data, other examples may use input data and/or output data of the first, second, and third layers 411, 412, and 413 of the image restoration model 410, instead of the input image 401, as the target data for the filter kernels. Such examples are described next with reference to FIG. 4B.

When the input image 401 is used as filter kernel target data, extracting operations 431, 432, and 433 may be parallelized. The image processing apparatus may perform the extracting operations 431, 432, and 433 in parallel and prepare auxiliary data for the first, second, and third layers 411, 412, and 413 and may use the prepared auxiliary data according to an execution sequence of the first, second, and third layers 411, 412, and 413. For example, when executing the third layer 413, the image processing apparatus may use prepared third auxiliary data to determine new third input data of the third layer 413. The term “prepared” means that auxiliary data (e.g., the third auxiliary data) of a current layer (e.g., the third layer 413) is prepared while (or before) the previous layer (e.g., the second layer 412) is being executed. Accordingly, the auxiliary data of the current layer may be readily available when an operation result of the previous layer is output, and thus, a time delay due to the generation of the auxiliary data may not occur.

FIG. 4B illustrates an example of a network structure in which input data of a layer and/or output data of the layer is used as target data. Referring to FIG. 4B, extracting operations 435, 436, and 437 may use input data and/or output data of first, second, and third layers 411, 412, and 413, other than an input image 401, as the target data. For example, for the extracting operation 435, an image processing apparatus may determine a second filtering result by filtering first input data 4111 of the first layer 411 with a second filter kernel of filter kernels 420 and may determine second auxiliary data by subtracting the second filtering result from the first input data 4111. The first input data 4111 may correspond to new first input data of the first layer 411 (see the description of “new” with reference to FIG. 2). The image processing apparatus may determine new second input data by fusing the second auxiliary data with second input data of the second layer 412. The second input data may correspond to output data 4112 of the first layer 411.

In another example, the image processing apparatus may determine the second filtering result by (i) filtering the output data 4112 of the first layer 411 with the second filter kernel or by (ii) filtering the input data 4111 and the output data 4112 of the first layer 411 with the second filter kernel. The extracting operations 436 and 437 may respectively use input data and/or output data of the second layer 412 and the third layer 413 as their respective target data.

When input data and/or output data are used as the target data, a sub-neural network may be used for the extracting operations 435, 436, and 437. For example, the sub-neural network of an extracting operation may include a convolutional layer (e.g., a 1*1 convolutional layer). The sub-neural network may decrease a channel dimension of the input data and/or the output data when it is used as target data for a filtering/extraction operation, and in the decreased channel dimension, a filtering/extraction operation by using the filter kernels 420 may be performed. The sub-neural network may decrease an operation quantity of the extracting operations 435, 436, and 437 and related subsequent operations.

FIG. 5 illustrates an example of a network structure in which auxiliary data (e.g., extracted with filter kernels) fuses (joins or combines) with input data of an initial layer and a final layer. Referring to FIG. 5, an image processing apparatus may determine first auxiliary data, based on an extracting operation 531 between an input image 501 and a first filter kernel of filter kernels 520. For example, the image processing apparatus may determine a first filtering result by filtering the input image 501 through the first filter kernel and determine the first auxiliary data by subtracting the first filtering result from the input image 501. The image processing apparatus may determine second auxiliary data, based on an extracting operation 532 between the input image 501 and a second filter kernel of the filter kernels 520.

The image processing apparatus may determine new first input data by fusing the first auxiliary data and the second auxiliary data with first input data of a first layer 511. When the first layer 511 is an initial layer of the image restoration model 510, the first input data of the first layer 511 may be the input image 501. For example, the image processing apparatus may concatenate the first auxiliary data and the second auxiliary data with the first input data in a channel dimension. When the number of channels of the first input data is c1, the number of channels of the new first input data may be c1 plus the number of channels of the first input data. For example, if the auxiliary data is black-and-white image data, then the number of channels of the new first input data may be c1+2 (1 channel for each the outputs of the extracting operations 531 and 532). If the auxiliar data is 3-channel color data, the number of channels of the new first input data may be c1+6 (3 channels for each of the outputs of the extracting operations 531 and 532). However, c1+2 and c1+6 are merely examples, and the number of channels is not limited thereto. The image processing apparatus may execute the first layer 511 by inputting the new first input data into the first layer 511. For example, the first layer 511 may be a convolutional layer (for example, when the image restoration model 501 includes a CNN).

The image processing apparatus may determine third auxiliary data by using the input image 501 and a third filter kernel of the filter kernels 520, may determine fourth auxiliary data by using the input image 501 and a fourth filter kernel of the filter kernels 520, may determine new nth input data by fusing the third auxiliary data and the fourth auxiliary data with nth input data of an nth layer 513, and may input the new nth input data to the nth layer 513. An n−1th layer 512 may include a convolutional layer and an activation layer, and the nth layer 513 may correspond to a convolutional layer. The image processing apparatus may generate a restored image of the input image 501 by subtracting an output image 502 from the input image 501 (as used herein “subtract” is operand-order agnostic; “subtracting” any two operands also means finding a difference between the two operands).

The filter kernels 520 may have different filter parameters (records of information about the filter kernels 520, stored, e.g., in memory and available before/during image restoration). For example, a filter parameter may include the type and characteristics of a correspondingly associated filter. The characteristics of the filter stored in the filter parameter (filter record) may include information about the associated filter kernel, for example a filtering factor, the size of the filter, and/or the like. The filter parameter(s) may be set based on the operational characteristics of the layers 511, 512, and 513. For example, a receptive field may gradually increase toward a following layer of the layers 511, 512, and 513 and the filter parameter may be set such that a filter size increases toward the following layer of the layers 511, 512, and 513. For example, when the filter sizes of first, second, third, and fourth filter kernels are respectively k1, k2, k3, and k4, the filter sizes may be k1=k2<k3=k4. In this case, a change in the filter size may affect an element (e.g., a filtering factor) other than the filter size among the filter parameters of the filter kernels 520.

Since, in the example of FIG. 5 at least (and possibly others), the input image 501 is used as the target data, extracting operations 531, 532, and 533 may be fully or partly parallelized (with respect each other and/or with respect to executions of model layers). Although two pieces of auxiliary data based on two respective extracting operations 531 and 532 fuse with input data of the first layer 511 and two pieces of auxiliary data based on two respective extracting operations 533 and 534 fuse with input data of the nth layer 513 in the examples described with reference to FIG. 5, one piece or three or more pieces of auxiliary data may fuse with pieces of input data of the first layer 511 and/or the nth layer 513. In addition, a different number of pieces of auxiliary data based on an extracting operation may be applied to the first layer 511 and the nth layer 513. For example, d1 pieces of auxiliary data may fuse with the input data of the first layer 511 and d2 pieces of auxiliary data may fuse with the input data of the nth layer 513. In this case, d1 and d2 may have different values.

FIG. 6 illustrates an example of a network structure in which auxiliary data fuses with input data of an initial layer. Referring to FIG. 6, an image processing apparatus may determine first auxiliary data, based on an extracting operation 631 between an input image 601 and a first filter kernel of filter kernels 620, may determine second auxiliary data, based on an extracting operation 632 between the input image 601 and a second filter kernel of the filter kernels 620, and may determine third auxiliary data, based on an extracting operation 633 between the input image 601 and a third filter kernel of the filter kernels 620.

The image processing apparatus may determine new first input data by fusing the first auxiliary data, the second auxiliary data, and the third auxiliary data with first input data of a first layer 611. When the first layer 611 is an initial layer, first input data of the first layer 611 may be the input image 601. For example, the image processing apparatus may concatenate the first auxiliary data, the second auxiliary data, and the third auxiliary data with the first input data in a channel direction. When the number of channels of the first input data is c1, the number of channels of the new first input data may be c1+3 or c1+9. c1+3 may represent black-and-white data and c1+9 may represent color data. However, c1+3 and c1+9 may just be examples, and the number of channels is not limited thereto. The image processing apparatus may execute the first layer 611 by inputting the new first input data into the first layer 611. For example, the first layer 611 may correspond to a convolutional layer. A second layer 612 and a third layer 613 may each include a respective convolutional layer and a respective activation layer (activation function). When an output image 602 is generated through the execution of the first, second, and third layers 611, 612, and 613, the image processing apparatus may generate a restored image of the input image 601 by subtracting the output image 602 from the input image 601. Although pieces of auxiliary data based on three extracting operations 631, 632, and 633 fuse with input data of the first layer 611 in the examples described with reference to FIG. 6, one, two, four, or more pieces of auxiliary data may fuse with the first layer 611.

FIG. 7 illustrates examples of points at which an image restoration model may be applied in an image processing pipeline. In an example, an image restoration process using an image restoration model 730 and filter kernels 740 may be divided and processed by a sensor-image signal processing (ISP) unit 721 and an application processor (AP)-ISP 722 (the application processor may be a general-purpose processor, for example, a neural processor, or the like). For example, the image restoration process may include at least some of restoration tasks, such as denoising, demosaicing, white balancing, enhancement, and the like, and at least some of the restoration tasks may be divided and processed by the sensor-ISP 721 and the AP-ISP 722. For example, when the image restoration model 730 includes n layers and n=m1+m2, first m1 layers may be processed by the sensor-ISP 721 and next m2 layers may be processed by the AP-ISP 722. Although FIG. 7 illustrates that the AP-ISP 722 performs the restoration tasks, such as demosaicing, white balancing, and enhancement, at least some of the restoration tasks may be divided and processed by the sensor-ISP 721 and the AP-ISP 722.

For example, the sensor-ISP 721 may generate a Bayer image 706, based on sensor data according to a Bayer pattern 701, and the AP-ISP 722 may generate a red, green, and blue (RGB) image 707 based on the Bayer image 706. Although FIG. 7 illustrates an example of using the Bayer pattern 701, there may be other examples using different color filters, such as a Tetra pattern or a Nona pattern.

The image restoration model 730 may improve the quality of the RGB image 707 by improving the performance of the restoration tasks, such as denoising, demosaicing, white balancing, and enhancement. Combining the image restoration model 730 with the filter kernels 740 may allow a decreased number of layers of the image restoration model 730, may increase the speed of image restoration, and may decrease memory use. In addition, load balancing/sharing using the sensor-ISP 721 and the AP-ISP 722 may optimize the restoration tasks. Accordingly, the image processing apparatus 100 may be applied to low-power or real-time applications.

FIG. 8 illustrates an example of a configuration of an image processing apparatus. Referring to FIG. 8, an image processing apparatus 800 may include a primary ISP block 810 and a secondary ISP block 820. The primary ISP block 810 and the secondary ISP block 820 may determine auxiliary data corresponding to a plurality of filter kernels by filtering target data with the plurality of filter kernels, may determine new input data by fusing the auxiliary data with at least some of input data of a plurality of layers of a neural network-based image restoration model, and may divide and perform an image restoration operation of generating, based on the new input data, a restored image of an input image by executing the neural network-based image restoration model. For example, the image processing apparatus 800 may execute a certain layer or some layers of an image restoration model specialized in a certain image restoration task by using the primary ISP block 810 and may execute the next layer or the other layers of the image restoration model by using the secondary ISP block 820. The primary ISP block 810 may correspond to the sensor-ISP 721 of FIG. 7 and the secondary ISP block 820 may correspond to the AP-ISP 722 of FIG. 7. In addition, the descriptions provided with reference to FIGS. 1 to 7 and 9 to 13 may be applied to the image processing apparatus 800. In some embodiments, the primary ISP block 810 and the second ISP block 820 are respective separate processors, which may each be configured for image signal processing.

FIG. 9 illustrates an example of restoration by using a network structure in which auxiliary data fuses with input data of an initial layer and a final layer. Referring to FIG. 9, in operation 910 an image processing apparatus may receive an input image in a degraded state in operation 910, and in operation 920 the apparatus may obtain a plurality of filtering results from the input image in parallel. The image processing apparatus may fuse an ith filtering result with ith input data in operation 930 and may execute an ith layer of an image restoration model by using (as input to the ith layer) a fusion result in operation 940. In operation 950, the image processing apparatus may compare i with N, and when i is not greater than N, may increment i (increase i by 1) and then perform operations 930 and 940 again. N may be the number of filter kernels and the number of layers to which filtering result fusion is applied. When i is greater than N, the image processing apparatus may execute the rest of the layers of the image restoration model in operation 960 and may obtain an output image in operation 970. The restoration operation of FIG. 9 may correspond to the example of FIG. 4.

FIG. 10 illustrates an example of restoration by using a network structure in which auxiliary data fuses with input data of an initial layer and a final layer. Referring to FIG. 10, an image processing apparatus may receive an input image in a degraded state in operation 1010 and may obtain a plurality of filtering results from the input image in parallel in operation 1020. The image processing apparatus may fuse input data of a first layer of an image restoration model and some of the plurality of filtering results in operation 1030 and execute the image restoration model by using a fusion result in operation 1040. In this case, the last layer of the image restoration model may not be executed. In operation 1050, the image processing apparatus may fuse input data of the last layer of the image restoration model and the rest of the plurality of filtering results. The image processing apparatus may obtain an output image in operation 1060 as a result of executing the last layer of the image restoration model by using a fusion result. The restoration operation of FIG. 10 may correspond to the example of FIG. 5.

FIG. 11 illustrates an example of restoration by using a network structure in which auxiliary data fuses with input data of an initial layer. Referring to FIG. 11, an image processing apparatus may receive an input image in a degraded state in operation 1110 and may obtain a plurality of filtering results from the input image in parallel in operation 1120. The image processing apparatus may fuse input data of a first layer of an image restoration model and all the plurality of filtering results in operation 1130 and may execute the image restoration model by using a fusion result in operation 1140. In operation 1150, the image processing apparatus may obtain an output image. The restoration operation of FIG. 11 may correspond to the example of FIG. 6.

FIG. 12 illustrates an example configuration of an electronic device. Referring to FIG. 12, an electronic device 1200 may include a processor 1210, a memory 1220, a camera 1230, a storage device 1240, an input device 1250, an output device 1260, and a network interface 1270 that may communicate with each other through a communication bus 1280. For example, the electronic apparatus 1200 may be implemented as a mobile device such as a mobile phone, a smartphone, a personal digital assistant (PDA), a netbook, a tablet computer, a laptop computer, and the like, or a wearable device, such as a smart watch, a smart band, smart glasses, and the like, or a home appliance, such as a television (TV), a smart TV, a refrigerator, and the like, or a security device, such as a door lock and the like, or a vehicle, such as an autonomous vehicle, a smart vehicle, and the like, to name some examples. The electronic device 1200 may structurally and/or functionally include the image processing apparatus 100 of FIG. 1 and/or the image processing apparatus 800 of FIG. 8.

The processor 1210 may execute instructions and functions to be executed by the electronic device 1200. For example, the processor 1210 may process instructions stored in the memory 1220 or the storage device 1240. The processor 1210 may perform one or more of the operations described above with reference to FIGS. 1 to 11 and 13. The memory 1220 may include a computer-readable storage medium or a computer-readable storage device (which are not signals per se). The memory 1220 may store the instructions to be executed by the processor 1210 and may store related information while software and/or an application is executed by the electronic device 1200.

The camera 1230 may capture a photo and/or a video. A capturing result may correspond to an input image such as various input images discussed above. The storage device 1240 may include a computer-readable storage medium or a computer-readable storage device (which are not signals per se). The storage device 1240 may store a greater storage capacity than the memory 1220 and may store data in a non-volatile way. For example, the storage device 1240 may include a magnetic hard disk, an optical disc, a flash memory, a floppy disk, or other non-volatile memories.

The input device 1250 may receive an input from a user through a keyboard and a mouse, through a touch input, a voice input, an image input, or the like. For example, the input device 1250 may include a keyboard, a mouse, a touch screen, a microphone, or any other device for detecting the input from the user and transmitting the detected input to the electronic device 1200. The output device 1260 may render an output of the electronic device 1200 to the user through a visual, auditory, or haptic channel. The output device 1260 may include, for example, a display, a touch screen, a speaker, a vibration generator, or any other device that provides the output to the user. The network interface 1270 may communicate with an external device through a wired or wireless network.

FIG. 13 illustrates an example of an image restoration method. Referring to FIG. 13, an image processing apparatus may receive an input image in operation 1310, may determine auxiliary data respectively corresponding to a plurality of filter kernels by filtering target data with the plurality of filter kernels in operation 1320, may determine new input data by fusing the auxiliary data with at least some of input data of a plurality of layers of a neural network-based image restoration model in operation 1330, and may generate, based on the new input data, a restored image of the input image by executing the neural network-based image restoration model in operation 1340.

The target data may include at least some of the input image, at least some of the input data of the plurality of layers, and at least some of output data of the plurality of layers. For example, the target data may be the input image, input data of a layer corresponding to a filter kernel, or input data of the previous layer of the layer corresponding to the filter kernel. The input data of the layer corresponding to the filter kernel may correspond to output data of the previous layer.

Operation 1320 may include determining first auxiliary data by filtering the target data with a first filter kernel of the plurality of filter kernels and determining second auxiliary data by filtering the target data with a second filter kernel of the plurality of filter kernels.

Operation 1330 may include determining new first input data by fusing the first auxiliary data with first input data of a first layer and determining new second input data by fusing the second auxiliary data with second input data of a second layer, in which the first layer corresponds to any one of the plurality of layers of the neural network-based image restoration model and the second layer corresponds to another one of the plurality of layers of the neural network-based image restoration model.

The first filter kernel and the second filter kernel may correspond to a Gaussian filter, the second layer may be closer to the output layer of the neural network-based image restoration model than the first layer, and the size of the second filter kernel may be greater than the size of the first filter kernel.

Operation 1330 may include determining the new first input data by fusing the first auxiliary data and the second auxiliary data with the first input data of the first layer corresponding to any one of the plurality of layers of the neural network-based image restoration model. For example, the first layer may correspond to an initial layer, an intermediate layer, or a final layer of the neural network-based image restoration model.

Determining the first auxiliary data and determining the second auxiliary data may be performed in parallel.

The target data may correspond to (or may be) the input image. Operation 1320 may include defining the plurality of filter kernels having different filter parameters from one another and determining auxiliary data, based on filtering results corresponding to the plurality of filter kernels by filtering the input image with the plurality of filter kernels in parallel. Operation 1340 may include generating, based on new input data according to the auxiliary data, a restored image by sequentially executing the plurality of layers of the neural network-based image restoration model.

The plurality of filter kernels may include the first filter kernel corresponding to a Gaussian filter for the first layer of the plurality of layers. Operation 1320 may include generating a first filtering result by filtering the target data with the first filter kernel and determining the first auxiliary data corresponding to a difference between the target data and the first filtering result. Operation 1330 may include determining the new first input data by concatenating the first auxiliary data with the first input data of the first layer.

Operation 1340 may generate the restored image, based on a difference between an output image of the neural network-based image restoration model and the input image.

Determining the auxiliary data or determining the new input data may be performed by a sensor-ISP, whichever of the two is not performed by the sensor-ISP may instead be performed by an AP-ISP. For example, operations related to certain layers of the neural network-based image restoration model may be performed by the sensor-ISP and operations related to the other layers of the neural network-based image restoration model may be performed by the AP-ISP. In other words, in some embodiments, some of the determining of the auxiliary data and the determining the new input data is performed by a sensor-ISP, and the rest of the determining of the auxiliary data and the determining the new input data is performed by the AP-ISP, and the division of computation between the AP-ISP and sensor-ISP can vary.