The inter-predictor 124 derives predicted motion vector candidates using the motion vectors of the neighboring blocks and determines a predicted motion vector for the motion vector of the current block using the predicted motion vector candidates. Then, a motion vector difference is calculated by subtracting the predicted motion vector from the motion vector of the current block.

The predicted motion vector may be obtained by applying a predefined function (e.g., a function for calculating a median, an average, or the like) to the predicted motion vector candidates. In this case, the video decoding apparatus also knows the predefined function. Since the neighboring blocks used to derive the predicted motion vector candidates have already been encoded and decoded, the video decoding apparatus already knows the motion vectors of the neighboring blocks as well. Accordingly, the video encoding apparatus does not need to encode information for identifying the predicted motion vector candidates. Therefore, in this case, the information about the motion vector difference and the information about the reference picture used to predict the current block are encoded.

The predicted motion vector may be determined by selecting any one of the predicted motion vector candidates. In this case, information for identifying the selected predicted motion vector candidate is further encoded along with the information about the motion vector difference and the information about the reference picture which are to be used to predict the current block.

The subtractor 130 subtracts the prediction block generated by the intra-predictor 122 or the inter-predictor 124 from the current block to generate a residual block.

The transformer 140 may transform residual signals within the residual block. A two-dimensional size of the residual block may be used as a transform unit (hereinafter, “TU”), which is a block size for performing transform. Alternatively, the residual block may be partitioned into a plurality of subblocks, and each subblock may be used as a TU to transform the residual signals within the corresponding subblock.

The transformer 140 may split the residual block into one or more subblocks and may apply the transformation to the one or more subblocks. Thus, the residual values of the transform blocks may be transformed from the sample domain to the frequency domain. In the frequency domain, the transformed blocks are referred to as coefficient blocks or transform blocks containing one or more transform coefficient values. A two-dimensional transform kernel may be used for transformation, and one-dimensional transform kernels may be used for horizontal transformation and vertical transformation, respectively. The transform kernels may be based on a discrete cosine transform (DCT), a discrete sine transform (DST), or the like.

The transformer 140 may transform the residual block in the horizontal direction and the vertical direction individually. For transformation, various types of transform kernels or transform matrices may be used. For example, pairs of transform kernels for horizontal transformation and vertical transformation may be defined as a multiple transform set (MTS). The transformer 140 may select one pair of transform kernels having the best transformation efficiency in the MTS and transform the residual block in the horizontal and vertical directions, respectively. The information (mts_idx) on the transform function pair selected in the MTS is encoded by the entropy encoder 155 and signaled to the video decoding apparatus.

The quantizer 145 quantizes transform coefficients output from the transformer 140 using quantization parameters and outputs the quantized transform coefficients to the entropy encoder 155. For some blocks or frames, the quantizer 145 may directly quantize a related residual block without transformation. The quantizer 145 may apply different quantization coefficients (scaling values) according to the positions of the transform coefficients in a transform block. A matrix of quantized coefficients applied to the two-dimensionally arranged quantized transform coefficients may be encoded and signaled to the video decoding apparatus.

The rearrangement unit 150 may re-sort the coefficient values for the quantized residual value. The rearrangement unit 150 may change the 2-dimensional array of coefficients into a 1-dimensional coefficient sequence through coefficient scanning. For example, the rearrangement unit 150 may scan coefficients from a DC coefficient to a coefficient in a high frequency region using a zig-zag scan or a diagonal scan to output a 1-dimensional coefficient sequence. Depending on the size of the transformation unit and the intra-prediction mode, a vertical scan in which a two-dimensional array of coefficients is scanned in a column direction or a horizontal scan in which two-dimensional block-shaped coefficients are scanned in a row direction may be used instead of the zig-zag scan. In other words, a scan mode to be used may be determined among the zig-zag scan, the diagonal scan, the vertical scan, and the horizontal scan according to the size of the transformation unit and the intra-prediction mode.

The entropy encoder 155 encodes the one-dimensional quantized transform coefficients output from the rearrangement unit 150 using various encoding techniques, such as Context-based Adaptive Binary Arithmetic Code (CABAC) and exponential Golomb, to generate a bitstream.

The entropy encoder 155 encodes information such as a CTU size, a CU split flag, a QT split flag, an MTT splitting type, and an MTT splitting direction, which are associated with block splitting, such that the video decoding apparatus may split the block in the same manner as in the video encoding apparatus. In addition, the entropy encoder 155 encodes information about a prediction type indicating whether the current block is encoded by intra-prediction or inter-prediction. The entropy encoder 155 also encodes intra-prediction information (i.e., information about an intra-prediction mode) or inter-prediction information (a merge index for the merge mode, information about a reference picture index and a motion vector difference for the AMVP mode) according to the prediction type. The entropy encoder 155 also encodes information related to quantization, i.e., information about quantization parameters and information about a quantization matrix.

The inverse quantizer 160 inversely quantizes the quantized transform coefficients output from the quantizer 145 to generate transform coefficients. The inverse transformer 165 transforms the transform coefficients output from the inverse quantizer 160 from the frequency domain to the spatial domain and reconstructs the residual block.

The adder 170 adds the reconstructed residual block to the prediction block generated by the predictor 120 to reconstruct the current block. The samples in the reconstructed current block are used as reference samples in performing intra-prediction of a next block.

The loop filter unit 180 filters the reconstructed samples to reduce blocking artifacts, ringing artifacts, and blurring artifacts generated due to block-based prediction and transformation/quantization. The loop filter unit 180 may include at least one of a deblocking filter 182, a sample adaptive offset (SAO) filter 184, or an adaptive loop filter (ALF) 186.

The deblocking filter 182 filters the boundary between the reconstructed blocks to remove blocking artifacts caused by block-by-block coding/decoding, and the SAO filter 184 performs additional filtering on the deblocking-filtered video. The SAO filter 184 is a filter used to compensate for a difference between a reconstructed sample and an original sample caused by lossy coding and performs filtering in a manner that adding a corresponding offset to each reconstructed sample. The ALF 186 performs filtering on a target sample to be filtered by applying filter coefficients to the target sample and neighboring samples of the target sample. The ALF 186 may divide the samples included in a picture into predetermined groups, and then determine one filter to be applied to a corresponding group to differentially perform filtering on each group. Information about filter coefficients to be used for the ALF may be encoded and signaled to the video decoding apparatus.

The reconstructed blocks filtered through the loop filter unit 180 are stored in the memory 190. Once all blocks in one picture are reconstructed, the reconstructed picture may be used as a reference picture for inter-prediction of blocks in a picture to be encoded next.

FIG. 5 is a functional block diagram of a video decoding apparatus capable of implementing the techniques of the present disclosure. Hereinafter, the video decoding apparatus and elements of the apparatus are described with reference to FIG. 5.

The video decoding apparatus may include an entropy decoder 510, a rearrangement unit 515, an inverse quantizer 520, an inverse transformer 530, a predictor 540, an adder 550, a loop filter unit 560, and a memory 570.

Similar to the video encoding apparatus of FIG. 1, each element of the video decoding apparatus may be implemented in hardware, software, or a combination of hardware and software. Further, the function of each element may be implemented in software, and the microprocessor may be implemented to execute the function of software corresponding to each element.

The entropy decoder 510 determines a current block to be decoded by decoding a bitstream generated by the video encoding apparatus and extracting information related to block splitting and extracts prediction information and information about a residual signal, and the like required to reconstruct the current block.

The entropy decoder 510 extracts information about the CTU size from the sequence parameter set (SPS) or the picture parameter set (PPS), determines the size of the CTU, and splits a picture into CTUs of the determined size. Then, the decoder determines the CTU as the uppermost layer, i.e., the root node of a tree structure and extracts splitting information about the CTU to split the CTU using the tree structure.

For example, when the CTU is split using a QTBTTT structure, a first flag (QT_split_flag) related to splitting of the QT is extracted to split each node into four nodes of a sub-layer. For a node corresponding to the leaf node of the QT, the second flag (MTT_split_flag) and information about a splitting direction (vertical/horizontal) and/or a splitting type (binary/ternary) related to the splitting of the MTT are extracted to split the corresponding leaf node in the MTT structure. Thus, each node below the leaf node of QT is recursively split in a BT or TT structure.

As another example, when a CTU is split using the QTBTTT structure, a CU split flag (split_cu_flag) indicating whether to split a CU may be extracted. When the corresponding block is split, the first flag (QT_split_flag) may be extracted. In the splitting operation, zero or more recursive MTT splitting may occur for each node after zero or more recursive QT splitting. For example, the CTU may directly undergo MTT splitting without the QT splitting or undergo only QT splitting multiple times.

As another example, when the CTU is split using the QTBT structure, the first flag (QT_split_flag) related to QT splitting is extracted, and each node is split into four nodes of a lower layer. Then, a split flag (split_flag) indicating whether a node corresponding to a leaf node of QT is further split in the BT and the splitting direction information are extracted.

Once the current block to be decoded is determined through splitting in the tree structure, the entropy decoder 510 extracts information about a prediction type indicating whether the current block is intra-predicted or inter-predicted. When the prediction type information indicates intra-prediction, the entropy decoder 510 extracts a syntax element for the intra-prediction information (intra-prediction mode) for the current block. When the prediction type information indicates inter-prediction, the entropy decoder 510 extracts a syntax element for the inter-prediction information, i.e., information indicating a motion vector and a reference picture referred to by the motion vector.

The entropy decoder 510 also extracts information about quantized transform coefficients of the current block as information related to quantization and information about residual signals.

The rearrangement unit 515 may change the sequence of the one-dimensional quantized transform coefficients entropy-decoded by the entropy decoder 510 to a 2-dimensional coefficient array (i.e., block) in a reverse order of the coefficient scanning performed by the video encoding apparatus.

The inverse quantizer 520 inversely quantizes the quantized transform coefficients using the quantization parameter. The inverse quantizer 520 may apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in two dimensions. The inverse quantizer 520 may perform inverse quantization by applying a matrix of quantization coefficients (scaling values) from the video encoding apparatus to a two-dimensional array of quantized transform coefficients.

The inverse transformer 530 inversely transforms the inversely quantized transform coefficients from the frequency domain to the spatial domain to reconstruct residual signals, thereby generating a reconstructed residual block for the current block. In addition, when the MTS is applied, the inverse transformer 530 determines transform functions or transform matrices to be applied in the horizontal and vertical directions, respectively, using the MTS information (mts_idx) signaled from the video encoding apparatus, and uses the determined transform functions to inversely transform the transform coefficients in the transform block in the horizontal and vertical directions.

The predictor 540 may include an intra-predictor 542 and an inter-predictor 544. The intra-predictor 542 is activated when the prediction type of the current block is intra-prediction, and the inter-predictor 544 is activated when the prediction type of the current block is inter-prediction.

The intra-predictor 542 determines an intra-prediction mode of the current block among a plurality of intra-prediction modes based on the syntax element for the intra-prediction mode extracted from the entropy decoder 510 and predicts the current block using the reference samples around the current block according to the intra-prediction mode.

The inter-predictor 544 determines a motion vector of the current block and a reference picture referred to by the motion vector using the syntax element for the inter-prediction extracted from the entropy decoder 510 and predicts the current block based on the motion vector and the reference picture.

The inter-predictor 544 may perform interpolation filtering according to the value of the motion vector of the current block. In other words, if the decimal part of the motion vector is not 0, the inter-predictor 544 generates sub samples indicated by the decimal part through interpolation. If the decimal part of the horizontal component (x component) of the motion vector is not 0, interpolation is performed in a horizontal direction. If the decimal part of the vertical component (y component) of the motion vector is not 0, interpolation is performed in a vertical direction.

The adder 550 reconstructs the current block by adding the residual block output from the inverse transformer and the prediction block output from the inter-predictor or the intra-predictor. The samples in the reconstructed current block are used as reference samples in intra-predicting a block to be decoded next.

The loop filter unit 560 may include at least one of a deblocking filter 562, a SAO filter 564, or an ALF 566. The deblocking filter 562 deblocking-filters the boundary between the reconstructed blocks to remove blocking artifacts caused by block-by-block decoding. The SAO filter 564 performs filtering in a manner of adding the reconstructed block after deblocking filtering to corresponding offsets so as to compensate for a difference between the reconstructed sample and the original sample caused by lossy coding. The ALF 566 performs filtering on a target sample to be filtered by applying filter coefficients to the target sample and neighboring samples of the target sample. The ALF 566 may divide the samples in a picture into predetermined groups and then determine one filter to be applied to a corresponding group to differentially perform filtering on each group. The filter coefficient of the ALF is determined based on the information about the filter coefficient decoded from the bitstream.

The reconstructed block filtered through the loop filter unit 560 is stored in the memory 570. When all blocks in one picture are reconstructed, the reconstructed picture is used as a reference picture for inter-prediction of blocks in a picture to be encoded next.

As explained above, in the case of inter-prediction, the video encoding apparatus encodes information on a motion vector of the current block and on a reference picture the motion vector refers to and sends the information to the video decoding apparatus. The video decoding apparatus decodes the information received from the video encoding apparatus and determines the motion vector of the current block and the reference picture to be used for inter-prediction. Also, a prediction block is generated by using the determined motion vector and reference picture.

Generally, pictures constituting one video sequence have the same resolution or size. However, in some applications, the pictures, which belong to the one video sequence, may have different resolutions. When inter-predicting the current block, a picture encoded/decoded prior to the current picture including the current block within the video sequence is used as a reference picture. Accordingly, the current picture including the current block to be encoded or decoded may have a different resolution than the reference picture the current block refers to. In this case, the video encoding or decoding apparatus predicts the current block by using a reference picture having a different resolution than the current picture.

The present disclosure relates to an inter-prediction method for a video sequence of pictures some of which have different resolutions or sizes.

The video encoding apparatus encodes information (first syntax element) on a reference size of pictures constituting a video sequence and sends the information to the video decoding apparatus. The first syntax element includes a reference width and reference height of the pictures. The first syntax element may be included in a high-level syntax structure of a bitstream. For example, it may be included in an SPS, the sequence's header, and signaled to the video decoding apparatus. Alternatively, one or more picture groups constituting a sequence may have different reference sizes. In this case, the first syntax element may be included and signaled in a picture parameter set (PPS) that the pictures in the corresponding picture group commonly refer to.

Meanwhile, the actual size of the pictures constituting the video sequence may be different from the reference size. In other words, the pictures may have a size scaled from the reference size. The video encoding apparatus encodes a second syntax element for indicating the size (width and height) of the pictures scaled from the reference size. Since the pictures constituting the video sequence have different resolutions, the second syntax element may be configured differently for each picture group comprised of one or more pictures. Accordingly, the second syntax element may be included and signaled in a PPS. Pictures referring to different PPSs may have the second syntax elements set to different values.

In some embodiments, the second syntax element may be the actual width and height of the pictures. In some other embodiments, the second syntax element may be a value indicating the ratio with respect to the reference size. For example, the second syntax element may include a horizontal ratio indicating the difference with the reference width and a vertical ratio indicating the difference with the reference height. In some other embodiments, the second syntax element may be defined by offset values from the left, right, upper, and lower boundaries of a picture having the reference size.

The video encoding apparatus encodes input pictures. Additionally, if the size of the inputted pictures is different from the reference size, the video encoding apparatus may convert the resolution of the pictures so that the size of the inputted pictures is the same as the reference size. Resolution conversion may be performed in such a manner of enlarging the input pictures by padding samples with a predefined value. Alternatively, resolution conversion may involve upsampling or downsampling using a filter.

The video encoding apparatus and the video decoding apparatus may store in advance one or more filters used for resolution conversion. Or else, information on filters may be signaled for each sequence, picture, subpicture, slice, tile, CTU, etc.

Alternatively, n fixed default filters (n is a natural number greater than or equal to 1) and m additional filters (m is a natural number greater than or equal to 1) may be used, and information on the additional filters may be transmitted to the video decoding apparatus. When the information on the filters is signaled, the filter coefficient values per se may be signaled, or differences from default filter coefficients may be signaled. The video decoding apparatus may reconstruct filter coefficients from the differences.

If a plurality of filters are available, the index of a filter used for resolution conversion, among the plurality of filters, may be signaled. Alternatively, the video encoding apparatus and the video decoding apparatus may select a filter depending on a scale difference between the reference size and the actual size, without signaling of the index.

The video encoding apparatus performs inter-prediction using pictures whose resolutions are converted to the reference size. In other words, the video encoding apparatus determines a reference picture to be used for inter-prediction of the current block to be encoded and determines a motion vector indicating a position of a block in the reference picture that is the most similar to the current block. As explained below, if reference pictures are stored in their actual size, the reference pictures may have a different resolution than the current picture (having the actual size or reference size). Accordingly, the video encoding apparatus may convert the resolution of the reference picture so as to be the same as the resolution of the current picture and then determine the motion vector.

The video encoding apparatus generates a prediction block for the current block by using the reference picture and the motion vector. Information on the reference picture, information on the motion vector, and information on residual signals (differences between samples in the current block and the prediction block) are transmitted to the video decoding apparatus.

Meanwhile, the video encoding apparatus stores a picture reconstructed by the inverse quantizer 160, inverse transformer 165, adder 170, and filter 180 of FIG. 1 in the memory 190. Before storing it in the memory, the video encoding apparatus may convert the reconstructed picture to various resolutions or sizes and then store it. For example, the video encoding apparatus converts the reconstructed picture having the reference size into a picture having an actual size and then store the converted picture.

With reference to FIG. 6, hereinafter, a method is described that the video decoding apparatus decodes an encoded video sequence of pictures some of which have different resolutions or sizes.

The video decoding apparatus decodes a first syntax element indicating the reference size of pictures from the high-level syntax structure (e.g., SPS or PPS) of a bitstream. Also, it decodes a second syntax element indicating a picture size scaled from the reference size (S610). Moreover, if the current block to be decoded is inter-predicted, the video decoding apparatus decodes information on a motion vector and a reference picture of the current block and thereby determines the motion vector and the reference picture, which are related to the current block (S620).

The video decoding apparatus predicts the current block by using the reference picture and the motion vector. Since a picture decoded prior to the current picture, among the pictures belonging to the video sequence, is used as the reference picture, the reference picture indicated by the information on the reference picture may have a different size or resolution than the current picture where the current block is included. Like the video encoding apparatus, reference pictures stored in the memory of the video decoding apparatus may have the actual size by conversion, not the reference size. Meanwhile, the motion vector transmitted by the video encoding apparatus is a value determined according to the resolution of the current picture. Accordingly, prediction efficiency may be deteriorated if the decoded motion vector itself is applied to the reference picture.

As an embodiment for solving this problem, the video decoding apparatus may convert the resolution of the reference picture to the resolution of the current picture and then generate a prediction block from the resolution-converted reference picture, by using a decoded motion vector. At this time, as described above, the resolution conversion may be done by using a filter shared by the video encoding apparatus and the video decoding apparatus or a filter defined by filter information signaled from the video encoding apparatus.

In another embodiment, the decoded motion vector may be modified by using the first and second syntax elements. Such a modification process is used so as to modify a sample position indicated by the original motion vector in a reference picture having a different resolution than the current picture.

Referring to FIG. 6, the video decoding apparatus may calculate a scale factor indicating a scale difference between the current picture and the reference picture, by using the first syntax element for indicating the reference size and the second syntax element for indicating a picture size scaled from the reference size (S630).

For example, the video decoding apparatus may calculate a first value indicating a scale difference or scale ratio between the size of the reference picture and the reference size, by using a first syntax element indicating a reference size included in a PPS, which the reference picture refers to, and by using a second syntax element indicating a picture size scaled from the reference size. Also, the video decoding apparatus may calculate a second value indicating a scale difference or scale ratio between the size of the current picture and the reference size, by using a first syntax element indicating a reference size included in a PPS, which the current picture refers to, and by using a second syntax element indicating a picture size scaled from the reference size. Using the first value and the second value, the scale factor between the current picture and the reference picture is calculated.

Meanwhile, an offset factor for additionally modifying a motion vector may be calculated. Motion vectors may be expressed by a predefined fixed number of bits. Accordingly, a motion vector to which the scale factor is applied should be expressed by the same number of bits as the fixed number of bits. The offset factor may be used as a shift value, in a bit shift operation for adjusting the number of bits of the motion vector to which the scale factor is applied. The value of the motion vector is modified by the bit shift, and therefore a sample position indicated by the motion vector within the reference picture is modified. The offset factor for the bit shift is derived from the first and second syntax elements. For example, the value of the offset factor may be derived according to the value of the scale factor calculated by the first and second syntax elements.

Moreover, the offset factor may be used to adjust a position indicated by the motion vector. Actual pictures may have a different scale from a picture of a reference size and also have a different position within the picture of the reference size. The offset factor may be used to compensate for a position difference between the actual pictures within the reference picture. If the second syntax element is defined by offset values from the left, right, upper, and lower boundaries of a picture having the reference size, the offset factor may be set as offset values from the left and upper boundaries included in the second syntax element.

Meanwhile, the scale factor and the offset factor may be calculated for each of horizontal and vertical directions. A horizontal scale factor and a horizontal scale factor are applied to a horizontal component (x component) of the motion vector, and a vertical scale factor and a vertical offset factor may be applied to a vertical component (y component) of the motion vector.

The motion vector of the current block decoded from the bitstream is modified by the aforementioned scale factor and offset factor. In other words, a sample position indicated by the decoded motion vector within the reference picture is modified through the scale factor and the offset factor. The video decoding apparatus generates a prediction block having the same size as the current block from the modified sample position within the reference picture (S640).

Meanwhile, a motion compensation process for generating the prediction block from the reference picture stored in the memory may involve filtering. The filtering may be filtering for interpolation of the reference picture. The decimal part of the motion vector (modified motion vector) indicates the position of a sub sample, i.e., fractional sample. A sample value corresponding to the subsample position may be generated through interpolation from integer samples within the reference picture.

One or more filters are variously defined according to filter shapes, sizes (number of taps), and filter coefficients. One of these filters may be selected by the subsample position indicated by the motion vector, the difference (i.e., the aforementioned scale factor) in resolution or size between the current picture and the reference picture, a color component (luma or chroma) to be decoded, and so on.