(
0
)

∂
y

⁢

(

i
,
j

)

’

⁢
s

and second gradient values

$\frac{\partial I^{(1)}}{\partial x} (i, j) ’ s and \frac{\partial I^{(1)}}{\partial y} (i, j) ’ s .$

FIG. 4 shows an example method 400 for generating CIIP in accordance to the present disclosure. For example, the method includes a uni-prediction based inter prediction and MPM-based intra prediction for generating CIIP.

In step 410, obtaining a reference picture in a reference picture list associated with a current prediction block.

In step 412, generating a inter prediction based on a first motion vector from a current picture to a first reference picture.

In step 414, obtaining a intra prediction mode associated with the current prediction block.

In step 416, generating a intra prediction of the current prediction block based on the intra prediction.

In step 418, generating a final prediction of the current prediction block by averaging the inter prediction and the intra prediction.

In step 420, determining whether the current prediction block is treated as inter mode or intra mode for most probable mode (MPM) based intra mode prediction.

FIG. 5A shows a diagram illustrating block quaternary partitions in a multi-type tree structure, according to an example of the present disclosure.

FIG. 5B shows a diagram illustrating block vertical binary partition in a multi-type tree structure, according to an example of the present disclosure.

FIG. 5C shows a diagram illustrating block horizontal binary partition in a multi-type tree structure, according to an example of the present disclosure.

FIG. 5D shows a diagram illustrating block vertical ternary partition in a multi-type tree structure, according to an example of the present disclosure.

FIG. 5E shows a diagram illustrating block horizontal ternary partition in a multi-type tree structure, according to an example of the present disclosure.

Combined Inter and Intra Prediction

As shown in FIG. 1 and FIG. 2, inter and intra prediction methods are used in the hybrid video coding scheme, where each PU is only allowed to select inter prediction or intra prediction for exploiting the correlation in either temporal or spatial domain while never in both. However, as pointed out in previous literature, the residual signal generated by inter-predicted blocks and intra-predicted blocks could present very different characteristics from each other. Therefore, when the two kinds of predictions can be combined in an efficient way, one more accurate prediction can be expected for reducing the energy of prediction residuals and therefore improving the coding efficiency. Additionally, in nature video content, the motion of moving objects could be complicated. For example, there could exist areas which contain both old content (e.g., the objects that are included in previously coded pictures) and emerging new content (e.g., the objects that are excluded in previously coded pictures). In such scenario, neither inter prediction nor intra prediction can provide one accurate prediction of current block.

To further improve the prediction efficiency, combined inter and intra prediction (CIIP), which combines the intra prediction and the inter prediction of one CU that is coded by merge mode, is adopted in the VVC standard. Specifically, for each merge CU, one additional flag is signaled to indicate whether the CIIP is enabled for the current CU. For luma component, the CIIP supports four frequently-used intra modes, including planar, DC, horizontal and vertical modes. For chroma components, DM (i.e., chroma reuses the same intra mode of luma component) is always applied without extra signaling. Additionally, in the existing CIIP design, weighted average is applied to combine the inter prediction samples and the intra prediction samples of one CIIP CU. Specifically, when planar or DC mode is selected, equal weight (i.e., 0.5) is applied. Otherwise (i.e., either horizontal or vertical mode is applied), the current CU is firstly split horizontally (for horizontal mode) or vertically (for vertical mode) into four equal-size areas.

Four weight sets, denoted as (w_intra_i, w_inter_i), will be applied to combine the inter and intra prediction samples in different areas, where i=0 and i=3 represent the areas that are closest and furthest to the reconstructed neighboring samples used for intra prediction. In the current CIIP design, the values of weight sets are set as (w_intra₀, w_inter₀) (0.75, 0.25), (w_intra₁, w_inter₁) (0.625, 0.375), (w_intra₂, w_inter₂) (0.375, 0.625) and (w_intra₃, w_inter₃) (0.25, 0.75). FIG. 6A, FIG. 6B, and FIG. 6C (described below) provide examples to illustrate the CIIP mode.

Additionally, in the current VVC working specification, the intra mode of one CIIP CU can be used as the predictor to predict the intra mode of its neighboring CIIP CUs through most probable mode (MPM) mechanism. Specifically, for each CIIP CU, when its neighboring blocks are also CIIP CUs, the intra modes of those neighbors are firstly rounded to the closest mode within planar, DC, horizontal and vertical modes and then added into the MPM candidate list of the current CU. However, when constructing the MPM list for each intra CU, when one of its neighboring blocks is coded by CIIP mode, it is regarded as unavailable, i.e., the intra mode of one CIIP CU is disallowed to predict the intra modes of its neighboring intra CUs. FIG. 7A and FIG. 7B (described below) compare the MPM list generation processes of intra CUs and CIIP CUs.

Bi-Directional Optical Flow

Conventional bi-prediction in video coding is a simple combination of two temporal prediction blocks obtained from the reference pictures that are already reconstructed. However, due to the limitation of the block-based motion compensation, there could be remaining small motion that can be observed between the samples of two prediction blocks, thus reducing the efficiency of motion compensated prediction. To solve this problem, bi-directional optical flow (BDOF) is applied in the VVC to lower the impacts of such motion for every sample inside one block. Specifically, as shown in FIG. 6A, FIG. 6B, and FIG. 6C (described below), the BDOF is sample-wise motion refinement that is performed on top of the block-based motion-compensated predictions when bi-prediction is used. The motion refinement (v_x, v_y) of each 4×4 sub-block is calculated by minimizing the difference between L0 and L1 prediction samples after the BDOF is applied inside one 6×6 window Ω around the sub-block. Specifically, the value of (v_x, v_y) is derived as

$v_{x} = S_{1} > 0 ? clip3 (- {th}_{BDOF}, {th}_{BDOF}, - ((S_{3} \cdot 2^{3}) ≫ ⌊ \log_{2} S_{1} ⌋)) : 0$

$v_{y} = S_{5} > 0 ? clip ⁠ 3 ⁠ (- {th}_{BDOF}, {th}_{BDOF}, - ((S_{6} \cdot 2^{3} - ((v_{x} S_{2, m}) ≪ n_{S_{2}} + v_{x} S_{2, s}) / 2) ≫ ⌊ \log_{2} S_{5} ⌋)) :$

where └·┘ is the floor function; clip3(min, max, x) is a function that clips a given value x inside the range of [min, max]; the symbol » represents bitwise right shift operation; the symbol « represents bitwise left shift operation; th_BDOFis the motion refinement threshold to prevent the propagated errors due to irregular local motion, which is equal to 2^13-BD, where BD is the bit-depth of input video. In (1),

$S_{2, m} = S_{2} >> n_{S_{2}}, S_{2, s} = S_{2} & (2^{n_{S_{2}}} - 1) .$

The values of S₁, S₂, S₃, S₅and S₆are calculated as

$S_{1} = \sum_{(i, j) \in Ω} ψ_{x} (i, j) \cdot ψ_{x} (i, j), S_{3} = \sum_{(i, j) \in Ω} θ (i, j) \cdot ψ_{x} (i, j)$

$S_{2} = \sum_{(i, j) \in Ω} ψ_{x} (i, j) \cdot ψ_{y} (i, j)$

$S_{5} = \sum_{(i, j) \in Ω} ψ_{y} (i, j) \cdot ψ_{y} (i, j) S_{6} = \sum_{(i, j) \in Ω} θ (i, j) \cdot ψ_{y} (i, j)$

$where$

$ψ_{x} (i, j) = (\frac{\partial I^{(1)}}{\partial x} (i, j) + \frac{\partial I^{(0)}}{\partial x} (i, j)) ≫ 3$

$ψ_{y} (i, j) = (\frac{\partial I^{(1)}}{\partial y} (i, j) + \frac{\partial I^{(0)}}{\partial y} (i, j)) ≫ 3$

$θ (i, j) = (I^{(1)} (i, j) ≫ 6) - (I^{(0)} (i, j) ≫ 6)$

where I^(k)(i,j) are the sample value at coordinate (i,j) of the prediction signal in list k, k=0,1, which are generated at intermediate high precision (i.e., 16-bit);

$\frac{\partial I^{(k)}}{\partial x} (i, j) and \frac{\partial I^{(k)}}{\partial y} (i, j)$

are the horizontal and vertical gradients of the sample that are obtained by directly calculating the difference between its two neighboring samples, i.e.,

$\frac{\partial I^{(k)}}{\partial x} (i, j) = (I^{(k)} (i + 1, j) - I^{(k)} (i - 1, j)) ≫ 4$

$\frac{\partial I^{(k)}}{\partial y} (i, j) = (I^{(k)} (i, j + 1) - I^{(k)} (i, j - 1)) ≫ 4$

Based on the motion refinement derived in (1), the final bi-prediction samples of the CU are calculated by interpolating the L0/L1 prediction samples along the motion trajectory based on the optical flow model, as indicated by

${pred}_{BDOF} (x, y) = (I^{(0)} (x, y) + I^{(1)} (x, y) + b + o_{offset}) ≫ shift$

$b = rnd ((v_{x} (\frac{\partial I^{(1)} (x, y)}{\partial x} - \frac{\partial I^{(0)} (x, y)}{\partial x})) / 2) + rnd ((v_{y} (\frac{\partial I^{(1)} (x, y)}{\partial y} - \frac{\partial I^{(0)} (x, y)}{\partial y})) / 2)$

- where shift and o_offsetare the right shift value and the offset value that are applied to combine the L0 and L1 prediction signals for bi-prediction, which are equal to 15−BD and 1«(14−BD)+2·(1«13), respectively.

FIG. 6A shows a diagram illustrating a combined inter and intra prediction for horizontal mode, according to an example of the present disclosure.

FIG. 6B shows a diagram illustrating a combined inter and intra prediction for vertical mode, according to an example of the present disclosure.

FIG. 6C shows a diagram illustrating a combined inter and intra prediction for planar and DC modes, according to an example of the present disclosure.

FIG. 7A shows a flowchart of the MPM candidate list generation processes of intra CUS, according to an example of the present disclosure.

FIG. 7B shows a flowchart of the MPM candidate list generation processes of CIIP CUs, according to an example of the present disclosure.

Improvements to CIIP

Although the CIIP can enhance the efficiency of conventional motion compensated prediction, its design can still be further improved. Specifically, the following issues in the existing CIIP design in the VVC are identified in this disclosure.

First, as discussed in “combined inter and intra prediction” section, because the CIIP combines the samples of inter and intra prediction, each CIIP CU needs to use its reconstructed neighboring samples to generate the prediction signal. This means that the decoding of one CIIP CU is dependent on the full reconstruction of its neighboring blocks. Due to such interdependency, for practical hardware implementations, CIIP needs to be performed in the reconstruction stage where neighboring reconstructed samples become available for intra prediction. Because the decoding of the CUs in the reconstruction stage must be performed sequentially (i.e., one by one), the number of computational operations (e.g., multiplications, additions and bit-shifts) involved in the CIIP process cannot be too high in order to ensure enough throughput of real-time decoding.

As mentioned in the “bi-directional optical flow” section, the BDOF is enabled to enhance the prediction quality when one inter-coded CU is predicted from two reference blocks from both forward and backward temporal directions. As shown in FIG. 8 (described below), in the current VVC, the BDOF is also involved to generate the inter prediction samples for the CIIP mode. Given the additional complexity introduced by the BDOF, such design could severely lower the encoding/decoding throughput of hardware codec when the CIIP is enabled.

Second, in the current CIIP design, when one CIIP CU refers to one merge candidate that is bi-predicted, both the motion compensated prediction signals in list L0 and L1 need to be generated. When one or more MVs are not in integer precision, additional interpolation processes must be invoked to interpolate the samples at fractional sample positions. Such process not only increase the computational complexity but also increase the memory bandwidth given that more reference samples need to be accessed from external memory.

Third, as discussed in “combined inter and intra prediction” section, in the current CIIP design, the intra modes of CIIP CUs and the intra modes of intra CUs are treated differently when constructing the MPM list of their neighboring blocks. Specifically, when one current CU is coded by CIIP mode, its neighboring CIIP CUs are regarded as intra, i.e., the intra modes of the neighboring CIIP CUs can be added into the MPM candidate list. However, when the current CU is coded by intra mode, its neighboring CIIP CUs are regarded as inter, i.e., the intra modes of the neighboring CIIP CUs are excluded from the MPM candidate list. Such non-unified design may not be optimal for the final version of the VVC standard.

FIG. 8 shows a diagram illustrating a workflow of the existing CIIP design in the VVC, according to an example of the present disclosure.

Simplifying CIIP

In this disclosure, methods are provided to simplify the existing CIIP design to facilitate hardware codec implementations. In general, the main aspects of the proposed technologies in this disclosure are summarized as follows.

First, to improve the CIIP encoding/decoding throughput, it is proposed to exclude the BDOF from the generation of the inter prediction samples in the CIIP mode.

Second, to reduce the computational complexity and the memory bandwidth consumption, when one CIIP CU is bi-predicted (i.e., having both L0 and L1 MVs), methods are proposed to convert the block from bi-prediction to uni-prediction for generating the inter prediction samples.

Third, two methods are proposed to harmonize the intra modes of CIIP and intra CUs when forming the MPM candidates of their neighboring blocks.

The CHIP without the BDOF

As pointed out in “problem statement” section, the BDOF is always enabled to generate the inter prediction samples for the CIIP mode when the current CU is bi-predicted. Due to the additional complexity of the BDOF, the existing CIIP design could significantly lower the encoding/decoding throughput, especially making the real-time decoding become difficult for the VVC decoders. On the other hand, for CIIP CUs, their final prediction samples are generated by averaging the inter prediction samples and intra prediction samples. In other words, the refined prediction samples by the BDOF will not be directly used as the prediction signal for CIIP CUs. Therefore, compared to the conventional bi-predicted CUs (where the BDOF is directly applied to generate the prediction samples), the corresponding improvement obtained from the BDOF is less efficient for CIIP CUs. Therefore, based on the above considerations, it is proposed to disable the BDOF when generating the inter prediction samples of the CIIP mode. FIG. 9 (described below) illustrates the corresponding workflow of the proposed CIIP process after removing the BDOF.

FIG. 9 shows a diagram illustrating a workflow of the proposed CIIP method by removing BDOF, according to an example of the present disclosure.