As a result of pruning, a pruning mask including information on whether each pixel in an additional image is valid or invalid may be generated. A pruning mask may be a binary image which represents whether each pixel in an additional image is valid or invalid. In an example, in a pruning mask, a pixel determined as overlapping data with a reference image may have a value of 0 and a pixel determined as non-overlapping data with a reference image may have a value of 1.

While a non-overlapping region may have a non-square shape, a patch is limited to a square shape. Accordingly, a patch may include an invalid region as well as a valid region. Here, a valid region refers to a region composed of non-overlapping pixels between an additional image and a reference image. In other words, a valid region represents a region that includes data which is included in an additional image, but is not included in a reference image. An invalid region refers to a region composed of overlapping pixels between an additional image and a reference image. A pixel/data included by a valid region may be referred to as a valid pixel/valid data and a pixel/data included by an invalid region may be referred to as an invalid pixel/invalid data.

An aggregation unit 124 combines a pruning mask generated in a frame unit in an intra-period unit.

In addition, an aggregation unit 124 may extract a patch from a combined pruning mask image through a clustering process. Specifically, a square region including valid data in a combined pruning mask image may be extracted as a patch. Regardless of a shape of a valid region, a patch is extracted in a square shape, so a patch extracted from a square valid region may include invalid data as well as valid data.

In this case, an aggregation unit 124 may repartition a L-shaped or C-shaped patch which reduces encoding efficiency. Here, a L-shaped patch represents that distribution of a valid region is L-shaped and a C-shaped patch represents that distribution of a valid region is C-shaped.

When distribution of a valid region is L-shaped or C-shaped, a region occupied by an invalid region in a patch is relatively large. Accordingly, a L-shaped or C-shaped patch may be partitioned into a plurality of patches to improve encoding efficiency.

For an unpruned view image, a whole view image may be treated as one patch. Specifically, a whole 2D image which develops an unpruned view image in a predetermined projection format may be treated as one patch. A projection format may include at least one of an Equirectangular Projection Format (ERP), a Cube-map or a Perspective Projection Format.

Here, an unpruned view image refers to a basic image with a highest pruning priority. Alternatively, an additional image that there is no overlapping data with a reference image and a basic image may be defined as an unpruned view image. Alternatively, regardless of whether there is overlapping data with a reference image, an additional image arbitrarily excluded from a pruning target may be also defined as an unpruned view image. In other words, even an additional image that there is data overlapping with a reference image may be defined as an unpruned view image.

A packing unit 126 packs a patch in a rectangle image. In patch packing, deformation such as size transform, rotation, or flip, etc. of a patch may be accompanied. An image that patches are packed may be defined as an atlas.

Specifically, a packing unit 126 may generate a texture atlas by packing a basic texture image and/or texture patches and may generate a depth atlas by packing a basic depth image and/or depth patches.

For a basic image, a whole basic image may be treated as one patch. In other words, a basic image may be packed in an atlas as it is. When a whole image is treated as one patch, a corresponding patch may be referred to as a complete image (complete view) or a complete patch.

The number of atlases generated by an atlas generation unit 120 may be determined based on at least one of an arrangement structure of a camera rig, accuracy of a depth map or the number of view images.

A metadata generation unit 130 generates metadata for image synthesis. Metadata may include at least one of camera-related data, pruning-related data, atlas-related data or patch-related data.

Pruning-related data includes information for determining a pruning priority between view images. In an example, at least one of a flag representing whether a view image is a root node or a flag representing whether a view image is a leaf node may be encoded. A root node represents a view image with a highest pruning priority (i.e., a basic image) and a leaf node represents a view image with a lowest pruning priority.

When a view image is not a root node, a parent node index may be additionally encoded. A parent node index may represent an image index of a view image, a parent node.

Alternatively, when a view image is not a leaf node, a child node index may be additionally encoded. A child node index may represent an image index of a view image, a child node.

Atlas-related data may include at least one of size information of an atlas, number information of an atlas, priority information between atlases or a flag representing whether an atlas includes a complete image. A size of an atlas may include at least one of size information of a texture atlas and size information of a depth atlas. In this case, a flag representing whether a size of a depth atlas is the same as that of a texture atlas may be additionally encoded.

When a size of a depth atlas is different from that of a texture atlas, reduction ratio information of a depth atlas (e.g., scaling-related information) may be additionally encoded. Atlas-related information may be included in a “View parameters list” item in a bitstream.

In an example, geometry_scale_enabled_flag, a syntax representing whether it is allowed to reduce a depth atlas, may be encoded/decoded. When a value of a syntax geometry_scale_enabled_flag is 0, it represents that it is not allowed to reduce a depth atlas. In this case, a depth atlas has the same size as a texture atlas.

When a value of a syntax geometry_scale_enabled_flag is 1, it represents that it is allowed to reduce a depth atlas. In this case, information for determining a reduction ratio of a depth atlas may be additionally encoded/decoded. In an example, geometry_scaling_factor_x, a syntax representing a horizontal directional reduction ratio of a depth atlas, and geometry_scaling_factor_y, a syntax representing a vertical directional reduction ratio of a depth atlas, may be additionally encoded/decoded.

An immersive video output device may restore a reduced depth atlas to its original size after decoding information on a reduction ratio of a depth atlas.

Patch-related data includes information for specifying a position and/or a size of a patch in an atlas image, a view image to which a patch belongs and a position and/or a size of a patch in a view image. In an example, at least one of position information representing a position of a patch in an atlas image or size information representing a size of a patch in an atlas image may be encoded. In addition, a source index for identifying a view image from which a patch is derived may be encoded. A source index represents an index of a view image, an original source of a patch. In addition, position information representing a position corresponding to a patch in a view image or position information representing a size corresponding to a patch in a view image may be encoded. Patch-related information may be included in an “Atlas data” item in a bitstream.

An image encoding unit 140 encodes an atlas. When view images are classified into a plurality of groups, an atlas may be generated per group. Accordingly, image encoding may be performed independently per group.

An image encoding unit 140 may include a texture image encoding unit 142 encoding a texture atlas and a depth image encoding unit 144 encoding a depth atlas.

A bitstream generation unit 150 generates a bitstream based on encoded image data and metadata. A generated bitstream may be transmitted to an immersive video output device.

FIG. 2 is a block diagram of an immersive video output device according to an embodiment of the present disclosure.

In reference to FIG. 2, an immersive video output device according to the present disclosure may include a bitstream parsing unit 210, an image decoding unit 220, a metadata processing unit 230 and an image synthesizing unit 240.

A bitstream parsing unit 210 parses image data and metadata from a bitstream. Image data may include data of an encoded atlas. When a spatial random access service is supported, only a partial bitstream including a watching position of a user may be received.

An image decoding unit 220 decodes parsed image data. An image decoding unit 220 may include a texture image decoding unit 222 for decoding a texture atlas and a depth image decoding unit 224 for decoding a depth atlas.

A metadata processing unit 230 unformats parsed metadata.

Unformatted metadata may be used to synthesize a specific view image. In an example, when motion information of a user is input to an immersive video output device, a metadata processing unit 230 may determine an atlas necessary for image synthesis and patches necessary for image synthesis and/or a position/a size of the patches in an atlas and others to reproduce a viewport image according to a user's motion.

An image synthesizing unit 240 may dynamically synthesize a viewport image according to a user's motion. Specifically, an image synthesizing unit 240 may extract patches required to synthesize a viewport image from an atlas by using information determined in a metadata processing unit 230 according to a user's motion. Specifically, a viewport image may be generated by extracting patches extracted from an atlas including information of a view image required to synthesize a viewport image and the view image in the atlas and synthesizing extracted patches.

FIGS. 3 and 5 show a flow chart of an immersive video processing method and an immersive video output method, respectively.

In the following flow charts, what is italicized or underlined represents input or output data for performing each step. In addition, in the following flow charts, an arrow represents processing order of each step. In this case, steps without an arrow indicate that temporal order between corresponding steps is not determined or that corresponding steps may be processed in parallel. In addition, it is also possible to process or output an immersive video in order different from that shown in the following flow charts.

An immersive video processing device may receive at least one of a plurality of input images, a camera internal variable and a camera external variable and evaluate depth map quality through input data S301. Here, an input image may be configured with a pair of a texture image (Attribute component) and a depth image (Geometry component).

An immersive video processing device may classify input images into a plurality of groups based on positional proximity of a plurality of cameras S302. By classifying input images into a plurality of groups, pruning and encoding may be performed independently between adjacent cameras whose depth value is relatively coherent. In addition, through the process, a spatial random access service that rendering is performed by using only information of a region a user is watching may be enabled.

But, the above-described S301 and S302 are just an optional procedure and this process is not necessarily performed.

When input images are classified into a plurality of groups, procedures which will be described below may be performed independently per group.

An immersive video processing device may determine a pruning priority of view images S303. Specifically, view images may be classified into a basic image and an additional image and a pruning priority between additional images may be configured.

Subsequently, based on a pruning priority, an atlas may be generated and a generated atlas may be encoded S304. A process of encoding atlases is shown in detail in FIG. 4.

Specifically, a pruning parameter (e.g., a pruning priority, etc.) may be determined S311 and based on a determined pruning parameter, pruning may be performed for view images S312. As a result of pruning, a basic image with a highest priority is maintained as it is originally. On the other hand, through pruning for an additional image, overlapping data between an additional image and a reference image is removed. Through a warping process based on a depth image, overlapping data between an additional image and a reference image may be removed.

As a result of pruning, a pruning mask may be generated. If a pruning mask is generated, a pruning mask is combined in a unit of an intra-period S313. And, a patch may be extracted from a texture image and a depth image by using a combined pruning mask S314.

Specifically, a combined pruning mask may be masked to texture images and depth images to extract a patch.

In this case, for a non-pruned view image (e.g., a basic image), a whole view image may be treated as one patch.

Subsequently, extracted patches may be packed S315 and an atlas may be generated S316. Specifically, a texture atlas and a depth atlas may be generated.

In addition, an immersive video processing device may determine a threshold value for determining whether a pixel is valid or invalid based on a depth atlas S317. In an example, a pixel that a value in an atlas is smaller than a threshold value may correspond to an invalid pixel and a pixel that a value is equal to or greater than a threshold value may correspond to a valid pixel. A threshold value may be determined in a unit of an image or may be determined in a unit of a patch.

For reducing the amount of data, a size of a depth atlas may be reduced by a specific ratio S318. When a size of a depth atlas is reduced, information on a reduction ratio of a depth atlas (e.g., a scaling factor) may be encoded. In an immersive video output device, a reduced depth atlas may be restored to its original size through a scaling factor and a size of a texture atlas.

Metadata generated in an atlas encoding process (e.g., a parameter set, a view parameter list or atlas data, etc.) and SEI (Supplemental Enhancement Information) are combined S305. In addition, a sub bitstream may be generated by encoding a texture atlas and a depth atlas respectively S306, and a single bitstream may be generated by multiplexing encoded metadata and an encoded atlas S307.

An immersive video output device demultiplexes a bitstream received from an immersive video processing device S501. As a result, video data, i.e., atlas data and metadata may be extracted respectively S502 and S503.

An immersive video output device may restore an atlas based on parsed video data S504. In this case, when a depth atlas is reduced at a specific ratio, a depth atlas may be scaled to its original size by acquiring related information from metadata S505.

When a user's motion occurs, based on metadata, an atlas required to synthesize a viewport image according to a user's motion may be determined and patches included in the atlas may be extracted. A viewport image may be generated and rendered S506. In this case, in order to synthesize viewpoint image with the patches, size/position information of each patch and a camera parameter, etc. may be used.

In the present disclosure, a spatio-temporal merge (STM) technique is proposed to improve MIV sequence encoding efficiency and reduce decoding complexity. With a STM technique proposed in the present disclosure, an effect of improving encoding efficiency and reducing decoding complexity and rendering complexity may be generated. Specifically, an embodiment to be described in the present disclosure includes a process of separating a background and foreground frame from a MIV sequence in a unit of a GOP. Each of a separated background frame and foreground frame may be encoded/decoded independently.

A key term proposed in the present disclosure may be defined as follows.

- A spatio-temporal merge (STM) technique: A STM technique is to merge multiple background frames into a single background frame (i.e., temporal merge) or merge multiple views into a single center view background frame (i.e., spatial merge) for a given GOP. When using a STM technique, a method may improve coding efficiency and reduce decoding and rendering complexity by reducing the overlap of background frames between consecutive images or between a plurality of viewpoints.
- Separation of Background and Foreground: It shows separating a background frame and a foreground frame from a MIV sequence within a GOP unit. If a background frame and a foreground frame are separated, each of a background patch and a foreground patch may be packed in a separate atlas. As an example, a background atlas in which only background patches are packed and a foreground atlas in which only foreground patches are packed may be generated, respectively. When a background and a foreground are separated, it may provide greater flexibility in processing a heterogeneous individual parameter such as duration and sampling speed.

In an embodiment described later, the encoding and decoding efficiency of a MIV sequence may be improved by applying the above-described STM technique and a background and foreground separation technique together. Specifically, a STM technique may contribute to improving coding efficiency and reducing decoding and rendering complexity by reducing the overlap of backgrounds of each viewpoint. A background and foreground separation technique may provide greater flexibility in processing a heterogeneous individual parameter.

Based on the above-described description, a STM technique proposed in the present disclosure will be described in detail below.

FIG. 6 is a flowchart of a STM technique according to an embodiment of the present disclosure.

First, a background and a foreground of a picture may be separately extracted S601. Specifically, for each picture belonging to a Group of Pictures (GOP), a foreground and a background may be separated. In this case, under the assumption that a background pixel has a static characteristic during a GOP period, each pixel included in one picture may be divided into a foreground pixel or a background pixel. In other words, all pixels belonging to a foreground object may have a geometrical position change during a GOP period.

Meanwhile, through this operation, one picture may be divided into a foreground frame and a background frame. Here, a foreground frame represents a picture composed only of foreground pixels, and a background frame represents a picture composed only of background pixels.

Based on an entity map, a background pixel may be also extracted from an original picture. Here, an entity map is used to define a background entity, and it may also include information on each object in a picture.

FIG. 7 illustrates an entity map.

Each object included in a picture may be defined as an entity. In addition, a unique identifier (ID) may be allocated to each entity. In an example shown in FIG. 7, an ID of background entities are 0 to 5, and an ID of foreground entities are 6 to 24.

An entity map may be examined to easily extract a background pixel from an original picture. In other words, a pixel in an original picture corresponding to a pixel occupied by a background entity on an entity map may be set as a background pixel.

In distinguishing between a foreground frame and a background frame based on an entity map,

- at least one of a method for preserving a foreground pixel forming an entity boundary or a method for discarding a boundary pixel forming an entity boundary may be applied. In this case, a method for extracting a foreground frame and a background frame may be set by considering a resolution difference between channels, i.e., that a resolution of an UV channel is relatively low compared to a Y channel. In other words, by considering that an image size for an UV channel is smaller than that of a Y channel, one of a method for extracting a background frame by preserving a foreground pixel or a method for extracting a background frame by discarding a foreground pixel may be used.

FIG. 8 shows a difference according to a processing method of an entity boundary pixel.

A left side of FIG. 8 shows an example in which a background frame is extracted by preserving a foreground pixel forming an entity boundary for an UV channel.

On the other hand, a right side of FIG. 8 shows an example in which a background frame is extracted by discarding a foreground pixel forming an entity boundary for an UV channel.

Meanwhile, a background frame may be generated by using a method for discarding a boundary pixel, and a foreground frame may be generated by using a method for preserving a boundary pixel. In this case, a shadow effect may appear around a boundary of a foreground object, correcting reconstruction quality.

After separating a picture into a foreground and a background, background data may be spatially merged S602. Specifically, a spatio-merge (SM) technique may be used to merge background frames extracted from pictures with a different viewpoint into one background frame.

In order to use a SM technique, a pixel coordinate may be transformed based on geometric information provided for a depth map. Specifically, a coordinate independently defined per each view may be transformed to a world coordinate based on a center view.

In other words, a depth map captured from an individual camera may be utilized to reconstruct a coordinate included in each view into a general world coordinate.

If coordinates transformed to world coordinates are reprojected onto a center view image plane, data extracted from each of a plurality of views may be rearranged into a single view.

FIGS. 9 and 10 show an example in which data from a plurality of views is rearranged into a single view.

FIG. 9 shows data extracted from each of a plurality of views, and FIG. 10 shows an example in which data in FIG. 9 is rearranged into a single view. Rearranged data shown in FIG. 10 may include all data extracted from each of a plurality of views shown in FIG. 9.

Meanwhile, a center view may represent a view including the most background data among a plurality of views. When a plurality of views are partitioned into a plurality of groups, a center view may be set for each group.

As in an example shown in FIGS. 9 and 10, a center view frame may be generated by rearranging background data extracted from each of a plurality of views into a single view. In other words, a center view frame may be a single view frame generated by reprojecting background data extracted from each of a plurality of views onto a center view image plane.

Meanwhile, in order to maintain an optimal spatial merge result, a center view frame may be also generated by using TMIV renderer implementation.

FIG. 11 illustrates a center view frame generated by utilizing TMIV renderer implementation.

Specifically, one center view may be created by using a TMIV renderer. However, in order to utilize spatial merge data according to a SM technique as input of a TMIV encoder/decoder, an inpainter function may be turned off for an optimal reference configuration.

Meanwhile, a plurality of viewpoints may be classified into a plurality of groups. In this case, an atlas may be generated per each group, and spatial merge may be performed by using only viewpoints belonging to an atlas. In other words, a spatially merged background frame may be generated per atlas.

A spatially merged background frame may be also merged on a temporal axis S603. Through temporal merge, background data of pictures with a different Picture Order Counter (POC) may be merged into one background frame. In order to merge a background frame along a temporal axis, a process of filling an empty space covered with a foreground entity within a background frame with background pixels may be performed.

FIG. 12 shows an example in which an empty space corresponding to a foreground entity is filled.

Specifically, an empty space occupied by a foreground entity in a current background frame may be filled with background pixels of reference data. Here, reference data may represent data included in a frame with a POC that is different from a POC of a current frame from which a current background frame is extracted. As an example, if a current frame is a frame with the smallest POC in a GOP, a reference frame may be a future frame with a larger POC than a current frame.

Meanwhile, a current frame may be a representative frame within a GOP period. Here, a representative frame may be a first frame within a GOP period (i.e., a frame with the smallest POC) or a frame with the smallest ratio occupied by a foreground object within a GOP period.

A background frame for the entire GOP period may be generated by combining all reference pixels within a GOP period to which a current background frame belongs with a current background frame. As a result, a background frame covering one GOP period may be generated. As a result, a background for one GOP period may be expressed with one background frame, reducing and simplifying the amount of data expressing a background. Meanwhile, a background frame for the entire GOP period may be referred to as a GOP background frame.

Meanwhile, in temporally merging a background frame, a shadow effect occurring at a boundary of a foreground object which may affect background quality must be considered.

FIG. 13 illustrates a shadow effect occurring at a boundary of a foreground object, and FIG. 14 shows an example in which an error according to a shadow effect of a foreground object occurs in a temporally merged background frame.

As in an example shown in FIG. 13, a shadow effect such as blur processing may occur around a foreground object. Due to this shadow effect, distortion may occur in a background pixel around a foreground object as in an example shown in FIG. 14.

In order to solve a problem caused by a shadow from a foreground object, a pixel around a foreground object may be set as a dirty pixel. Here, a dirty pixel may represent a pixel to be updated, i.e., a pixel replaced by a reference pixel.

FIG. 15 shows an example in which a pixel around a foreground object is set as a dirty pixel, and FIG. 16 shows a GOP background frame generated by merging a background frame in which a pixel around a foreground object is set as a dirty pixel with a reference pixel.

In order to preserve natural background data within a GOP period, a dirty pixel may be replaced with a reference pixel.

In other words, a dirty pixel may be processed in the same way as a pixel in an empty region occupied by a foreground object.

Alternatively, a dirty pixel may be updated based on a weighted sum operation of a reference background pixel and a dirty pixel. In this case, a weight for the weighted sum operation may be determined based on a distance from a boundary of a foreground object to the dirty pixel. As an example, when a dirty pixel is positioned close to a boundary of a foreground object, a weight allocated to a dirty pixel may have a smaller value than a weight allocated to a reference pixel. On the other hand, if a dirty pixel is positioned far from a boundary of a foreground object, a weight allocated to a dirty pixel may have a value equal to or greater than a weight allocated to a reference pixel.

As a background image is merged spatially and temporally according to the present disclosure, a merged background image may be used to reproduce a background for a target viewpoint. Specifically, an immersive image output device may determine a foreground object included in a target viewpoint to be reproduced and decode a foreground object included in a target viewpoint. Then, it may combine a decoded background image and a decoded foreground object to reproduce an image for a target viewpoint.

In FIG. 6, it was shown that temporal merge is performed after background data is spatially merged. Unlike an example shown, spatial merge may be performed after background data is temporally merged.

Each of merged background frames and foreground frames may be packed into a separate atlas. As an example, patches extracted from a merged background frame may be packed into a first atlas, and patches extracted from foreground frames may be packed into a second atlas. Here, a patch extracted from a background frame, i.e., an atlas generated by packing background patches may be referred to as a background atlas, and a patch extracted from a foreground frame, i.e., an atlas generated by packing foreground patches may be referred to as a foreground atlas.

Meanwhile, the number of background atlases may be equal to or smaller than the number of foreground atlases. As an example, when the number of foreground atlases is two, the number of background atlases may be one or two.

In the above-described example, it was exemplified that temporal merge is performed in a unit of a GOP period. Unlike a described example, temporal merge may be also performed in a unit of a predetermined number of pictures. Here, a predetermined number is an integer equal to or greater than 2, and it may be predefined in an encoder and a decoder. Alternatively, information representing a predetermined number may be encoded and signaled.

Meanwhile, in the above-described example, it was illustrated that a merged background frame (i.e., a merged background frame) is generated by merging background frames spatially and temporally. Unlike the above-described example, a merged background frame may be generated by merging background frames only spatially or temporally.

As an example, when background frames are merged only spatially, a merged background frame may be obtained per POC. On the other hand, when background frames are merged only temporally, a merged background frame may be obtained per viewpoint for a predetermined period (e.g., a GOP period).

Meanwhile, when a background and a foreground are separated and processed, reallocation processing of a viewpoint identifier may be performed.

As an example, a different viewpoint identifier may be allocated to each of a background and a foreground separated from a specific viewpoint image. In other words, one viewpoint image may be partitioned into a plurality of images (i.e., a background image and a foreground image), and a different viewpoint identifier may be allocated to each of a plurality of images.

Alternatively, background images extracted from viewpoint images may be merged to generate at least one background frame and then allocate a new viewpoint identifier to a merged background frame.

Alternatively, a different viewpoint identifier may be allocated to each foreground object.

When a picture is separated into a background and a foreground and processed, metadata separate from image data may be generated and encoded.

As an embodiment of the present disclosure, background information and foreground information may be identified by using a patch label. Specifically, information identifying whether a corresponding patch belongs to a background region or a foreground region may be encoded per patch.

Alternatively, an identifier of an entity related to a patch may be encoded/decoded, and information showing whether the entity is a foreground entity or a background entity may be encoded/decoded.

Alternatively, background-related information may be encoded/decoded through an atlas. Specifically, when a general foreground atlas and an additional atlas that requires an additional rendering process, i.e., a background atlas, are encoded/decoded separately, information showing whether an atlas is a foreground atlas or a background atlas may be encoded and signaled. Meanwhile, a foreground atlas follows a traditional decoding process, while an additional atlas may have a rendering process different from that of a foreground atlas.

Alternatively, at a sequence level, information related to background and foreground separation may be encoded and signaled. Specifically, through a sequence, information showing at least one of whether a background and a foreground are separated and processed, a background and foreground separate processing period or a frame rate may be encoded and signaled.

Table 1 illustrates a syntax structure including information related to background and foreground separation.

TABLE 1

Descriptor

casps_miv_2_extension( ) {

casme_decoder_side_depth_estimation_flag
u(1)

casme_chroma_scaling_present_flag
u(1)

casme_capture_device_information_present_flag
u(1)

if( casme_capture_device_information_present_flag )

capture_device_information( )

casme_background_seperation_enable_flag
u(1)

casme_reserved_zero_8bits
u(8)

}

In Table 1, a syntax, casme_background_separation_enabled_flag shows whether data related to separate processing into a background and a foreground exists in a bitstream. As an example, when a value of the flag is 1, it shows that viewpoint images are separated into a background and a foreground and processed. In this case, information related to separate processing between a background and a foreground may exist in a bitstream. On the other hand, when a value of the flag is 0, it shows that viewpoint images are not separated into a background and a foreground. In this case, information related to separate processing between a background and a foreground may not exist in a bitstream.

The information may be encoded and signaled through a sequence parameter set for an atlas (e.g., a Common Atlas Sequence Parameter Set, casps), as illustrated in Table 1.

Meanwhile, a background frame may be generated only when viewpoint images are separated into a background and a foreground and processed. In other words, when a spatially and/or temporally merged background frame is encoded/decoded, a value of the flag must be set as 1.

Separately from the information, information showing whether a background frame is merged may be additionally encoded/decoded. As an example, if a value of a flag showing that viewpoint images are separated into a background and a foreground and processed is true (i.e., 1), information showing whether a merged background frame is encoded/decoded may be additionally encoded/decoded. Information showing whether a merged background frame is encoded/decoded may be encoded/decoded through a sequence parameter set for an atlas (e.g., a Common Atlas Sequence Parameter Set) or may be encoded at a different level.

If the information shows that a merged background frame is encoded/decoded, information related to a merge type may be additionally encoded/decoded. The merge type may indicate one of spatial merge, temporal merge, or spatial and temporal merge.

Meanwhile, a merge type of a background frame may be independently determined per atlas. In other words, a merge type for a first atlas and a merge type for a second atlas may be different from each other. Alternatively, a merge type of a background frame may be independently determined between a color image and a depth image. In other words, a merge type for a color image and a merge type for a depth image may be different from each other.