106 and also a budget map 306 is computed by the remote rendering device. In the example of FIG. 3B the budget map is sent 314 to the display device.

In the example of FIG. 3A the remote rendering device renders a color image from a complex 3D model and also renders a corresponding depth image. The remote rendering device sends 300 the color and the depth image to the display device 102. In an example the color and the depth image are compressed by the remote rendering device 106 using any conventional image compression process such as run-length encoding, predictive coding, entropy encoding, adaptive dictionary algorithms, high efficiency video coding (HEVC), H.264 or any other image compression process and sent 300 to the display device.

The remote rendering device 106 computes 302 an error map. The error map is a 2D array of numerical values corresponding to tiles of the depth image where the depth image is divided into tiles which tesselate over the 2D depth image array. The numerical values in the error map are planar deviation values. A planar deviation value is an amount by which a depth image tile differs in depth from a plane of a surface depicted in the depth image. A planar deviation error value may be calculated as a root mean square error between depth values in a tile and a least-square plane fitted to depth values in a tile. In an example, depth values in a tile are perspectively projected by being non-linearly mapped between near and far planes.

The remote rendering device 106 sends 304 the error map to the display device 102. The error map is compressed by the remote rendering device in some cases such as by using any conventional image compression process such as run-length encoding, predictive coding, entropy encoding, adaptive dictionary algorithms, or any other loss less image compression process. By compressing the error map latency of sending 304 the error map to the display device 102 is reduced. The display device 102 receives the error map and decompresses it if appropriate.

The display device computes 306 a budget map. A budget map is a 2D array of values corresponding to tiles of the depth image (one value per tile). The numerical values in the budget map are integers. An integer of a budget map specifies how many vertices, or how many polygons, are to be used in a 3D mesh model of the corresponding depth map tile. The display device computes 306 the budget map by analyzing the color image and depth image received from the remote rendering device 106 at operation 300, to determine which regions contain complex content, simple content, or no content at all. The images are analyzed to compute statistics such as amount of color variation, entropy, number of edges, rate of change of depth or other statistics. The statistics are compared with thresholds in order to determine regions of complex content, simple content or no content. Regions of no content are regions where color or depth values are zero or near to zero. Given a budget for total computational cost available for reprojection, such as power budget, time budget, or a suitable proxy for these, such as polygon budget available for rendering, the display device distributes that budget between each region of the source image. In an example, the budget map integers are related to a number of polygons to be used. Regions with complex content are assigned a higher proportion of the total budget, regions with simple content are assigned a lower proportion, and region with no content are assigned no budget at all.

The display device 102 generates 308 a mesh. The mesh is formed from polygons (referred to as primitives), and the density of polygons per unit surface area of the mesh varies according to the budget map. The polygons are connected together so as to form a mesh that generally follows how surfaces depicted in the depth image differ with respect to a planar surface depicted in the depth image. The mesh is generated using tessellation or subdivision and represents an approximation of the depth image.

The display device 102 renders 310 from the mesh according to an updated pose of the display device to produce a reprojected image. The display device 102 obtains the updated pose of the display device from sensors in the display device 102 and/or a pose tracker as explained above with reference to FIG. 1. In some cases the updated pose of the display device is received from the remote rendering device 106.

The display device optionally carries out post processing 312 of the reprojected image to produce a final image as output. In an example, the post processing comprises filling disoccluded regions as explained in more detail below with reference to FIG. 6. A disoccluded region is a region of a reprojected image which was previously occluded based on the predicted pose and then uncovered based on the updated pose.

By carrying out the error map computation at the remote rendering device 106 it is possible to significantly reduce the computational burden at the display device 102. This leads to significantly better visual quality of the reprojected image as compared with situations where the display device computes the error map.

FIG. 3B is another example reprojection process split between a remote rendering device 106 and a display device 102 such as those of FIG. 1. In the example of FIG. 3B an error map is computed by the remote rendering device 106. The remote rendering device also computes a budget map 306. In the example of FIG. 3B the budget map is sent 314 to the display device. The budget map is optionally compressed by the remote rendering device 106 prior to being sent 314 to the display device 102. The budget map may be compressed by such as by using any conventional image compression process such as run-length encoding, predictive coding, entropy encoding, adaptive dictionary algorithms, or any other loss less image compression process. By compressing the budget map latency of sending 304 the budget map to the display device 102 is reduced. The display device 102 receives the budget map and decompresses it if appropriate.

The display device 102 generates 308 a mesh using the depth image received at operation 300 and according to the budget map received at operation 314. The display device 102 renders from the mesh 310 according to an updated pose of the display device 102 to produce a reprojected image being a reprojection of the color image received at operation 300.

The display device 102 optionally post processes 312 the reprojected image such as by filling disoccluded regions as explained in more detail below.

By using the remote rendering device to compute both the error map 302 and the budget map 306 the computational requirements of the display device 102 are drastically reduced leading to significantly better visual quality of the reprojected image.

FIG. 3C is another example reprojection process split between a remote rendering device 106 and a display device 102 such as those of FIG. 1. In the example of FIG. 3C the remote rendering device 106 performs computation 302 of the error map, computation of the budget map 306 and generation of the mesh 308. The remote rendering device 106 compresses the mesh using any image compression technology and sends the compressed mesh 316 to the display device 102. The display device receives the compressed mesh, decompresses the mesh, renders from the mesh 310 and optionally does post processing 312 as explained above.

FIG. 4 is a schematic diagram of an example method of planar-deviation reprojection using an approach where the display device 102 carries out a complete reprojection process. In the example of FIG. 4 a rendering device 106 receives pose data 446 from a display device. The pose data comprises a predicted pose (3D position and orientation) of the display device 102. The rendering device 106 comprises a rendering engine 108 which renders a color image 402 and a corresponding depth image 404 from a complex 3D model according to the pose data. The color image and the depth image are compressed by the rendering device 106. A color image compressor 406 compresses the color image. A depth image compressor 408 compresses the depth image. The color image compressor 406 and the depth image compressor 408 are any image compressors as described above with reference to FIGS. 3A and 3B. The compressed color image 410 and the compressed depth image 412 are sent from the rendering device 106 to the display device 102 via a network connection 400 such as a wireless network connection or a wired connection.

The display device 102 receives the color image 410 and decompresses it using a color image decompressor 414 to produce a color image 418. The display device 102 receives the depth image 412 and decompresses it using a depth image decompressor 416 to produce a depth image 420.

The display device 102 analyzes the depth image using a depth analysis process 422. For each tile of the depth image a plane is fitted to depth values of the tile by using a least square plane. For each tile a root means square error between sample depth values in the tile and the least square plane is computed.

The result of the depth analysis 422 is an error map 424 comprising, for each tile, a planar deviation error value. The display device has a performance budget 428 which is pre-configured or determined dynamically according to available resources at the display device 102. The display device 102 allocates 426 the performance budget 428 between tiles of the depth map to produce a budget map 430. In an example, the performance budget 428 is a number of polygons and the number of polygons allocated to a tile is related to the planar deviation error value. Higher planar deviation error values receive higher numbers of polygons.

The display device 102 generates a mesh 434 by tessellating polygons using tessellator 432 to generally follow surfaces depicted in the depth image. The density of the polygons in the mesh is related to the budget map 430. The result is a tessellated mesh 434.

A reprojection process 436 at the display device 102 receives the mesh 434, the color image 418 and the depth image 420, as well as an updated pose 438. The reprojection process produces a reprojected image 440 which may comprise disoccluded regions. The reprojected image may be output to a display 444 at the display device in some cases.

The reprojected image is optionally post processed to fill disoccluded regions to produce a corrected color image which is then displayed at the display device 102. The updated pose 438 is obtained in some cases from sensors and a pose tracker 442 in the display device 102 or in any other suitable way. In various examples, pose data 446 is sent from display device 102 to rendering device 106.

The arrangement of FIG. 4 is workable although where the display device has limited resource, and/or where the rendered images depict complex content the visual quality of the reprojected image may be reduced.

FIG. 5 is a schematic diagram of an example method of planar-deviation based reprojection using one example of split compute. In this example the rendering device 106 computes the error map 424 and the budget map 430 as explained with reference to FIG. 3B. The display device receives a compressed budget map 502 from the rendering device 106. The display device decompresses using budget map decompressor 504 the budget map 502 and uses decompressed budget map 506 to generate a mesh 434 generally following surfaces depicted in the depth image. A reprojection process in the display device uses the mesh 434, the color image 418 and the depth image 420, together with an updated pose 438, to compute a reprojected image 440. The reprojected image may be displayed 444 at the display device. In another example the reprojected image is post processed to fill disoccluded regions prior to being displayed at the display device. The costly operations of depth analysis and budget distribution are performed on the remote rendering device. This reduces the work on the display device to generate the budget map to a much cheaper decompression. The higher performance of the remote rendering device allows performing a more accurate budget distribution by possibly running multiple correction passes to ensure the performance budget is not exceeded.

In the example of FIGS. 5 and 3B the budget map may be compressed losslessly by budget map compressor 500 before transmission in order to reduce impact on the total network bandwidth.

Where a rate control mechanism is used to control the compression ratio of the color and depth images based on network characteristics, such a rate control mechanism is modified to incorporate the bandwidth requirements for the compressed error map. The modification enables the combined bitrate of compressed color, depth and error maps to be below a maximum supported bitrate of the communications network.

FIG. 6 shows how FIG. 5 may be extended to use cross-visibility maps 614. A cross visibility map is a 2D array of binary values where 1 indicates a part of a scene which is depicted in a first image of the scene and which is absent from a second image of the scene. The first and second images are generally stereo images of the scene or images of the scene taken by a moving camera at successive time instances. Given two RGBD (red, green, blue, depth) images Image1 and Image2, the cross-visibility map between Image1 (source) and Image2 (target) is a pixel map marking the regions in Image1 that contain color information of Image1 that are not included in Image2.

Cross-visibility maps 614 are useful because they provide information about how to fill regions of a reprojected image that became disoccluded during reprojection as now explained. While performing reprojection of the right eye into a corrected pose, areas may be disoccluded due to the relative parallax motion of the object. Because there is no information available in the right eye on the color and texture to use to fill these disoccluded areas, additional sources of information are to be employed. A method to do so is to use the RGBD image of the left eye, and perform an additional reprojection into the same corrected pose of the right eye, thus projecting the missing data from the left into the right eye. However, performing an additional forward reprojection of the left eye or search through the left eye can be prohibitively computationally expensive.

By making use of the cross-visibility map between the left (source) and right (target) eye images, an additional forward reprojection or search can be constrained to those areas that uniquely contain information not provided in the right eye image, significantly reducing the computational cost of using additional source RGBD images to infill disoccluded areas. Conversely, a cross-visibility map between right (source) and left (target) marks areas in the right eye image not included in the left eye image, and may be used to infill missing data upon reprojection of the left image in an additional composition step.

Cross-visibility maps are not restricted to just left and right eyes, but may use other source images such as additional views that were rendered specifically just to provide infill information, or may use previously rendered frames.

The computation of a cross-visibility map uses a reprojection of the source into the target image, and thus cannot be used to lower the cost of an additional infill reprojection on the display device, as it is by definition as expensive as a full reprojection. However, in a split compute architecture, a single or a set of cross-visibility maps 614 can be computed cheaply on the more powerful remote rendering device 106, and transmitted in a suitably compressed form 618 (such as transmitting a list of rectangles or polygonal areas, or by using a downscaled map of pixel tiles marked with either a 0 or 1 to indicate the presence or absence of novel information). Thus FIG. 6 illustrates the rendering device computing one or more cross-visibility maps. The cross-visibility maps are computed by a cross visibility estimator 612 which takes as input an RGBD image and other metadata 600 rendered by the rendering engine 108. The cross-visibility estimator takes as input additional images 610 such as previous RGBD images computed by the rendering engine or an RGBD image rendered from a slightly different viewpoint so as to form a stereo pair with the RGBD image. The cross-visibility estimator 612 computes a comparison between a pair of RGBD images it takes as input (such as by computing a subtraction or other comparison). The results produce a cross-visibility map which is optionally compressed 616 as explained above. The compressed cross-visibility map 618 is sent to the display device 102.

The display device decompresses 620 the cross-visibility map and uses it to compute additional partial reprojections 622 of the RGBD image. The result is a corrected RGBD image of any disoccluded areas 624. A composition operation 628 fills disocclusions in the reprojected image 626 with data from the corrected RGBD image. Reprojected image 626 is generated by reprojection 630 as explained with reference to reprojection 436 in FIG. 4 and FIG. 5 using decompressed color and depth images 608. After compression by a compressor 602, compressed color and depth images 604 are sent to display device 102 and decompressed by a decompressor 606.

FIG. 7 illustrates various components of an exemplary computing-based device 700 which are implemented as any form of a computing and/or electronic device, and in which a rendering device or a display device 706 are implemented in some examples. In an example where the computing-based device 700 is a display device it is a head mounted display device HMD, a smart phone, a tablet computer or other display device. In an example where the computing device 700 is a rendering device it is a server such as a cloud server or other server, or a companion computing device of an HMD, or another computing device which has greater resources than the display device.

Computing-based device 700 comprises one or more processors 702 which are microprocessors, controllers or any other suitable type of processors for processing computer executable instructions to control the operation of the device in order to contribute to computing planar-deviation based reprojection of images. In some examples, for example where a system on a chip architecture is used, the processors 702 include one or more fixed function blocks (also referred to as accelerators) which implement a part of the method of any of FIGS. 3A, 3B, 4, 5, 6 in hardware (rather than software or firmware). Where the computing-based device 700 is a display device it optionally comprises sensors 720 such as an inertial measurement unit IMU, an accelerometer, a gyroscope, a global positioning system. The computing-based device 700 optionally comprises a pose tracker 718 to compute a 3D position and orientation of the computing based device. The pose tracker 718 is any conventional pose tracker such as an IMU, accelerometer, gyroscope, global positioning system or pose tracker using captured image data depicting an environment of the computing-based device 700. Data store 714 holds pose data, depth images, color images, cross-visibility maps, sensor data or other data. Platform software comprising an operating system 712 or any other suitable platform software is provided at the computing-based device to enable application software 716 to be executed on the device.

The computer executable instructions are provided using any computer-readable media that is accessible by computing based device 700. Computer-readable media includes, for example, computer storage media such as memory 710 and communications media. Computer storage media, such as memory 710, includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or the like. Computer storage media includes, but is not limited to, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM), electronic erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that is used to store information for access by a computing device. In contrast, communication media embody computer readable instructions, data structures, program modules, or the like in a modulated data signal, such as a carrier wave, or other transport mechanism. As defined herein, computer storage media does not include communication media. Therefore, a computer storage medium should not be interpreted to be a propagating signal per se. Although the computer storage media (memory 710) is shown within the computing-based device 710 it will be appreciated that the storage is, in some examples, distributed or located remotely and accessed via a network or other communication link (e.g. using communication interface 704). The computing-based device 700 has an optional capture device 708 such as a depth camera, color camera, video camera, web camera or other image capture device.

Alternatively or in addition to the other examples described herein, examples include any combination of the following:

Clause A A computer-implemented method performed by a rendering device comprising: