Δβ
_

)

0

1

0

sin
⁡
(

Δβ
_

)

0

cos
⁡
(

Δβ
_

)

]

×

[

1

0

0

0

cos
⁡
(

Δγ
_

)

sin
⁡
(

Δγ
_

)

0

-

sin
⁡
(

Δγ
_

)

cos
⁡
(

Δγ
_

)

]

(
5
)

- where Δα, Δβ, and Δγ are rotational displacements integrated from {circumflex over (ω)} using the first-order Runge-Kutta method.

Stage 2 is called a range image conversion stage/step. In this step, after transformation, each point cloud is converted to a range image for subsequent computations. Specifically, every point (x, y, z) in the 3D Cartesian space is converted to a pixel at coordinates (θ, φ) in the range image with a pixel value r:

$\begin{matrix} r = \sqrt{x^{2} + y^{2} + z^{2}}; \\ θ = \arctan (\frac{x}{y}) / θ_{r}; ϕ = \arccos (\frac{z}{r}) / ϕ_{r} \end{matrix}$

Where θ_rand φ_rare the horizontal and vertical resolutions of the LiDAR, respectively. If θ_rand φ_rare the same as the resolutions of the LiDAR, the range image is a lossless compression of the corresponding point cloud. Larger θ_rand φ_rwould lead to a lower range image resolution, providing a lossy compression of the original point cloud.

The θ_rand φ_rcan be adjusted to make the range image conversion to be lossless or lossy, but the entire compression is still lossy. Moreover, p and y are used to define the number of rows and columns in the range map. For example, LiDAR has a field of view (FOV) of 360 horizontally and 30 degrees vertically, then p is the horizontal degree granularity and y is the vertical degree granularity. If p is 0.18, then the range map will have 2000 columns; and if y is 0.45, then the range map has about 67 rows. So, p and y do not have to be in the range 0-1.

Stage 3 is called a spatially encoding stage. In this stage, the K-frame is spatially encoded by fitting planes. All the points that lie on the same plane are encoded by using that plane. Specifically, a “divide and conquer” strategy is used. A range image is divided into unit tiles (e.g., 4×4). Then, it fits a plane for points in the first tile, and gradually grows to include adjacent tiles to form a bigger tile. Each time the tile grows, it will be tested whether the plane fit so far can be used to encode all the points in the new (bigger) tile under a predefined threshold. If so, all the points in the new tile are encoded with the plane. The process is repeated until the range images are processed.

FIG. 6 shows an example spatial encoding process 600. In the example 600, the range image 610 is first tiled, and then iteratively plane-fitted. Horizontally adjacent tiles fitted by the same plane are shaded by the same stripe pattern. Tiles that are plane-fitted are encoded using the format 620. Points in unfitted tiles are encoded individually using their range values (not shown in FIG. 6).

Stage 4 is called a temporal encoding stage. The fitted planes in the K-frame are then (re-) used to temporally encode the P-frames.

Several popular applications are used to evaluate the real-time spatio-temporal LiDAR point cloud compression as follows. For registration, a recent iterative closest point (ICP)-based registration pipeline is developed using the widely-used PCL (point cloud library). For object detection, a second, CNN and PointNet-based approach is used. For scene segmentation, a SqueezeSegV3, CNN-based approach is used.

Then, two different datasets are chosen for the evaluation experiments. One is Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset, which is for evaluation of registration and object detection. The other one is SemanticKITTI, which is used for the evaluation of segmentation.

As an example KITTI dataset, in a Velodyne laser scanner dataset (bin files), each point is stored with its (x, y, z) coordinate and an additional reflectance value (r). While the number of points per scan is not constant, on average each file/frame has a size of ˜1.9 MB which corresponds to approximately 120,0003D points and reflectance values. It should be noted that the Velodyne laser scanner rotates continuously around its vertical axis (counter-clockwise), which can be taken into account using the timestamp files.

The KITTI benchmarks are used for 3D object detection, which includes Car, Pedestrian and Cyclist benchmarks. The dataset contains a training set of 3712 samples, an evaluation set of 3769 samples, and a testing set of 7518 samples.

For scene segmentation, a large-scale Semantic KITTI dataset is used. The dataset has 21 sequences provided by the odometry task, in which there are 11 sequences for training set and 11 sequences for testing set. The dataset contains 28 classes including classes distinguishing non-moving and moving objects. Overall, the classes used herein cover traffic participants, but also functional classes for ground, like parking areas, and sidewalks.

In this disclosure, several baselines are used to compare with the evaluation experiment results to validate whether the disclosed real-time spatio-temporal LiDAR point cloud compression is better than other existing methods. The baselines include 1) G-PCC by MPEG Group designed to compress LiDAR point cloud data, which constructs an Octree for a point cloud and encodes the Octree; 2) V-PCC by MPEG Group designed to compress dense point clouds used in volumetric rendering, which maps point clouds to images and uses existing video compression to compress the images; 3) JPEG which compresses each point cloud's range image individually using the (lossy) JPEG codec; and 4) H.264 which compresses a sequence of point clouds by compressing the corresponding range image sequence using the H.264 video codec, both lossless and lossy.

It is noted that the evaluation formula such as Translation Error, Detection Accuracy, and Segmentation Error can be referred in three applications to have the evaluation formula. If use small p and y values, it should retain more information.

An assessment of performance is evaluated and provided on the validation set using pre-trained models. The 3D object detection results are evaluated by the average precision (AP) and intersection over union (IoU score) is used as the primary accuracy metric in the scene segmentation experiments.

For object detection results, results of object detection are given in terms of average precision (AP) and results of joint object detection and orientation estimation are provided in terms of average orientation similarity (AOS). For cars, a 3D bounding box overlap of 70% is required, while for pedestrians and cyclists, a 3D bounding box overlap of 50% is required. Difficulties are defined as follows: 1) Easy: minimum bounding box height: 40 Px, maximum occlusion level: fully visible, and maximum truncation: 15%; 2) Moderate: minimum bounding box height: 25 Px, maximum occlusion level: partly occluded, and maximum truncation: 30%; and 3) Hard: minimum bounding box height: 25 Px, maximum occlusion level: difficult to see, and maximum truncation: 50%.

TABLE 1

Evaluation results of the real-time spatio-temporal LiDAR point cloud

compression using the Second, CNN and PointNet-based approach.

Benchmark
Easy
Moderate
Hard

Car (3D Detection)
88.61%
78.62%
77.22%

Car (Bird's Eye View)
90.01%
87.93%
86.45%

Pedestrian (3D Detection)
56.55
52.98
47.73

Pedestrian (Bird's Eye View)
61.99
56.66
53.81

Cyclist (3D Detection)
80.59
67.16
63.11

Cyclist (Bird's Eye View)
84.02
70.70
65.48

According to Table 1 above, the Second architecture achieves the highest AP score. Specifically, the Second model has an AP of 78.62% and 88.61% respectively in terms of moderate and easy for 3D detection in Cars. The Second model also has AP of 86.45% and 90.01% for hard and easy difficulties, respectively. In Pedestrian, the Second architecture performs the lowest AP score, in which it has 56.55% and 52.98% on easy and moderate APs, respectively. The Second model also achieves a great detection of Cyclists with a result of 67.16% moderate AP and 80.59% easy AP.

In this disclosure, videos for the object detection evaluation are also recorded. Two kinds of videos have been prepared, which are RGB and LiDAR. Detection results are based on the KITTI validation set. The RGB video is the bird's eye view (BEV) representation of the detected cars. The LiDAR video is from camera images corresponding to those BEVs. Cars are detected as green bounding boxes, cyclists are detected as yellow bounding boxes, and pedestrians are detected as blue bounding boxes.

TABLE 2

Evaluation results of the scene segmentation.

Acc avg
0.949

IoU avg
0.788

IoU class 1 [car]
0.944

IoU class 2 [bicycle]
0.439

IoU class 3 [motorcycle]
0.795

IoU class 4 [truck]
0.775