An Efficient Deep Learning Approach Using Improved Generative Adversarial Networks for Incomplete Information Completion of Self-driving

09/01/2021
by   Jingzhi Tu, et al.
0

Autonomous driving is the key technology of intelligent logistics in Industrial Internet of Things (IIoT). In autonomous driving, the appearance of incomplete point clouds losing geometric and semantic information is inevitable owing to limitations of occlusion, sensor resolution, and viewing angle when the Light Detection And Ranging (LiDAR) is applied. The emergence of incomplete point clouds, especially incomplete vehicle point clouds, would lead to the reduction of the accuracy of autonomous driving vehicles in object detection, traffic alert, and collision avoidance. Existing point cloud completion networks, such as Point Fractal Network (PF-Net), focus on the accuracy of point cloud completion, without considering the efficiency of inference process, which makes it difficult for them to be deployed for vehicle point cloud repair in autonomous driving. To address the above problem, in this paper, we propose an efficient deep learning approach to repair incomplete vehicle point cloud accurately and efficiently in autonomous driving. In the proposed method, an efficient downsampling algorithm combining incremental sampling and one-time sampling is presented to improves the inference speed of the PF-Net based on Generative Adversarial Network (GAN). To evaluate the performance of the proposed method, a real dataset is used, and an autonomous driving scene is created, where three incomplete vehicle point clouds with 5 different sizes are set for three autonomous driving situations. The improved PF-Net can achieve the speedups of over 19x with almost the same accuracy when compared to the original PF-Net. Experimental results demonstrate that the improved PF-Net can be applied to efficiently complete vehicle point clouds in autonomous driving.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 4

page 7

page 10

03/01/2020

3D Point Cloud Processing and Learning for Autonomous Driving

We present a review of 3D point cloud processing and learning for autono...
08/08/2020

VPC-Net: Completion of 3D Vehicles from MLS Point Clouds

Vehicles are the most concerned investigation target as a dynamic and es...
11/21/2020

MoNet: Motion-based Point Cloud Prediction Network

Predicting the future can significantly improve the safety of intelligen...
03/16/2018

The ApolloScape Dataset for Autonomous Driving

Scene parsing aims to assign a class (semantic) label for each pixel in ...
07/23/2019

Conf-Net: Predicting Depth Completion Error-Map ForHigh-Confidence Dense 3D Point-Cloud

This work proposes a new method for depth completion of sparse LiDAR dat...
07/23/2019

Conf-Net: Predicting Depth Completion Error-Map For High-Confidence Dense 3D Point-Cloud

This work proposes a new method for depth completion of sparse LiDAR dat...
04/18/2021

Self-Supervised Pillar Motion Learning for Autonomous Driving

Autonomous driving can benefit from motion behavior comprehension when i...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

With the wide use of internet-based sensors, a new concept termed as the Internet of Things (IoT) has emerged to facilitate the development of modern industry 1 ; 19 . The ingenious application of intelligent logistics by introducing Industrial Internet of Things (IIoT) significantly promotes the development of the transportation industry, where autonomous driving is the key technology. 3D vision using LiDAR has widely used in autonomous driving due to its lower cost, more robust, richer information, and meeting the mass-production standards 2

. Typically, to detect obstacles and other relevant driving information, the computer vision system receives and analyzes the original point cloud from LiDAR, where the appearance of incomplete point clouds is inevitable owing to limitations of weather, sensor capability, and viewing angle

3 . The emergence of incomplete point clouds, especially fragmentary vehicle point clouds, may cause the reduction of the accuracy of autonomous driving vehicles in object detection, traffic alert, and collision avoidance.

For the above problem, traditional methods, such as interpolation algorithms

20 ; 21 ; 22

, cannot describe the complex geometric characteristics of the vehicle without reference points. In recent years, deep learning is gradually widely applied to 3D vision systems with the rapid development of machine learning. Since the advent of the classic PointNet

4 showing powerful point clouds operating ability, various deep-learning based approaches are developed to solve 3D vision tasks directly 5 ; 6 ; 7 ; 8 . For the point cloud completion, Panos et al. 1 proposed the first deep-learning based network by utilizing an Encoder-Decoder framework. Point Completion Network (PCN) 10 performs shape completion tasks by combining the advantages of Latent-space Generative Adversarial Network (L-GAN) 9 and FoldingNet 11

. Moreover, the extensions of Convolutional Neural Network (CNN) in 3D have been demonstrated to perform well when 3D voxel grids are used to represent 3D shapes

12 ; 13 ; 14 , where the CNN is computationally expensive for high voxel resolution. Muhammad et al. 15

introduced the first reinforcement learning agent controlled Generative Adversarial Network (GAN) to generate point clouds quickly. Moreover, to repair the incomplete point cloud without modifying the position of known points, an unsupervised point repair network based on GAN, Point Fractal Network (PF-Net)

3 , is proposed by paying attention to the local details of an object, which makes it perform well in accuracy.

However, these existing deep-learning based networks for point cloud completion focus on the accuracy of point cloud completion, without considering the efficiency of the inference process, which makes it difficult for them to be deployed for vehicle point cloud repair in autonomous driving, e.g., the state-of-the-art point cloud repair network, PF-Net, has unsatisfactory inference speed due to the low efficiency of its downsampling module.

Downsampling is a basic operation for large-scale point clouds, which can improve the efficiency of subsequent algorithms, including 3D reconstruction 28 ; 29 , point cloud generation 30 ; 31 , and recognition 32 ; 33 . Uniform downsampling, one of downsampling, is used to obtain uniform sparse point clouds consistent with the geometric shape of the target, so as to remove noise while preserving the geometric features of the target. In the uniform downsampling algorithms, there are two commonly used algorithms: Iterative Farthest Point Sampling (IFPS) 17 and cell sampling 34 . As an incremental sampling algorithm 35 ; 36 , the IFPS is widely applied in point cloud networks such as PointNet++ 17 and PF-Net because of its capability to control number of sampling points. However, the IFPS has poor performance in efficiency due to the characteristics of the incremental sampling algorithm. Cell sampling, as a one-time sampling method 36 , has an excellent performance in efficiency, but it cannot precisely control the number of sampling points.

In this paper, our objective is to repair incomplete vehicle point clouds accurately and efficiently, which can serve object detection, traffic alert, and collision avoidance to improve driving safety in autonomous driving. To address the problem, we proposed to complete incomplete vehicle point clouds of self-driving vehicles using the deep learning approach, i.e., the improved PF-Net. In the improved method, we proposed an efficient uniform downsampling algorithm (Cell-IFPS) by combining the advantages of the IFPS and cell sampling, which significantly improves the inference efficiency of original PF-Net.

The contributions of this paper can be summarized as follows:

(1) We employ a deep learning approach (i.e., the improved PF-Net) to repair incomplete vehicle point clouds that cannot be addressed by traditional methods in autonomous driving.

(2) We improve the efficiency of original PF-Net to meet the requirement of point cloud completion of self-driving vehicles, where an efficient uniform downsampling algorithm by combining the advantages of the IFPS and cell sampling (i.e., the high efficiency and the ability to precisely control the number of sampling points) is presented.

(3) We evaluate the method on a real-world vehicle point cloud collected from a real vehicle self-driving process and three incomplete vehicle point clouds with 5 different sizes collected from high-fidelity vehicle models.

The rest of the paper is organized as follows. In Section 2, the significance and the implementation process of incomplete vehicle information completion, and the architecture of the improved PF-Net are introduced. In Section 3, the experimental design and data are briefly introduced, then the repair results of incomplete vehicle point clouds are shown. In Section 4, we had some discussions based on the experimental results. In Section 5, some conclusions are obtained.

2 Method

2.1 Overview

In this paper, we propose to complete incomplete vehicle point clouds of self-driving vehicles using the deep learning approach due to the limitations of traditional data completion methods, such as interpolation algorithm. As demonstrated in Fig. 1, the incomplete point cloud obtained by using LiDAR is inevitable owing to limitations of viewing angle (see Fig. 1(a)-(c)), which may cause the self-driving vehicle to misjudge the size of the surrounding car, resulting in the appearance of collision (see Fig. 1 Case 1). To avoid this potentially dangerous situation, we first normalize the incomplete vehicle point cloud obtained by using LiDAR and then input it into the point cloud completion networks based on GAN (i.e., the improved PF-Net) to repair the point incomplete cloud. Finally, the self-driving vehicle design a safe route according to the complete vehicle point cloud to avoid collision (see Fig. 1 Case 2).

Figure 1: Illustration of the proposed deep learning approach using improved GAN for incomplete information completion of self-driving vehicle

2.2 Improved GAN-based Point Cloud Completion

In this section, the specific framework of improved PF-Net will be introduced (see Fig. 2), which predicts the missing part of the point cloud from its incomplete known configuration. Compared with the original PF-Net, we mainly modified the sampling step in the overall architecture. The overall architecture of improved PF-Net consists of three fundamental parts, including Multi-Resolution Encoder (MRE), Point Pyramid Decoder (PPD), and Discriminator Network. The PF-Net is a generative adversarial network 16 , and the Generator consists of MRE and PPD.

Figure 2: The overall architecture of improved PF-Net

First, the improved PF-Net uses Cell-IFPS to extract feature points from the input point cloud in the MRE, which are representative points that can map the whole shape of the input. Then, three scales point clouds including the extracted feature points and the original points are mapped into three individual latent vectors

() by three independent Combined Multi-Layer Perceptions (CMLPs). Finally, the combined latent vector is mapped into the final feature vector by a Multi-Layer Perception (MLP) after those three individual latent vectors are concatenated (see Fig. 2). In the improved PF-Net, CMLP is proposed as the feature extractor. The common MLP, such as PointNet-MLP, cannot combine the low-level and middle-level features well when it only applies the final single layer to obtain the latent vector. Compared with common MLP, the CMLP is found to be able to extract combined latent vector after concatenating multiple dimensional potential vectors stemed from the last four layers (low-level, middle-level, and high-level features), which leads to better use of multi-layer features.

The objective of PDD is to output point clouds with three different resolutions which represent the shape of the missing region by the final feature vector obtained in the previous step. Therefore, the vector is first transformed into three feature results with different dimensions (where neurons, and ) by three fully-connected layers. Then, point clouds with different resolutions are predicted according to different features .

The loss function is created by combining multi-stage completion loss (MCS) and adversarial loss in the improved PF-Net. The MCS is used to minimize the difference between the predicted three stages of point cloud and Ground Truth (GT). The combined loss for three stages is as follows:

(1)

where is the primary center points which be predicted by , is the secondary center points which be predicted by and . The generation of similar to . Moreover, is the ground truth of missing point cloud, and and are obtained from by applying IFPS.

is the hyperparameter.

The equation of Chamber Distance (CD) is as follows:

(2)

where is the predicted point cloud, and is the ground truth point cloud.

The adversarial loss which is inspired by GAN is defined as follows:

(3)

where which is the partial input, which is the real missing region, ( is the dataset size of ,). is the discriminator which is used to distinguish the predicted missing point cloud and the real missing point cloud. The function is defined as which maps the partial input into predicted missing region .

Finally, the joint loss of the MCS and the adversarial loss is defined as:

(4)

where is the weight of completion loss , is the weight of adversarial loss , .

2.3 Cell-IFPS

Herein, the proposed Cell-IFPS is introduced in detail because it is the key to improve the efficiency of the original PF-Net.

In the original PF-Net, the inefficient IFPS is used due to its ability to precisely control the number of samples, resulting in the original PF-Net inefficiencies for autonomous driving scenarios. As shown in Fig. 3(a), the IFPS uses a circular iterative approach to sampling, the characteristics of which make it inefficient and difficult to accelerate using parallel strategies 23 ; 24 ; 25 . In the IFPS, we first establish the point cloud set and the sampling set where the sampling points are stored, and then randomly sample a point from as the seed (the starting sampling point) and puts it into set . Finally, we sample one farthest point at a time and puts it into set until the number of sampling points meets the requirements. The farthest point is the one with the greatest distance from the set among the remaining points set , and the greatest distance is defined as follows:

(5)

where indicates that the distance between and each point in the set is calculated.

Figure 3: An illustration of downsampling algorithms ((a) IFPS; (b) Cell sampling; (c) Cell-IFPS.)

As shown in Fig. 3(b), there is another common method: cell sampling, which downsamples the point cloud by constructing spheres with a specified radius, and outputs the point in each sphere closest to the center of the sphere as the sampling point. The sampling method is extremely efficient, but the number of samples can only be controlled by adjusting the radius of the sphere.

Therefore, to sample efficiently and exactly control the number of samples, the efficient Cell-IFPS by combining the advantages of the IFPS and cell sampling is proposed, as shown in Fig. 3(c), the sampling steps are as follows:

(1) Set number of samples .

(2) Sample the point cloud using the cell sampling, and create the point cloud , the sampling set , and the remaining points set .

(3) Determine whether the current number of samples in the set . is greater than the number of samples . If it is equal, the sampling is finished, if not, the next step is performed.

(4) Randomly sample a point from as the seed, and determine whether the current number of samples in the set is equal to the number of samples . If , the IFPS is used to sample ) points from and these points are moved to ; If , the IFPS is used to sample ) points from and these points are moved to .

3 Results

In this section, to evaluate the performance of the improved PF-Net, we use a real-world vehicle point cloud and three incomplete vehicle point clouds collected from high-fidelity models to test it.

3.1 Experimental Environment

The details of the experimental environment are listed in Table 1.

Specifications Details
CPU Intel Xeon 5118
CPU RAM 128 GB
CPU Frequency 2.30 GHz
GPU Quadro P6000
CUDA Cores 3840
GPU RAM 24 GB
OS Window 10
Python Version 3.7
Pytorch Version 1.8
CUDA version Version 10.2
Table 1: The platforms used for testing

3.2 Data Description

The training dataset of the model is the ShapeNet Part 39 , which is a dataset that selects 16 categories from the ShapeNetCore dataset 40 and annotated with semantic information for the semantic segmentation task of point clouds.

As shown in Fig. 4, to simulate real autonomous driving scenarios, we set up a two-way road with two lanes in an industrial estate (see Fig. 3(d)). In our experiment, there are three different cars around the red autonomous vehicle, and in addition, we obtained point clouds of 5 different scales from each car. The missing regions of the vehicle point clouds obtained by LiDAR scanning are also different due to the limitation of viewing angle (LiDAR Scan in Fig. 4). Therefore, we have completed these three missing vehicle point clouds that are common in the real world (see Fig. 4(a)-(c)). It should be noted that the vehicle point clouds with missing regions are actually complete, but for testing, incomplete vehicle point clouds are assumed.

Figure 4: An Illustration of three incomplete vehicle point clouds completion for the experimental test in autonomous driving

As shown in Fig. 5, there is an incomplete vehicle point cloud with 2341 points from the famous KITTIT dataset 37 ; 38 , which is the real dataset collected in the self-driving scenarios. Thus, it should be noted that the real vehicle point cloud is incomplete, and we don’t have its real missing region.

Figure 5: An incomplete vehicle point cloud from the real self-driving dataset KITTIT

3.3 Completion Results on Vehicle Point Clouds of Different Scales

In this section, we used the original PF-Net and the improved PF-Net to complete the vehicle point clouds of different sizes, where we utilized the original PF-Net as a baseline. Then, we used Pred GT (prediction to ground truth) error and GT Pred (ground truth to prediction) error to evaluate the results of completion. Finally, we obtained the running time of each step of the models.

3.3.1 Completion Accuracy on Vehicle Point Clouds of Different Scales

To evaluate the accuracy of completion results in autonomous driving, we use Pred GT error and GT

Pred error as evaluation metrics, which are widely used to evaluate the quality of point cloud generation

10 ; 17 . Pred GT error means the mean squared distance between each predicted point and its nearest point in the ground truth. It measures the difference between the prediction and the truth. GT Pred error is similar to the Pred → GT error.

Number of Points Algorithm Vehicle 1 Vehicle 2 Vehicle 3 Mean
2048 Original PF-Net 1.030/0.995 1.300/1.505 1.292/1.220 1.207/1.240
Improved PF-Net 1.092/0.985 1.270/1.354 1.322/1.200 1.228/1.180
4048 Original PF-Net 0.744/0.937 0.937/1.600 0.910/1.223 0.864/1.253
Improved PF-Net 0.775/0.922 0.971/1.435 0.898/1.236 0.881/1.198
6048 Original PF-Net 0.627/0.934 0.804/1.571 0.738/1.222 0.723/1.242
Improved PF-Net 0.659/0.924 0.820/1.453 0.760/1.222 0.746/1.200
8048 Original PF-Net 0.582/0.936 0.785/1.539 0.704/1.247 0.690/1.240
Improved PF-Net 0.641/0.934 0.753/1.536 0.704/1.244 0.699/1.238
10048 Original PF-Net 0.584/0.945 0.682/1.603 0.653/1.218 0.640/1.255
Improved PF-Net 0.620/0.952 0.694/1.502 0.651/1.219 0.655/1.224
Table 2: The completion errors (Pred GT/GT Pred) of the incomplete vehicle point clouds in autonomous driving

As listed in Table 2, the point cloud repair for Vehicle 1 has the best performance, and the accuracy of the point cloud completion for Vehicle 2 is the lowest. Moreover, when completing the same vehicle, as the point cloud size (number of points) increases, the Pred GT error of completion tends to decrease significantly, and the GT Pred error of completion doesn’t decrease or increase significantly (see Fig. 6).

Figure 6: Average Pred GT error and GT Pred error of point cloud completion for different sizes ((a) Average Pred GT error; (b) Average GT → Pred error.)

In conclusion, as shown in Fig. 6, The average Pred GT error of the improved PF-Net is slightly larger (approximately 2.1%) than that of the original PF-Net, and the average GT Pred error of the improved PF-Net is slightly smaller (approximately 3.1%) than that of the original PF-Net. The results indicated that there is no significant difference in the accuracy of the improved PF-Net and the original PF-Net, and both increase with the increase of the number of points.

3.3.2 Completion Efficiency on Vehicle Point Clouds of Different Scales

Since the point cloud completion serves self-driving vehicles, we focus on the inference efficiency of deep neural networks in this paper. The inference process of completion networks consists of two modules: (1) sampling module and (2) generation module. As shown in Fig.

2, the sampling module includes two downsampling processes. First, the input vehicle point cloud needs to be downsampled into a point cloud with 1024 points. Second, the point cloud 1024 points are downsampled into a point cloud with 512 points. Finally, we can obtain three scales point clouds that meet the input requirements of the generation module (i.e., the generator).

Number of Points Algorithm Vehicle 1 Vehicle 2 Vehicle 3
t1 t2 t3 t1 t2 t3 t1 t2 t3
2048 Original PF-Net 271.1 5.9 277.0 263.3 6.0 269.3 279.2 5.0 284.2
Improved PF-Net 6.0 7.0 13.0 14.9 5.0 19.9 6.0 5.0 11.0
4048 Original PF-Net 392.9 6.9 399.8 396.9 5.0 401.9 411.0 5.9 416.9
Improved PF-Net 7.9 6.0 13.9 9.0 6.0 15.0 8.9 5.9 14.8
6048 Original PF-Net 527.6 6.0 533.6 533.7 6.0 539.7 524.6 6.0 530.6
Improved PF-Net 11.1 6.0 17.1 15.0 6.0 21.0 11.0 6.0 17.0
8048 Original PF-Net 637.3 6.0 643.3 652.2 6.0 658.2 639.8 6.0 645.8
Improved PF-Net 14.0 5.1 19.1 15.9 5.0 20.9 11.9 6.0 17.9
10048 Original PF-Net 759.9 5.3 765.2 775.9 6.0 781.9 790.9 6.8 797.7
Improved PF-Net 14.9 5.0 19.9 17.9 6.0 23.9 14.0 6.0 20.0
Table 3: Running time (ms) of point cloud completion on vehicle point clouds of different scales (Where is the sampling module running time, is the generation module running time, and is the total time, i.e., .)

As listed in Table 3, when the number of points and the model are the same, there is little difference in the running time of each module for the completion of different vehicle point clouds. As the number of points increases, the sampling module running time and the total time tend to increase when the model is the same, and the sampling module running time and the total time of the original PF-Net are much larger than those of the improved PF-Net when the number of points is the same (see Fig. 7). In any case, the generation module running time is always approximately 6 ms.

Figure 7: Average total time and average sampling module running time in original PF-Net and improved PF-Net ((a) Average total time; (b) Average sampling module running time.)

As shown in Fig. 8(a), for the original PF-Net, the sampling module running time is much larger (265.6
769.6 ms) than the generation module running time, and the difference between them increases as the number of points increases (the percentage of the sampling module running time in the total time increases from 97.98% to 99.23%), which means that the sampling module is the key factor in the efficiency of point cloud completion, and its impact increases with the number of points. As shown in Fig. 8(b), for the improved PF-Net, the sampling module running time is only a little larger (2.29.9 ms) than the generation module running time, and the difference between them increases as the number of points increases (the percentage of the sampling module running time in the total time increases from 60.96% to 73.24%).

Figure 8: Average running time (ms) bar plots of sampling module and generation module ((a) Original PF-Net; (b) Improved PF-Net.)

As shown in Fig. 9, compared with the original PF-Net, the efficiency of the proposed completion networks (the improved PF-Net) is greatly improved, and the speedup of overall completion is 1936.7. Moreover, compared with the IFPS algorithm, the efficiency of the proposed Cell-IFPS is greatly improved, the speedup of the sampling module using the proposed Cell-IFPS is 30.549.7.

Figure 9: Speedups of overall completion and sampling module in the improved PF-Net (, where is the running time of overall completion or sampling module in the original PF-Net, and is the running time of overall completion or sampling module in the improved PF-Net.)

The aforementioned comparative results indicate that:

(1) The efficiency of the sampling module is the key factor in the efficiency of point cloud completion, and its impact increases with the number of points.

(2) Compared with the original PF-Net, the efficiency of the improved PF-Net is greatly improved due to the efficient Cell-IFPS, and the maximum speedup is 36.7.

(3) Compared with the IFPS algorithm, as the scale of the point clouds increases, the acceleration effect of the proposed Cell-IFPS increases, and the maximum speedup is 49.7.

3.4 Completion Results on a Real-world Vehicle Point Cloud

In this section, for the incomplete vehicle point cloud from the real self-driving process, we specifically introduce the completion efficiency of the original PF-Net and the improved PF-Net, and briefly describe the completion effect.

As shown in Fig. 10, for the incomplete vehicle point cloud, both original PF-Net and improved PF-Net successfully complete the missing area of the vehicle so that the point cloud has the complete shape of the vehicle after the repair, and the missing point clouds generated by the original PF-Net and the improved PF-Net are almost identical.

Figure 10: An Illustration of vehicle point cloud completion on a real-world vehicle point cloud in self-driving

As listed in Table 4, the total running time of the improved PF-Net is much less than that of the original PF-Net, and the speedup is 30.9; the sampling module running time of the improved PF-Net is much less than that of the original PF-Net, and the speedup is 21.7. The improved PF-Net performs a point cloud completion in only 20 ms during autonomous driving, i.e., the speed of the improved PF-Net is approximately 46 fps during autonomous driving, which satisfies the requirement for real-time vehicle point cloud completion for autonomous driving.

Module Original PF-Net Improved PF-Net Speedup
Sampling Module 368.1 11.9 30.9
Generation Module 4.9 5.3 0.9
Total 373.0 17.2 21.7
Table 4: Running time (ms) of point cloud completion on a real self-driving vehicle point cloud (, where is the total running time or the sampling module running time in the original PF-Net, and is the total running time or the sampling module running time in the improved PF-Net.)

4 Discussion

4.1 Advantages of the proposed deep learning method

The advantage of the proposed method is to consider the efficiency of point cloud completion networks for self-driving vehicles. More specifically, in this paper, we found that the sampling algorithm is the key factor that restricts the efficiency of point cloud completion networks. Therefore, we proposed an efficient sampling algorithm Cell-IFPS by combining the one-time sampling algorithm Cell sampling with the incremental sampling algorithm IFPS, which can precisely control the number of sampling points. The proposed Cell-IFPS substantially improves the inference speed of the original PF-Net without using a parallel strategy for acceleration. In autonomous driving scenarios, the tasks are generally time-critical. As listed in Table 4, the point cloud completion efficiency of the original PF-Net is difficult to meet the needs of self-driving, only 3 fps, and the point cloud completion efficiency of the improved PF-Net is 46 fps, which is enough to meet the requirement of speed. Moreover, as shown in Fig. 7, the efficiency of the improved PF-Net is insensitive to changes in the size of the point cloud, i.e., an increase in the number of points doesn’t lead to a sharp decrease in its efficiency, which gives it the potential to serve tasks such as the 3D reconstruction of large-scale point clouds.

On the other hand, for the incomplete vehicle point cloud repair, the accuracy of improved PF-Net is almost the same as that of the original PF-Net, which performs well in accuracy and surpassing the widely used point cloud generation model PCN [3,10]. The improved PF-Net with high accuracy can be effectively used to realize the real-time vehicle point cloud completion to enhance the safety of autonomous driving.

4.2 Shortcomings of the proposed deep learning method

The shortcoming of the proposed method is that the accuracy of the point cloud completion may be reduced or even the point cloud completion may fail when the scale of the input point cloud is too small. According to the network architecture, the input point cloud needs to use downsampling to extract the feature points, in this paper, the input point cloud is downsampled by 1024 points and 512 points, which means that the improved PF-Net cannot complete the point cloud when the size of the input point cloud is less than 1024 points. On the other hand, for the real self-driving data, we lack the real point cloud of the missing part of the incomplete vehicle point cloud. Therefore, we don’t use the corresponding evaluation metric to evaluate the completion accuracy.

4.3 Outlook and Future Work

In the future, we plan to apply the improved PF-Net to repair incomplete point clouds that include more kinds of obstacles (e.g., trees, roadblocks, telegraph poles) in autonomous driving. Further, the proposed efficient uniform sampling method, Cell-IFPS, will be applied to more tasks for point cloud (e.g., 3D reconstruction, object recognition, point cloud segmentation) due to the sampling being one of the foundations of point cloud processing.

There are several techniques to improve the efficiency of deep learning, including CPU/GPU-accelerated parallel technology and TensorRT technology 18 ; 26 ; 27 . In the future, we will use parallel technology to speed up the sampling process or use the TensorRT technology to accelerate the generation process at the cost of slightly reducing the accuracy.

Moreover, the deployment of deep learning in actual production is still difficult. Compared with Python language, the deployment of deep learning algorithms developed by C/C++ language is more convenient. Therefore, we will consider developing or using API to convert the Python version of the improved PF-Net to the C/C++ version to facilitate deployment.

5 Conclusion

In this paper, to meet the requirements of algorithm efficiency for incomplete point cloud completion in the process of self-driving, we proposed the improved PF-Net to repair incomplete vehicle point cloud accurately and efficiently in autonomous driving. The essential idea of the method is to use efficient deep learning for real-time incomplete vehicle point cloud completion to improve the safety of self-driving vehicles. In the proposed method, an efficient down sampling combining incremental sampling and one-time sampling, Cell-IFPS, is presented to significantly improve the inference speed of the original PF-Net. To evaluate the performance of the proposed method, a real dataset is used, and an autonomous driving scene is created, where three incomplete vehicle point clouds with 5 different sizes are set for three autonomous driving situations. The results show that: (1) inefficient sampling module is the key to restrict the efficiency of the original PF-Net; (2) the proposed Cell-IFPS can greatly improve the efficiency of the original PF-Net without using a parallel strategy for acceleration, and the enhancement effect increases with the increase of point cloud scale; (3) the improved PF-Net has far greater speed than the original PF-Net and almost the same accuracy as the original PF-Net, which is the state-of-the-art point cloud repair network in accuracy. The improved PF-Net has the capability to complete incomplete vehicle point clouds of self-driving vehicles.

Acknowledgements.
This research was jointly supported by the National Natural Science Foundation of China (Grant No. 11602235), and the Fundamental Research Funds for China Central Universities (2652018091).

References

  • [1] Ruhul Amin Khalil, Nasir Saeed, Mudassir Masood, Yasaman Moradi Fard, Mohamed-Slim Alouini, and Tareq Y. Al-Naffouri. Deep learning in the industrial internet of things: Potentials, challenges, and emerging applications. IEEE Internet of Things Journal, 8(14):11016–11040, 2021.
  • [2] M. Barbareschi, V. Casola, A. Debenedictis, E. La Montagna, and N. Mazzocca. On the adoption of physically unclonable functions to secure iiot devices. IEEE Transactions on Industrial Informatics, 2021.
  • [3] J. Fang, D. Zhou, F. Yan, T. Zhao, F. Zhang, Y. Ma, L. Wang, and R. Yang. Augmented lidar simulator for autonomous driving. IEEE Robotics and Automation Letters, 5(2):1931–1938, 2020.
  • [4] Z. Huang, Y. Yu, J. Xu, F. Ni, and X. Le. Pf-net: Point fractal network for 3d point cloud completion.

    Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

    , pages 7659–7667, 2020.
  • [5] V. Skala. Rbf interpolation with csrbf of large data sets. Procedia Computer Science, 108:2433–2437, 2017.
  • [6] G. Mei and H. Tian. Impact of data layouts on the efficiency of gpu-accelerated idw interpolation. SpringerPlus, 5(1):1–18, 2016.
  • [7] A. Breglia, A. Capozzoli, C. Curcio, and A. Liseno. Nufft-based interpolation in backprojection algorithms. IEEE Geoscience and Remote Sensing Letters, 2020.
  • [8] C.R. Qi, H. Su, K. Mo, and L.J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-January:77–85, 2017.
  • [9] X. Han, Z. Li, H. Huang, E. Kalogerakis, and Y. Yu. High-resolution shape completion using deep neural networks for global structure and local geometry inference. Proceedings of the IEEE International Conference on Computer Vision, 2017-October:85–93, 2017.
  • [10] C.-L. Li, M. Zaheer, Y. Zhang, B. Póczos, and R. Salakhutdinov. Point cloud gan. Deep Generative Models for Highly Structured Data, DGS@ICLR 2019 Workshop, 2019.
  • [11] J. Wu, C. Zhang, T. Xue, W.T. Freeman, and J.B. Tenenbaum. Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. Advances in Neural Information Processing Systems, pages 82–90, 2016.
  • [12] Y. Li, A. Dai, L. Guibas, and M. Nießner. Database-assisted object retrieval for real-time 3d reconstruction. Computer Graphics Forum, 34(2):435–446, 2015.
  • [13] W. Yuan, T. Khot, D. Held, C. Mertz, and M. Hebert. Pcn: Point completion network. Proceedings - 2018 International Conference on 3D Vision, 3DV 2018, pages 728–737, 2018.
  • [14] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas. Learning representations and generative models for 3d point clouds. 6th International Conference on Learning Representations, ICLR 2018 - Workshop Track Proceedings, 2018.
  • [15] Y. Yang, C. Feng, Y. Shen, and D. Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 206–215, 2018.
  • [16] D. Stutz and A. Geiger. Learning 3d shape completion under weak supervision. International Journal of Computer Vision, 128(5):1162–1181, 2020.
  • [17] A. Dai, C.R. Qi, and M. Nießner. Shape completion using 3d-encoder-predictor cnns and shape synthesis. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-January:6545–6554, 2017.
  • [18] A. Dai, D. Ritchie, M. Bokeloh, S. Reed, J. Sturm, and M. Niebner. Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 4578–4587, 2018.
  • [19] M. Sarmad, H.J. Lee, and Y.M. Kim. Rl-gan-net: A reinforcement learning agent controlled gan network for real-time point cloud shape completion. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-June:5891–5900, 2019.
  • [20] S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, and A. Fitzgibbon. Kinectfusion: Real-time 3d reconstruction and interaction using a moving depth camera. UIST’11 - Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pages 559–568, 2011.
  • [21] A. Morales, G. Piella, and F.M. Sukno. Survey on 3d face reconstruction from uncalibrated images. Computer Science Review, 40, 2021.
  • [22] S. Shi, X. Wang, and H. Li. Pointrcnn: 3d object proposal generation and detection from point cloud. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-June:770–779, 2019.
  • [23] G. Yang, X. Huang, Z. Hao, M.-Y. Liu, S. Belongie, and B. Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. Proceedings of the IEEE International Conference on Computer Vision, 2019-October:4540–4549, 2019.
  • [24] K. Kim, C. Kim, C. Jang, M. Sunwoo, and K. Jo. Deep learning-based dynamic object classification using lidar point cloud augmented by layer-based accumulation for intelligent vehicles. Expert Systems with Applications, 167, 2021.
  • [25] X.-F. Han, X.-Y. Yan, and S.-J. Sun. Novel methods for noisy 3d point cloud based object recognition. Multimedia Tools and Applications, 80(17):26121–26143, 2021.
  • [26] C.R. Qi, L. Yi, H. Su, and L.J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in Neural Information Processing Systems, 2017-December:5100–5109, 2017.
  • [27] Dominika Mikšová, Christopher Rieser, Peter Filzmoser, Maarit Middleton, and Raimo Sutinen. Identification of mineralization in geochemistry for grid sampling using generalized additive models. Mathematical Geosciences, 2021.
  • [28] Z. Xu, D. Deng, and K. Shimada. Autonomous uav exploration of dynamic environments via incremental sampling and probabilistic roadmap. IEEE Robotics and Automation Letters, 6(2):2729–2736, 2021.
  • [29] X. Zhang, L. Zong, Q. You, and X. Yong.

    Sampling for nyström extension-based spectral clustering: Incremental perspective and novel analysis.

    ACM Transactions on Knowledge Discovery from Data, 11(1), 2016.
  • [30] I.J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems, 3(January):2672–2680, 2014.
  • [31] V.E. Malyshkin. Parallel computing technologies 2018: Automatic parallel implementation of applications. Journal of Supercomputing, 75(12):7747–7749, 2019.
  • [32] Z. Huo, G. Mei, G. Casolla, and F. Giampaolo. Designing an efficient parallel spectral clustering algorithm on multi-core processors in julia. Journal of Parallel and Distributed Computing, 138:211–221, 2020.
  • [33] Y. Cai, X. Cui, G. Li, and W. Liu. A parallel finite element procedure for contact-impact problems using edge-based smooth triangular element and gpu. Computer Physics Communications, 225:47–58, 2018.
  • [34] L. Yi, V.G. Kim, D. Ceylan, I.-C. Shen, M. Yan, H. Su, C. Lu, Q. Huang, A. Sheffer, and L. Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions on Graphics, 35(6), 2016.
  • [35] D. Stutz and A. Geiger. Learning 3d shape completion under weak supervision. International Journal of Computer Vision, 128(5):1162–1181, 2020.
  • [36] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. pages 3354–3361, 2012.
  • [37] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The kitti dataset. International Journal of Robotics Research, 32(11):1231–1237, 2013.
  • [38] I. Pacal and D. Karaboga. A robust real-time deep learning based automatic polyp detection system. Computers in Biology and Medicine, 134, 2021.
  • [39] E. Smistad. Fast: A framework for high-performance medical image computing and visualization. ACM International Conference Proceeding Series, 2021.
  • [40] J. Fang, Q. Liu, and J. Li. A deployment scheme of yolov5 with inference optimizations based on the triton inference server. 2021 IEEE 6th International Conference on Cloud Computing and Big Data Analytics, ICCCBDA 2021, pages 441–445, 2021.