Deep Learning for 3D Point Clouds
Point cloud learning has lately attracted increasing attention due to its wide applications in many areas, such as computer vision, autonomous driving, and robotics. As a dominating technique in AI, deep learning has been successfully used to solve various 2D vision problems. However, deep learning on point clouds is still in its infancy due to the unique challenges faced by the processing of point clouds with deep neural networks. Recently, deep learning on point clouds has become even thriving, with numerous methods being proposed to address different problems in this area. To stimulate future research, this paper presents a comprehensive review of recent progress in deep learning methods for point clouds. It covers three major tasks, including 3D shape classification, 3D object detection and tracking, and 3D point cloud segmentation. It also presents comparative results on several publicly available datasets, together with insightful observations and inspiring future research directions.READ FULL TEXT VIEW PDF
Point cloud is point sets defined in 3D metric space. Point cloud has be...
This paper presents a new 3D point cloud classification benchmark data s...
Over the past decade deep learning has driven progress in 2D image
In this paper, we propose a method to segment regions in three-dimension...
We present a new deep learning architecture (called Kd-network) that is
To help bridge the gap between internet vision-style problems and the go...
Researchers have now achieved great success on dealing with 2D images us...
Deep Learning for 3D Point Clouds
With the rapid development of 3D acquisition technologies, 3D sensors are becoming increasingly available and affordable, including various types of 3D scanners, LiDARs, and RGB-D cameras (such as Kinect, RealSense and Apple depth cameras) . 3D data acquired by these sensors can provide rich geometric, shape and scale information [51, 50]. Complemented with 2D images, 3D data provides an opportunity for a better understanding of the surrounding environment for machines. 3D data has numerous applications in different areas, including autonomous driving, robotics, remote sensing, medical treatment, and design industry .
3D data can usually be represented with different formats, including depth images, point clouds, meshes, and volumetric grids. As a commonly used format, point cloud representation preserves the original geometric information in 3D space without any discretization. Therefore, it is the preferred representation for many scene understanding related applications such as autonomous driving and robotics. Recently, deep learning techniques have dominated many research areas, such as computer vision, speech recognition, Natural Language Processing (NLP), and bioinformatics. However, deep learning on 3D point clouds still face several significant challenges, such as the small scale of datasets, the high dimensionality and the unstructured nature of 3D point clouds. On this basis, this paper focuses on the analysis of deep learning methods which have been used to process 3D point clouds.
Deep learning on point clouds has been attracting more and more attention, especially in the last five years. Several publicly available datasets are also released, such as ModelNet , ShapeNet , ScanNet , Semantic3D , and the KITTI Vision Benchmark Suite . These datasets have further boosted the research of deep learning on 3D point clouds, with an increasingly number of methods being proposed to address various problems related to point cloud processing, including 3D shape classification, 3D object detection and tracking, and 3D point cloud segmentation. Few surveys of deep learning on 3D data are also available, such as [63, 2, 178, 133]. However, our paper is the first to specifically focus on deep learning methods for point clouds. Besides, our paper comprehensively covers different applications including classification, detection, tracking, and segmentation. A taxonomy of existing deep learning methods for 3D point clouds in shown in Fig. 1.
Compared to the existing literature, the major contributions of this work can be summarized as follows:
To the best of our knowledge, this is the first survey paper to comprehensively cover deep learning methods for several important point cloud related tasks, including 3D shape classification, 3D object detection and tracking, and 3D point cloud segmentation.
This paper covers the most recent and advanced progress of deep learning on point clouds. Therefore, it provides the reader with the state-of-the-art methods.
The structure of this paper is as follows. Section 2 reviews the methods for 3D shape classification. Section 3 provides a survey of existing methods for 3D object detection and tracking. Section 4 presents a review of methods for point cloud segmentation, including semantic segmentation, instance segmentation, and part segmentation. Finally, Section 5 concludes the paper. We also provide a regularly updated project page on: https://github.com/QingyongHu/SoTA-Point-Cloud.
These methods usually learn the embedding of each point first and then extract a global shape embedding from the whole point cloud using an aggregation method. Classification is finally achieved by several fully connected layers. Based on the way that feature learning is performed on each point, existing 3D shape classification methods can be divided into projection-based networks and point-based networks. Several milestone methods are illustrated in Fig. 2.
Projection-based methods first project an unstructured point cloud into an intermediate regular representation, and then leverage the well-established 2D or 3D convolution to achieve shape classification. In contrast, point-based methods directly work on raw point clouds without any voxelization or projection. Point-based methods do not introduce explicit information loss and becoming increasingly popular. In this paper, we mainly focus on point-based networks, but also include few projection-based networks for completeness.
These methods project 3D point clouds into different representation modalities (e.g., multi-view, volumetric representations) for feature learning and shape classification.
These methods first project a 3D object into multiple views and extract the corresponding view-wise features, and then fuse these features for accurate object recognition. How to aggregate multiple view-wise features into a discriminative global representation is a key challenge. MVCNN 
is a pioneering work, which simply max-pools multi-view features into a global descriptor. However, max-pooling only retains the maximum elements from a specific view, resulting in information loss. MHBN integrates local convolutional features by harmonized bilinear pooling to produce a compact global descriptor. Yang et al.  first leveraged a relation network to exploit the inter-relationships (e.g., region-region relationship and view-view relationship) over a group of views, and then aggregated these views to obtain a discriminative 3D object representation. In addition, several other methods [130, 43, 160, 109] have also been proposed to improve the recognition accuracy.
Early methods usually apply 3D Convolution Neural Network (CNN) build upon the volumetric representation of 3D point clouds. Daniel et al. introduced a volumetric occupancy network called VoxNet to achieve robust 3D object recognition. Wu et al. 
proposed a convolutional deep belief-based 3D ShapeNets to learn the distribution of points from various 3D shapes. A 3D shape is usually represented by a probability distribution of binary variables on voxel grids. Although encouraging performance has been achieved, these methods are unable to scale well to dense 3D data since the computation and memory footprint grows cubically with the resolution. To this end, a hierarchical and compact graph structure (such as octree) is introduced to reduce the computational and memory costs of these methods. OctNet
first hierarchically partitions a point cloud using a hybrid grid-octree structure, which represents the scene with several shallow octrees along a regular grid. The structure of octree is encoded efficiently using a bit string representation, and the feature vector of each voxel is indexed by simple arithmetic. Wang et al. proposed an Octree-based CNN for 3D shape classification. The average normal vectors of a 3D model sampled in the finest leaf octants are fed into the network, and 3D-CNN is applied on the octants occupied by the 3D shape surface. Compared to a baseline network based on dense input grids, OctNet requires much less memory and runtime for high-resolution point clouds. Le et al.  proposed a hybrid network called PointGrid, which integrates the point and grid representation for efficient point cloud processing. A constant number of points is sampled within each embedding volumetric grid cell, which allows the network to extract geometric details by using 3D convolutions.
According to the network architecture used for the feature learning of each point, methods in this category can be divided into pointwise MLP, convolution-based, graph-based, data indexing-based networks and other typical networks.
These methods model each point independently using several Multi-Layer Perceptrons (MLPs) and then aggregate a global feature using a symmetric function, as shown in Fig.3. These networks can achieve permutation invariance for unordered 3D point clouds. However, the geometric relationships among 3D points are not fully considered.
As a pioneering work, PointNet  learns pointwise features with several MLP layers and extracts global shape features with a max-pooling layer. The classification score is obtained using several MLP layers. Zaheer et al.  also theoretically demonstrated that the key to achieve permutation invariance is by summing up all representations and applying nonlinear transformations. They also designed a fundamental architecture, DeepSets, for various applications including shape classification .
Since features are learned independently for each point in PointNet , the local structural information between points cannot be captured. Therefore, Qi et al.  proposed a hierarchical network PointNet++ to capture fine geometric structures from the neighborhood of each point. As the core of PointNet++ hierarchy, its set abstraction level is composed of three layers: the sampling layer, the grouping layer and the PointNet layer. By stacking several set abstraction levels, PointNet++ can learn features from a local geometric structure and abstract the local features layer by layer.
introduced a deep autoencoder network to learn point cloud representation. Its encoder follows the design of PointNet and learns point features independently using five 1-D convolutional layers, a ReLU non-linear activation, batch normalization and max-pooling operations. In Point Attention Transformers (PATs), each point is represented by its own absolute position and relative positions with respect to its neighbors. Then, Group Shuffle Attention (GSA) is used to capture relations between points, and a permutation invariant, differentiable and trainable end-to-end Gumbel Subset Sampling (GSS) layer is developed to learn hierarchical features. The architecture of Mo-Net  is similar to PointNet 
but it takes a finite set of moments as the input of its network.PointWeb  is also built upon PointNet++ and uses context of the local neighborhood to improve point features using Adaptive Feature Adjustment (AFA). Duan et al.  proposed a Structural Relational Network (SRN) to learn structural relational features between different local structures using MLP. Lin et al.  accelerated the inference process by constructing a lookup table for both input and function spaces learned by PointNet. The inference time on the ModelNet and ShapeNet datasets is sped up by 1.5 ms and 32 times over PointNet on a moderate machine. SRINet  first projects a point cloud to obtain rotation invariant representations, and then utilizes PointNet-based backbone to extract a global feature and graph-based aggregation to extract local features.
Compared to kernels defined on 2D grid structures (e.g., images), convolutional kernels for 3D point clouds are hard to design due to the irregularity of point clouds. According to the type of convolutional kernels, current 3D convolution networks can be divided into continuous convolution networks and discrete convolution networks, as shown in Fig. 4.
3D Continuous Convolution Networks. These methods define convolutional kernels on a continuous space, where the weights for neighboring points are related to the spatial distribution with respect to the center point.
3D convolution can be interpreted as a weighted sum over a given subset. MLP is a simple way to learn weights. As the core layer of RS-CNN , RS-Conv takes a local subset of points around a certain point as its input, and the convolution is implemented using an MLP by learning the mapping from low-level relations (such as Euclidean distance and relative position) to high-level relations between points in the local subset. In , kernel elements are selected randomly in a unit sphere. An MLP-based continuous function is then used to establish relation between the locations of the kernel elements and the point cloud. In DensePoint , convolution is defined as a Single-Layer Perceptron (SLP) with a nonlinear activator. Features are learned by concatenating features from all previous layers to sufficiently exploit the contextual information.
Some methods also use existing algorithms to perform convolution. In PointConv 
, convolution is defined as a Monte Carlo estimate of the continuous 3D convolution with respect to an importance sampling. The convolutional kernels consist of a weighting function (which is learned with MLP layers) and a density function (which is learned by a kernelized density estimation and an MLP layer). To improve memory and computational efficiency, the 3D convolution is further reduced into two operations: matrix multiplication and 2D convolution. With the same parameter setting, its memory consumption can be reduced by about 64 times. In MCCNN, convolution is considered as a Monte Carlo estimation process relying on a sample’s density function (which is implemented with MLP). Poisson disk sampling is then used to construct a point cloud hierarchy. This convolution operator can be used to perform convolution between two or multiple sampling methods and can handle varying sampling densities. In SpiderCNN , SpiderConv is proposed to define convolution as the product of a step function and a Taylor expansion defined on the
nearest neighbors. The step function captures the coarse geometry by encoding the local geodesic distance, and the Taylor expansion captures the intrinsic local geometric variations by interpolating arbitrary values at the vertices of a cube. Besides, a convolution network PCNN
is also proposed for 3D point clouds based on the radial basis function. Thomas et al. proposed both rigid and deformable kernel point convolution (KPConv) operators for 3D point clouds using a set of learnable kernel points.
Several methods have been proposed to address the rotation equivariant problem faced by 3D convolution networks. Esteves et al. 
proposed 3D spherical convolutional neural networks (Spherical CNN) to learn rotation equivariant representation for 3D shapes, which takes multi-valued spherical functions as its input. Localized convolutional filters are obtained by parameterizing spectrum with anchor points in the spherical harmonic domain. Tensor field networks are proposed to define the point convolution operation as the product of a learnable radial function and spherical harmonics, which are locally equivariant to 3D rotations, translations, and permutations of points. The convolution in 
is defined based on the spherical cross-correlation and implemented using a generalized Fast Fourier Transformation (FFT) algorithm. Based on PCNN, SPHNet achieves rotation invariance by incorporating spherical harmonic kernels during convolution on volumetric functions. ConvPoint  separates the convolution kernel into spatial and feature parts. The locations of the spatial part are randomly selected from a unit sphere and the weighting function is learned through a simple MLP.
To accelerate computing speed, Flex-Convolution  defines weights of convolution kernel as standard scalar product over nearest neighbors, which can be accelerated using CUDA. Experimental results have demonstrated its competitive performance on a small dataset with fewer parameters and lower memory consumption.
3D Discrete Convolution Networks. These methods define convolutional kernels on regular grids, where the weights for neighboring points are related to the offsets with respect to the center point.
Hua et al.  transformed non-uniform 3D point clouds into uniform grids, and defined convolutional kernels on each grid. Unlike 2D convolutions (which assign a weight to each pixel), the proposed 3D kernel assigns the same weights to all points falling into the same grid. For a given point, the mean features of all the neighboring points that are located on the same grid are computed from the previous layer. Then, mean features of all grids are weighted and summed to produce the output of the current layer. Lei et al.  defined a spherical convolutional kernel by partitioning a 3D spherical neighboring region into multiple volumetric bins and associating each bin with a learnable weighting matrix. The output of the spherical convolutional kernel for a point is determined by the non-linear activation of the mean of weighted activation values of its neighboring points. In GeoConv , the geometric relationship between a point and its neighboring points is explicitly modeled based on six bases. Edge features along each direction of the basis is weighted independently by a learnable matrix according to the basis of the neighboring point. These direction-associated features are then aggregated according to an angle formed by the given point and its neighboring points. For a given point, its feature at the current layer is defined as the sum of features of the given point and its neighboring edge features at the previous layer. PointCNN  achieves permutation invariance through a -conv transformation (which is implemented through MLP). By interpolating point features to neighboring discrete convolutional kernel-weight coordinates, Mao et al.  proposed an interpolated convolution operator InterpConv to measure the geometric relations between input point clouds and kernel-weight coordinates. Zhang et al.  proposed a RIConv operator to achieve rotation invariance, which takes low-level rotation invariant geometric features as input and then turns the convolution into 1D by a simple binning approach.
A-CNN  defines an annular convolution by looping the array of neighbors with respect to the size of kernel on each ring of the query point.A-CNN learns the relationship between neighboring points in a local subset. To reduce the computational and memory cost of 3D CNNs, Kumawat et al.  proposed a Rectified Local Phase Volume (ReLPV) block to extract phase in a 3D local neighborhood based on 3D Short Term Fourier Transform (STFT), which significantly reduces the number of parameters. In SFCNN , a point cloud is projected onto regular icosahedral lattices with aligned spherical coordinates. Convolutions are then conducted upon the features concatenated from vertices of spherical lattices and their neighbors through convolution-maxpooling-convolution structures. SFCNN is resistant to rotations and perturbations.
Graph-based networks consider each point in a point cloud as a vertex of a graph, and generate directed edges for the graph based on the neighbors of each point. Feature learning is then performed in spatial or spectral domains . A typical graph-based network is shown in Fig. 5.
Graph-based Methods in Spatial Domain. These methods define operations (e.g., convolution and pooling) in spatial domain. Specifically, convolution is usually implemented through MLP over spatial neighbors, pooling is produces a new coarsened graph by aggregating information from each point’s neighbors. Features at each vertex are usually assigned with coordinates, laser intensities or colors, while features at each edge are usually assigned with geometric attributes between two connected points.
As a pioneering work, Simonovsky et al.  considered each point as a vertex of the graph, and connected each vertex to all its neighbors by a directed edge. Then, Edge-Conditioned Convolution (ECC) is proposed using a filter-generating network (e.g., MLP). Max pooling is adopted to aggregate neighborhood information and graph coarsening is implemented based on the VoxelGrid  algorithm. For shape classification, convolutions and pooling are first interlaced. Then, global average pooling and fully connected layers are followed to produce classification scores. In DGCNN , a graph is constructed in the feature space and dynamically updated after each layer of the network. As the core layer of EdgeConv, an MLP is used as the feature learning function for each edge, channel-wise symmetric aggregation is also applied onto the edge features associated with the neighbors of each point. Further, LDGCNN 
removed the transformation network and linked the hierarchical features from different layers in DGCNN to improve its performance and reduce the mode size. An end-to-end unsupervised deep AutoEncoder network (namely FoldingNet ) is also proposed to use the concatenation of a vectorized local covariance matrix and point coordinates as its input.
Inspired by Inception  and DGCNN , Hassani et al.  proposed an unsupervised multi-task autoencoder to learn point and shape features. The encoder is constructed based on mutli-scale graphs. The decoder is constructed using three unsupervised tasks including clustering, self-supervised classification and reconstruction, which are trained jointly with a mutli-task loss. Liu et al.  proposed a Dynamic Points Agglomeration Module (DPAM) based on graph convolution to simplify the process of points agglomeration (sampling, grouping and pooling) into a simple step, which is implemented through multiplication of the agglomeration matrix and points feature matrix. Based on the PointNet architecture, a hierarchical learning architecture is constructed by stacking multiple DPAMs. Compared to the hierarchy strategy of PointNet++, DPAM dynamically exploits the relation of points and agglomerates points in a semantic space.
To exploit the local geometric structures, KCNet  is proposed to learn features based on kernel correlation. Specifically, a set of learnable points characterizing geometric types of local structures are defined as kernels. Then, affinity between the kernel and the neighborhood of a given point is calculated. In G3D , convolution is defined as a variant of polynomial of adjacency matrix, and pooling is defined as multiplying the Laplacian matrix and the vertex matrix by a coarsening matrix. ClusterNet 
utilizes Rigorously Rotation-Invariant (RRI) module to extract rotation-invariant features for each point, and constructs hierarchical structures of a point cloud based on the unsupervised agglomerative hierarchical clustering method with ward-linkage criteria. The features in each sub-cluster are first learned through an EdgeConv block and then aggregated through max pooling.
Graph-based Methods in Spectral Domain.
These methods define convolutions as spectral filtering, which is implemented as the multiplication of signals on graph with eigenvectors of the graph Laplacian matrix.
To handle the challenges of high computation and non-localization, Defferrard et al.  proposed a truncated Chebyshev polynomials to approximate the spectral filtering. Their learned feature maps are located within the K-hops neighbors of each point. Note, eigenvectors are calculated from a fixed graph Laplacian matrix in  . In contrast, RGCNN 
constructs a graph by connecting each point with all other points in the point cloud and updates the graph Laplacian matrix in each layer. To make features of adjacent vertices more similar, a graph-signal smoothness prior is added into the loss function. To address the challenges caused by diverse graph topology of data, the SGC-LL layer in AGCN utilizes a learnable distance metric to parameterize the similarity between two vertices on the graph. The adjacency matrix obtained from graph is normalized using Gaussian kernels and learned distances. Feng et al.  proposed a Hypergraph Neural Network (HGNN) and built a hyperedge convolutional layer by applying spectral convolution on the hypergraph.
The aforementioned methods operate on full graphs. To exploit the local structural information, Wang et al.  proposed an end-to-end spectral convolution network LocalSpecGCN to work on a local graph (which is constructed from the nearest neighbors). This method does not require any offline computation of the graph Laplacian matrix and graph coarsening hierarchy. In PointGCN , a graph is constructed based on nearest neighbors from a point cloud and each edge is weighted using a Gaussian kernel. Convolutional filters are defined as Chebyshev polynomials in the graph spectral domain. Global pooling and multi-resolution pooling are used to capture global and local features of the point cloud. Pan et al.  proposed 3DTI-Net by applying convolution on the nearest neighboring graph in the spectral domain. The invariance to geometry transformation is achieved by learning from relative Euclidean and direction distances.
These networks are constructed based on different data indexing structures (e.g., octree and kd-tree). In these methods, point features are learned hierarchically from leaf nodes to the root node along a tree. Lei et al.  proposed an octree guided CNN using spherical convolutional kernels (as described in Section 4
). Each layer of the network corresponds to one layer of the octree and a spherical convolutional kernel is applied at each layer. The values of neurons in the current layer are determined as the mean values of all relevant children nodes in the previous layer. Unlike OctNet (which is based on octree), Kd-Net  is built using multiple K-d trees with different splitting directions at each iteration. Following a bottom-up approach, the representation of a non-leaf node is computed from representations of its children using MLP. The feature of the root node (which describes the whole point cloud) is finally fed to fully connected layers to predict classification scores. Note that, Kd-Net shares parameters at each level according to the splitting type of nodes. 3DContextNet  uses a standard balanced K-d tree to achieve feature learning and aggregation. At each level, point features are first learned through MLP based on local cues (which models inter-dependencies between points in a local region) and global contextual cues (which models the relationship for one position with respect to all other positions). Then, the feature of a non-leaf node is computed from its child nodes using MLP and aggregated by max pooling. For classification, the above process is repeated until the root node is attained.
The hierarchy of SO-Net network is constructed by performing point-to-node nearest neighbor search 
. Specifically, a modified permutation invariant Self-Organizing Map (SOM) is used to model the spatial distribution of a point cloud. Individual point features are learned from normalized point-to-node coordinates through a series of fully connected layers. The feature of each node in SOM is extracted from point features associated with this node using channel-wise max pooling. The final feature is then learned from node features using an approach similar to PointNet. Compared to PointNet++ , the hierarchy of SOM is more efficient and the spatial distribution of the point cloud is fully explored.
|Deep Sets ||Coordinates||-||87.1%||-||-||-|
|MC Convolution ||Coordinates||-||90.9%||-||-||-|
|Spherical CNNs ||Coordinates||0.5||88.9%||-||-||-|
|KPConv rigid ||Coordinates||-||92.9%||-||-||-|
|KPConv deform ||Coordinates||-||92.7%||-||-||-|
|Hassani et al. ||Coordinates||-||89.1%||-||-||-|
|Other Networks||3DmFV-Net ||Coordinates||4.6||91.6%||-||95.2%||-|
In addition to the above methods, many other schemes have also been proposed. In 3DmFV 
, a point cloud is voxelized into uniform 3D grids and fisher vectors are extracted based on the likelihood of a set of Gaussian mixture models defined on these grids. Since the components of fisher vector are summed over all points, the resulting representation is invariant to the order, structure and size of point clouds. RBFNet
explicitly models the spatial distribution of points by aggregating features from sparsely distributed Radial Basis Function (RBF) kernels. The RBF feature extraction layer computes responses from all kernels for each point, and then the kernel position and kernel size are optimized to capture the spatial distribution of points during training. Compared to fully connected layers, the RBF feature extraction layer produces more discriminative features while reducing the number of parameters by orders of magnitude. Zhao et al. proposed an unsupervised auto-encoder 3DPointCapsNet for generic representation learning of 3D point clouds. In the encoder stage, a pointwise MLP is first applied to the point cloud to extract point independent features, which are further fed into multiple independent convolutional layers. Then, a global latent representation is extracted by concatenating multiple max-pooled learned feature maps. Based on unsupervised dynamic routing, powerful representative latent capsules are learned. Inspired from the construction of shape context descriptor , Xie et al.  proposed a novel ShapeContextNet architecture by combining affinity point selection and compact feature aggregation into a soft alignment operation using dot-product self-attention . To address noise and occlusion in 3D point clouds, Bobkov et al.  fed handcrafted point pair function based 4D rotation invariant descriptors into a 4D convolutional neural network. Prokudin et al. 
first randomly sampled a basis point set with a uniform distribution from a unit ball, and then encoded point cloud as minimal distances to the basis point set, which converts the point cloud to a vector with a relatively small fixed length. The encoded representation can then be processed with existing machine learning methods. RCNet utilizes standard RNN and 2D CNN to construct a permutation-invariant network for 3D point cloud processing. The point cloud is first partitioned into parallel beams and sorted along a specific dimension, and each beam is then fed into a shared RNN. The learned features are further fed into an efficient 2D CNN for hierarchical feature aggregation. To enhance its description ability, RCNet-E is proposed to ensemble multiple RCNets along different partition and sorting directions. Point2Sequences  is another RNN-based model that captures correlations between different areas in local regions of point clouds. It considers features learned from a local region at multiple scales as sequences and feeds these sequences from all local regions into an RNN-based encoder-decoder structure to aggregate local region features. Qin et al.  proposed an end-to-end unsupervised domain adaptation-based network PointDAN for 3D point cloud representation. To capture the semantic properties of a point cloud, a self-supervised method is proposed to reconstruct the point cloud, whose parts have been randomly rearranged .
Several methods have also been proposed to learn from both 3D point clouds and 2D images. In PVNet 
, high-level global features extracted from multi-view images are projected into the subspace of point clouds through an embedding network, and fused with point cloud features through a soft attention mask. Finally, a residual connection is employed for fused features and multi-view features to perform shape recognition. Later, PVRNet is further proposed to exploit the relation between a 3D point cloud and its multiple views, which are learned by a relation score module. Based on the relation scores, the original 2D global view features are enhanced for point-single-view fusion and point-multi-view fusion.
The ModelNet10/40 datasets are the most frequently used datasets for shape classification. Table I shows the results achieved by different point-based networks. Several observations can be drawn:
Pointwise MLP networks are usually served as basic building blocks for other types of networks to learn pointwise features.
As a standard deep learning architecture, convolution-based networks can achieve superior performance on irregular 3D point clouds. More attention should be paid to both discrete and continuous convolution networks for irregular data.
Due to its inherent strong capability to handle irregular data, graph-based networks have attracted increasingly more attention in recent years. However, it is still challenging to extend graph-based networks in the spectral domain to various graph structures.
Most networks need to down-sample a point cloud into a fixed small size. This sampling process discards details of the shape. Developing networks that can deal with large-scale point clouds is still in its infancy .
In this section, we will review existing methods for 3D object detection, 3D object tracking and 3D scene flow estimation.
The task of 3D object detection is to accurately locate all objects of interest in a given scene. Similar to object detection in images , 3D object detection methods can be divided into two categories: region proposal-based methods and single shot methods. Several milestone methods are presented in Fig. 6.
These methods first propose several possible regions (also called proposals) containing objects, and then extract region-wise features to determine the category label of each proposal. According to their object proposal generation approach, these methods can further be divided into three categories: multi-view based, segmentation-based and frustum-based methods.
Multi-view Methods. These methods fuse proposal-wise features from different view maps (e.g., LiDAR front view, bird’s eye view (BEV) and image) to obtain 3D rotated boxes, as shown in Fig. 7(a). The computational cost of these methods is usually high.
Chen et al.  generated a group of highly accurate 3D candidate boxes from the BEV map and projected them to the feature maps of multiple views (e.g., LiDAR front view image, RGB image). They then combined these region-wise features obtained from different views to predict oriented 3D bounding boxes, as shown in Fig. 7(a). Although this method achieves a recall of 99.1% at an Intersection-over-Union (IoU) of 0.25 with only 300 proposals, its speed is too slow for practical applications. Subsequently, several approaches have been developed to improve multi-view 3D object detection methods from two aspects.
First, several methods have been proposed to efficiently fuse the information of different modalities. To generate 3D proposals with a high recall for small objects, Ku et al.  proposed a multi-modal fusion-based region proposal network. They first extracted equal-sized features from both BEV and image views using cropping and resizing operations, and then fused these features using element-wise mean pooling. Liang et al.  exploited continuous convolutions to enable effective fusion of image and 3D LiDAR feature maps at different resolutions. Specifically, they extracted nearest corresponding image features for each point in the BEV space and then used bilinear interpolation to obtain a dense BEV feature map by projecting image features into the BEV plane. Experimental results show that dense BEV feature maps are more suitable for 3D object detection than discrete image feature maps and sparse LiDAR feature maps. Liang et al.  presented a multi-task multi-sensor 3D object detection network for end-to-end training. Specifically, multiple tasks (e.g., 2D object detection, ground estimation and depth completion) are exploited to help the network learn better feature representations. The learned cross-modality representation is further exploited to produce highly accurate object detection results. Experimental results show that this method achieves a significant improvement on 2D, 3D and BEV detection tasks, and outperforms previous state-of-the-art methods on the TOR4D benchmark [184, 108].
Second, different methods have been investigated to extract robust representations of the input data. Lu et al.  explored multi-scale contextual information by introducing a Spatial Channel Attention (SCA) module, which captures the global and multi-scale context of a scene and highlights useful features. They also proposed an Extension Spatial Unsample (ESU) module to obtain high-level features with rich spatial information by combining multi-scale low-level features, thus generating reliable 3D object proposals. Although better detection performance can be achieved, the aforementioned multi-view methods take a long runtime since they perform feature pooling for each proposal. Subsequently, Zeng et al.  used a pre-RoI pooling convolution to improve the efficiency of . Specifically, they moved the majority of convolution operations to be ahead of the RoI pooling module. Therefore, RoI convolutions are performed once for all object proposals. Experimental results show that this method can run at a speed of 11.1 fps, which is 5 times faster than MV3D .
Segmentation-based Methods. These methods first leverage existing semantic segmentation techniques to remove most background points, and then generate a large amount of high-quality proposals on foreground points to save computation, as shown in Fig. 7(b). Compared to multi-view methods [19, 73, 200], these methods achieve higher object recall rates and are more suitable for complicated scenes with highly occluded and crowded objects.
Yang et al.  used a 2D segmentation network to predict foreground pixels and projected them into point clouds to remove most background points. They then generate proposals on the predicted foreground points and designed a new criterion named PointsIoU to reduce the redundancy and ambiguity of proposals. Following , Shi et al.  proposed a PointRCNN framework. Specifically, they directly segmented 3D point clouds to obtain foreground points and then fused semantic features and local spatial features to produce high-quality 3D boxes. Following the RPN stage of , Jesus et al.  proposed a pioneering work to leverage Graph Convolution Network (GCN) for 3D object detection. Specifically, two modules are introduced to refine object proposals using graph convolution. The first module R-GCN utilizes all points contained in a proposal to achieve per-proposal feature aggregation. The second module C-GCN fuses per-frame information from all proposals to regress accurate object boxes by exploiting contexts. Sourabh et al.  projected a point cloud into the output of the image-based segmentation network and appended the semantic prediction scores to the points. The painted points are fed into existing detectors [141, 213, 79] to achieve significant performance improvement. Yang et al.  associated each point with a spherical anchor. The semantic score of each point is then used to remove redundant anchors. Consequently, this method achieves a higher recall with lower computational cost as compared to previous methods [189, 141]. In addition, a PointsPool layer is proposed to learn compact features for interior points in proposals and a parallel IoU branch is introduced to improve localization accuracy and detection performance. Experimental results show that this method significantly outperforms other methods [141, 169, 89] on the hard set (Car Class) of the KITTI dataset  and runs at a speed of 12.5 fps.
Frustum-based Methods. These methods first leverage existing 2D object detectors to generate 2D candidate regions of objects and then extract a 3D frustum proposal for each 2D candidate region, as shown in Fig. 7 (c). Although these methods can efficiently propose possible locations of 3D objects, the step-by-step pipeline makes their performance limited by 2D image detectors.
F-PointNets  is a pioneering work in this direction. It generates a frustum proposal for each 2D region and applies PointNet  (or PointNet++ ) to learn point cloud features of each 3D frustum for amodal 3D box estimation. In a follow-up work, Zhao et al.  proposed a Point-SENet module to predict a set of scaling factors, which were further used to adaptively highlight useful features and suppress informative-less features. They also integrated the PointSIFT  module into the network to capture orientation information of point clouds, which achieved strong robustness to shape scaling. This method achieves significant improvement on both indoor and outdoor datasets [44, 148] as compared to F-PointNets .
Xu et al.  leveraged both 2D image region and its corresponding frustum points to accurately regress 3D boxes. To fuse image features and global features of point clouds, they presented a global fusion network for direct regression of box corner locations. They also proposed a dense fusion network for the prediction of point-wise offsets to each corner. Shin et al.  first estimated 2D bounding boxes and 3D poses of objects from a 2D image, and then extracted multiple geometrically feasible object candidates. These 3D candidates were fed into a box regression network to predict accurate 3D object boxes. Wang et al.  generated a sequence of frustums along the frustum axis for each 2D region and applied PointNet  to extract features for each frustum. The frustum-level features were reformed to generate a 2D feature map, which was then fed into a fully convolutional network for 3D box estimation. This method achieves the state-of-the-art performance among 2D image-based methods and was ranked in the top position of the official KITTI leaderboard. Lehner et al.  first obtained a preliminary detection results on the BEV map, and then extracted small point subsets (also called patches) based on the BEV predictions. A local refinement network is applied to learn the local features of patches to predict highly accurate 3D bounding boxes.
Other Methods. Motivated by the success of axis-aligned IoU in object detection in images, Zhou et al.  integrated the IoU of two 3D rotated bounding boxes into several state-of-the-art detectors [182, 79, 141] to achieve consistent performance improvement. Chen et al.  proposed a two-stage network architecture to use both point cloud and voxel representations. First, point clouds are voxelized and fed to a 3D backbone network to produce initial detection results. Second, the interior point features of initial predictions are further exploited for box refinements. Although this design is conceptually simple, it achieves comparable performance to PointRCNN  while maintaining a speed of 16.7 fps.
Inspired by Hough voting-based 2D object detectors, Qi et al.  proposed VoteNet to directly vote for virtual center points of objects from point clouds and to generate a group of high-quality 3D object proposals by aggregating vote features. VoteNet significantly outperforms previous approaches using only geometric information, and achieves the state-of-the-art performance on two large indoor benchmarks (i.e., ScanNet  and SUN RGB-D ). However, the prediction of virtual center point is unstable for a partially occluded object. Further, Feng et al.  added an auxiliary branch of direction vectors to improve the prediction accuracy of virtual center points and 3D candidate boxes. In addition, a 3D object-object relationship graph between proposals is built to emphasize useful features for accurate object detection. Inspired by the observation that the ground truth boxes of 3D objects provide accurate locations of intra-object parts, Shi et al.  proposed the Net, which is composed of a part-aware stage and a part-aggregation stage. The part-aware stage applies a UNet-like network with sparse convolution and sparse deconvolution to learn point-wise features for the prediction and coarse generation of intra-object part locations. The part-aggregation stage adopts RoI-aware pooling to aggregate predicted part locations for box scoring and location refinement.
These methods directly predict class probabilities and regress 3D bounding boxes of objects using a single-stage network. These methods do not need region proposal generation and post-processing. As a result, they can run at a high speed and are highly suitable for real-time applications. According to the type of input data, single shot methods can be divided into two categories: BEV-based and point cloud-based methods.
BEV-based Methods. These methods mainly take BEV representation as their input. Yang et al.  discretized the point cloud of a scene with equally spaced cells and encoded the reflectance in a similar way, resulting in a regular representation. A Fully Convolution Network (FCN) network was then applied to estimate the locations and heading angles of objects. This method outperforms most single shot methods (including VeloFCN , 3D-FCN  and Vote3Deep ) while running at 28.6 fps. Later, Yang et al.  exploited the geometric and semantic prior information provided by High-Definition (HD) maps to improve the robustness and detection performance of 
. Specifically, they obtained the coordinates of ground points from the HD map and then replaced the absolute distance in the BEV representation with the distance relative to the ground to remedy the translation variance caused by the slope of the road. In addition, they concatenated a binary road mask with the BEV representation along the channel dimension to focus on moving objects. Since HD maps were not available everywhere, they also proposed an online map prediction module to estimate the map priors from single LiDAR point cloud. This map-aware method significantly outperforms its baseline on the TOR4D[184, 108] and KITTI  datasets. However, its generalization performance to point clouds with different densities is poor. To solve this problem, Beltrán et al.  proposed a normalization map to consider the differences among different LiDAR sensors. The normalization map is a 2D grid with the same resolution as the BEV map, and it encodes the maximum number of points contained in each cell. It is shown that this normalization map significantly improves the generalization ability of BEV-based detectors.
|MV3D ||L & I||2.8||74.97||63.63||54.00||-||-||-||-||-||-|
|AVOD ||L & I||12.5||76.39||66.47||60.23||36.10||27.86||25.76||57.19||42.08||38.29|
|ContFuse ||L & I||16.7||83.68||68.78||61.67||-||-||-||-||-||-|
|MMF ||L & I||12.5||88.40||77.43||70.22||-||-||-||-||-||-|
|SCANet ||L & I||11.1||79.22||67.13||60.65||-||-||-||-||-||-|
|RT3D ||L & I||11.1||23.74||19.14||18.86||-||-||-||-||-||-|
|IPOD ||L & I||5.0||80.30||73.04||68.73||55.07||44.37||40.05||71.99||52.23||46.50|
|PointPainting ||L & I||2.5||82.11||71.70||67.08||50.32||40.97||37.87||77.63||63.78||55.89|
|F-PointNets ||L & I||5.9||82.19||69.79||60.59||50.53||42.15||38.08||72.27||56.12||49.01|
|SIFRNet ||L & I||-||-||-||-||-||-||-||-||-||-|
|PointFusion ||L & I||-||77.92||63.00||53.27||33.36||28.04||23.38||49.34||29.42||26.98|
|RoarNet ||L & I||10.0||83.71||73.04||59.16||-||-||-||-||-||-|
|F-ConvNet ||L & I||2.1||87.36||76.39||66.69||52.16||43.38||38.80||81.98||65.07||56.54|
|3D IoU loss ||L||12.5||86.16||76.50||71.39||-||-||-||-||-||-|
|Feng et al. ||L||-||-||-||-||-||-||-||-||-||-|
|3D FCN ||L||<0.2||-||-||-||-||-||-||-||-||-|
|MVX-Net ||L & I||16.7||84.99||71.95||64.88||-||-||-||-||-||-|
|LaserNet++ ||L & I||26.3||-||-||-||-||-||-||-||-||-|
|MV3D ||L & I||2.8||86.62||78.93||69.80||-||-||-||-||-||-|
|AVOD ||L & I||12.5||89.75||84.95||78.32||42.58||33.57||30.14||64.11||48.15||42.37|
|ContFuse ||L & I||16.7||94.07||85.35||75.88||-||-||-||-||-||-|
|MMF ||L & I||12.5||93.67||88.21||81.99||-||-||-||-||-||-|
|SCANet ||L & I||11.1||90.33||82.85||76.06||-||-||-||-||-||-|
|RT3D ||L & I||11.1||56.44||44.00||42.34||-||-||-||-||-||-|
|IPOD ||L & I||5.0||89.64||84.62||79.96||60.88||49.79||45.43||78.19||59.40||51.38|
|PointPainting ||L & I||2.5||92.45||88.11||83.36||58.70||49.93||46.29||83.91||71.54||62.97|
|F-PointNets ||L & I||5.9||91.17||84.67||74.77||57.13||49.57||45.48||77.26||61.37||53.78|
|SIFRNet ||L & I||-||-||-||-||-||-||-||-||-||-|
|PointFusion ||L & I||-||-||-||-||-||-||-||-||-||-|
|RoarNet ||L & I||10.0||88.20||79.41||70.02||-||-||-||-||-||-|
|F-ConvNet ||L & I||2.1||91.51||85.84||76.11||57.04||48.96||44.33||84.16||68.88||60.05|
|3D IoU loss ||L||12.5||91.36||86.22||81.20||-||-||-||-||-||-|
|Feng et al. ||L||-||-||-||-||-||-||-||-||-||-|
|3D FCN ||L||<0.2||70.62||61.67||55.61||-||-||-||-||-||-|
|MVX-Net ||L & I||16.7||92.13||86.05||78.68||-||-||-||-||-||-|
|LaserNet++ ||L & I||26.3||-||-||-||-||-||-||-||-||-|
Point Cloud-based Methods. These methods convert a point cloud into a regular representation (e.g., 2D map), and then apply CNN to predict both categories and 3D boxes of objects.
Li et al.  proposed the first method to use a FCN for 3D object detection. They converted a point cloud into a 2D point map and used a 2D FCN to predict the bounding boxes and confidences of objects. Later, they  discretized the point cloud into a 4D tensor with dimensions of length, width, height and channels, and extended the 2D FCN-based detection technologies to 3D domain for 3D object detection. Compared to , 3D FCN-based method  obtains a gain of over >20% in accuracy, but inevitably costs more computing resources due to 3D convolutions and the sparsity of the data. To address the sparsity problem of voxels, Engelcke et al.  leveraged a feature-centric voting scheme to generate a set of votes for each non-empty voxel and to obtain the convolutional results by accumulating the votes. Its computational complexity method is proportional to the number of occupied voxels. Li et al.  constructed a 3D backbone network by stacking multiple sparse 3D CNNs. This method is designed to save memory and accelerate computation by fully using the sparsity of voxels. This 3D backbone network extracts rich 3D features for object detection without introducing heavy computational burden.
Zhou et al.  presented a voxel-based end-to-end trainable framework VoxelNet. They partitioned a point cloud into equally spaced voxels and encoded the features within each voxel into a 4D tensor. A region proposal network is then connected to produce detection results. Although its performance is strong, this method is very slow due to the sparsity of voxels and 3D convolutions. Later, Yan et al.  used the sparse convolutional network  to improve the inference efficiency of . They also proposed a sine-error angle loss to solve the ambiguity between orientations of 0 and . Sindagi et al.  extended VoxelNet by fusing image and point cloud features at early stages. Specifically, they projected non-empty voxels generated by  into the image and used a pre-trained network to extract image features for each projected voxel. These image features were then concatenated with voxel features to produce accurate 3D boxes. This method can effectively exploit multi-modal information to reduce false positives and negatives compared to [213, 182]. Lang et al.  proposed a 3D object detector named PointPillars. This method leverages PointNet  to learn the feature of point clouds organized in vertical columns (Pillars) and encodes the learned features as a pesudo image. A 2D object detection pipeline is then applied to predict 3D bounding boxes. PointPillars outperforms most fusion approaches (including MV3D , RoarNet  and AVOD ) in terms of Average Precision (AP). Moreover, PointPillars can run at a speed of 62 fps on both the 3D and BEV KITTI  benchmarks, making it highly suitable for practical applications.
Other Methods. Meyer et al.  proposed an efficient 3D object detector called LaserNet. This method predicts a probability distribution over bounding boxes for each point and then combines these per-point distributions to generate final 3D object boxes. Further, the dense range view (RV) representation of point cloud is used as input and a fast mean-shift algorithm is proposed to reduce the noise produced by per-point prediction. LaserNet achieves the state-of-the-art performance at the range of 0 to 50 meters, and its runtime is significantly lower than existing methods. Meyer et al.  then extended LaserNet to exploit the dense texture provided by RGB images (e.g., 50 to 70 meters). Specifically, they associated LiDAR points with image pixels by projecting 3D point clouds onto 2D images and exploited this association to fuse RGB information into 3D points. They also considered 3D semantic segmentation as an auxiliary task to learn better representations. This method achieves a significant improvement in both long-range (e.g., 50 to 70 meters) object detection and semantic segmentation while maintaining high efficiency of LaserNet .
Given the locations of an object in the first frame, the task of object tracking is to estimate its state in subsequent frames [56, 97]. Since 3D object tracking can use the rich geometric information in the point clouds, it is expected to overcome several of the drawbacks that are faced by 2D image-based tracking, including occlusion, illumination and scale variation.
proposed a 3D Siamese network with shape completion regularization. Specifically, they first generated candidates using a Kalman filter, and encoded model and candidates into a compact representation using shape regularization. The cosine similarity is then used to search for the location of the tracked object in the next frame. This method can be used as an alternative for object tracking, and significantly outperforms most 2D object tracking methods, including and SiamFC . To efficiently search for the target object, Zarzar et al.  leveraged a 2D Siamese network to generate a large number of coarse object candidates on BEV representation. They then refined the candidates by exploiting the cosine similarity in 3D Siamese network. This method significantly outperforms  in terms of both precision (i.e., by 18%) and success rate (i.e., by 12%). Simon et al. 
proposed a 3D object detection and tracking architecture for semantic point clouds. They first generated voxelized semantic point clouds by fusing 2D visual semantic information, and then utilized the temporal information to improve accuracy and robustness of multi-target tracking. In addition, they introduced a powerful and simplified evaluation metric (i.e., Scale-Rotation-Translation score (SRFs)) to speed up training and inference. Their proposed Complexer-YOLO achieves promising tracking performance and can still run in real-time.
Analogous to optical flow estimation in 2D vision, several methods have started to learn useful information (e.g. 3D scene flow, spatial-temporary information) from a sequence of point clouds.
Liu et al.  proposed FlowNet3D to directly learn scene flows from a pair of consecutive point clouds. FlowNet3D learns both point-level features and motion features through a flow embedding layer. However, there are two problems with FlowNet3D. First, some predicted motion vectors differ significantly from the ground truth in their directions. Second, it is difficult to apply FlowNet to non-static scenes, especially for the scenes which are dominated by deformable objects. To solve this problem, Wang et al.  introduced a cosine distance loss to minimize the angle between the predictions and the ground truth. In addition, they also proposed a point-to-plane distance loss to improve the accuracy for both rigid and dynamic scenes. Experimental results show that these two loss terms improve the accuracy of FlowNet3D from 57.85% to 63.43%, and speed up and stabilize the training process. Gu et al.  proposed a Hierarchical Permutohedral Lattice FlowNet (HPLFlowNet) to directly estimate scene flow from large-scale point clouds. Several bilateral convolution layers are proposed to restore structural information from raw point clouds, while reducing the computational cost.
To effectively process sequential point clouds, Fan and Yang  proposed PointRNN, PointGRU and PointLSTM networks and a sequence-to-sequence model to track moving points. PointRNN, PointGRU, and PointLSTM are able to capture the spatial-temporary information and model dynamic point clouds. Similarly, Liu et al.  proposed MeteorNet to directly learn a representation from dynamic point clouds. This method learns to aggregate information from spatiotemporal neighboring points. Direct grouping and chained-flow grouping are further introduced to determine the temporal neighbors. However, the performance of the aforementioned methods is limited by the scale of datasets. Mittal et al. 
proposed two self-supervised losses to train their network on large unlabeled datasets. Their main idea is that a robust scene flow estimation method should be effective in both forward and backward predictions. Due to the unavailability of scene flow annotation, the nearest neighbor of the predicted transformed point is considered as pesudo ground truth. However, the true ground truth may not be the same as the nearest point. To avoid this problem, they computed the scene flow in the reverse direction and proposed a cycle consistency loss to translate the point to the original position. Experimental results show that this self-supervised method exceeds the state-of-the-art performance of supervised learning-based methods.
The KITTI  benchmark is one of the most influential datasets in autonomous driving and has been commonly used in both academia and industry. Tables II and III present the results achieved by different detectors on the KITTI test 3D and BEV benchmarks, respectively. The following observations can be made:
Region proposal-based methods are the most frequently investigated methods among these two categories, and outperform single shot methods by a large margin on both KITTI test 3D and BEV benchmarks.
There are two limitations for existing 3D object detectors. First, the long-range detection capability of existing methods is relatively poor. Second, how to fully exploit the texture information in images is still an open problem.
Multi-task learning is a future direction in 3D object detection. For example, MMF  learns a cross-modality representation to achieve state-of-the-art detection performance by incorporating multiple tasks.
3D object tracking and scene flow estimation are emerging research topics, and have gradually attracted increasing attention since 2019.
3D point cloud segmentation requires the understanding of both the global geometric structure and the fine-grained details of each point. According to the segmentation granularity, 3D point cloud segmentation methods can be classified into three categories:semantic segmentation (scene level), instance segmentation (object level) and part segmentation (part level).
Given a point cloud, the goal of semantic segmentation is to separate a point cloud into several subsets according to their semantic meanings. Similar to the taxonomy for 3D shape classification (see Section 2), there are two paradigms for semantic segmentation, i.e., projection-based and point-based methods. We show several representative methods in Fig. 8.
Intermediate regular representations can be organised or classfied into multi-view representation [80, 12], spherical representation [172, 173, 116], volumetric representation [113, 135, 46], permutohedral lattice representation [150, 137], and hybrid representation [27, 64], as shown in Fig. 9.
Multi-view Representation. Felix et al.  first projected a 3D point cloud onto 2D planes from multiple virtual camera views. Then, a multi-stream FCN is used to predict pixel-wise scores on synthetic images. The final semantic label of each point is obtained by fusing the re-projected scores over different views. Similarly, Boulch et al.  first generated several RGB and depth snapshots of a point cloud using multiple camera positions. They then performed pixel-wise labeling on these snapshots using 2D segmentation networks. The scores predicted from RGB and depth images are further fused using residual correction . Based on the assumption that point clouds are sampled from locally Euclidean surfaces, Tatarchenko et al.  introduced tangent convolutions for dense point cloud segmentation. This method first projects the local surface geometry around each point to a virtual tangent plane. Tangent convolutions are then directly operated on the surface geometry. This method shows great scalability and is able to process large-scale point clouds with millions of points. Overall, the performance of multi-view segmentation methods is sensitive to viewpoint selection and occlusions. Besides, these methods have not fully exploited the underlying geometric and structural information, as the projection step inevitably introduces information loss.
Spherical Representation. To achieve fast and accurate segmentation of 3D point clouds, Wu et al.  proposed an end-to-end network based on SqueezeNet  and Conditional Random Field (CRF). To further improve segmentation accuracy, SqueezeSegV2  is introduced to address domain shift by utilizing an unsupervised domain adaptation pipeline. Milioto et al. 
proposed RangeNet++ for real-time semantic segmentation of LiDAR point clouds. The semantic labels of 2D range images are first transferred to 3D point clouds, an efficient GPU-enabled KNN-based post-processing step is further used to alleviate the problem of discretization errors and blurry inference outputs. Compared to single view projection, spherical projection retains more information and is suitable for the labeling of LiDAR point clouds. However, this intermediate representation inevitably brings several problems such as discretization errors and occlusions.
Volumetric Representation. Huang et al.  first divided a point cloud into a set of occupancy voxels. They then fed these intermediate data to a fully-3D convolutional neural network for voxel-wise segmentation. Finally, all points within a voxel are assigned the same semantic label as the voxel. The performance of this method is severely limited by the granularity of the voxels and the boundary artifacts caused by the point cloud partition. Further, Tchapmi et al.  proposed SEGCloud to achieve fine-grained and global consistent semantic segmentation. This method introduces a deterministic trilinear interpolation to map the coarse voxel predictions generated by 3D-FCNN  back to the point cloud, and then uses Fully Connected CRF (FC-CRF) to enforce spatial consistency of these inferred per-point labels. Meng et al.  introduced a kernel-based interpolated variational autoencoder architecture to encode the local geometrical structures within each voxel. Instead of a binary occupancy representation, RBFs are employed for each voxel to obtain a continuous representation and capture the distribution of points in each voxel. VAE is further used to map the point distribution within each voxel to a compact latent space. Then, both symmetry groups and an equivalence CNN are used to achieve robust feature learning.
Good scalability is one of the remarkable advantages of volumetric representation. Specifically, volumetric-based networks are free to be trained and tested in point clouds with different spatial sizes. In Fully-Convolutional Point Network (FCPN) , different levels of geometric relations are first hierarchically abstracted from point clouds, 3D convolutions and weighted average pooling are then used to extract features and incorporate long-range dependencies. This method can process large-scale point clouds and has good scalability during inference. Angela et al.  proposed ScanComplete to achieve 3D scan completion and per-voxel semantic labeling. This method leverages the scalability of fully-convolutional neural networks and can adapt to different input data sizes during training and test. A coarse-to-fine strategy is used to hierarchically improve the resolution of the predicted results.
Volumetric representation is naturally sparse, as the number of non-zero values only accounts for a small percentage. Therefore, it is inefficient to apply dense convolution neural networks on the spatially-sparse data. To this end, Graham et al.  proposed submanifold sparse convolutional networks. This method significantly reduces memory and computational costs by restricting the output of convolution to be only related to occupied voxels. Meanwhile, its sparse convolution can also control the sparsity of the extracted features. This submanifold sparse convolution is suitable for efficient processing of high-dimensional and spatially-sparse data. Further, Choy et al. 
proposed a 4D spatio-temporal convolutional neural network called MinkowskiNet for 3D video perception. A generalized sparse convolution is proposed to effectively process high-dimensional data. A trilateral-stationary conditional random field is further applied to enforce consistency.
Overall, the volumetric representation naturally preserves the neighborhood structure of 3D point clouds. Its regular data format also allows direct application of standard 3D convolutions. These factors lead to a stead performance improvement in this area. However, the voxelization step inherently introduces discretization artifacts and information loss. Usually, a high resolution leads to high memory and computational costs, while a low resolution introduces loss of details. It is non-trivial to select an appropriate grid resolution in practice.
Permutohedral Lattice Representation. Su et al.  proposed the Sparse Lattice Networks (SPLATNet) based on Bilateral Convolution Layers (BCLs). This method first interpolates a raw point cloud to a permutohedral sparse lattice, BCL is then applied to convolve on occupied parts of the sparsely populated lattice. The filtered output is then interpolated back to the raw point cloud. In addition, this method allows flexible joint processing of multi-view images and point clouds. Further, Rosu et al.  proposed LatticeNet to achieve efficient processing of large point clouds. A data-dependent interpolation module called DeformsSlice is also introduced to back project the lattice feature to point clouds.
Hybrid Representation. To further leverage all available information, several methods have been proposed to learn multi-modal features from 3D scans. Angela and Matthias  present a joint 3D-multi-view network to combine RGB features and geometric features. A 3D CNN stream and several 2D streams are used to extract features, and a differentiable back-projection layer is proposed to jointly fuse the learned 2D embeddings and 3D geometric features. Further, Hung et al.  proposed a unified point-based framework to learn 2D textural appearance, 3D structures and global context features from point clouds. This method directly applies point-based networks to extracts local geometric features and global context from sparsely sampled point sets without any voxelization. Jaritz et al.  proposed Multi-view PointNet (MVPNet) to aggregate appearance features from 2D multi-view images and spatial geometric features in the canonical point cloud space.
Point-based networks directly work on irregular point clouds. However, point clouds are orderless and unstructured, making it infeasible to directly apply standard CNNs. To this end, the pioneering work PointNet  is proposed to learn per-point features using shared MLPs and global features using symmetrical pooling functions. Based on PointNet, a series of point-based networks have been proposed recently. Overall, these methods can be roughly divided into pointwise MLP methods, point convolution methods, RNN-based methods, and graph-based methods.
Pointwise MLP Methods. These methods usually use shared MLP as the basic unit in their network for its high efficiency. However, point-wise features extracted by shared MLP cannot capture the local geometry in point clouds and the mutual interactions between points . To capture wider context for each point and learn richer local structures, several dedicated networks have been introduced, including methods based on neighboring feature pooling, attention-based aggregation, and local-global feature concatenation.
Neighboring feature pooling: To capture local geometric patterns, these methods learn a feature for each point by aggregating the information from local neighboring points. In particular, PointNet++  groups points hierarchically and progressively learns from larger local regions, as illustrated in Fig. 10. Multi-scale grouping and multi-resolution grouping are also proposed to overcome the problems caused by non-uniformity and varying density of point clouds. Later, Jiang et al.  proposed a PointSIFT module to achieve orientation encoding and scale awareness. This module stacks and encodes the information from eight spatial orientations through a three-stage ordered convolution operation. Multi-scale features are extracted and concatenated to achieve adaptivity to different scales. Different from the grouping techniques used in PointNet++ (i.e., ball query), Francis et al.  utilized -means clustering and KNN to separately define two neighborhoods in the world space and learned feature space. Based on the assumption that points from the same class are expected to be closer in feature space, a pairwise distance loss and a centroid loss are introduced to further regularize feature learning. To model the mutual interactions between different points, Zhao et al.  proposed PointWeb to explore the relations between all pairs of points in a local region by densely constructing a locally fully-linked web. An Adaptive Feature Adjustment (AFA) module is proposed to achieve information interchange and feature refinement. This aggregation operation helps the network to learn a discriminative feature representation. Zhang et al.  proposed a permutation invariant convolution called Shellconv based on the statistics from concentric spherical shells. This method first queries a set of multi-scale concentric spheres, the max-pooling operation is then used within different shells to summarize the statistics, MLPs and 1D convolution are used to obtain the final convolution output. Hu et al.  proposed an efficient and lightweight network called RandLA-Net for large-scale point cloud processing. This network utilizes random point sampling to achieve a remarkable efficiency in terms of memory and computation. A local feature aggregation module is further proposed to capture and preserve the geometric features.
proposed a group shuffle attention to model the relations between points, and presented a permutation-invariant, task-agnostic and differentiable Gumbel Subset Sampling (GSS) to replace the widely used Furthest Point Sampling (FPS) approach. This module is less sensitive to outliers and can select a representative subset of points. To better capture the spatial distribution of a point cloud, Chen et al. proposed a Local Spatial Aware (LSA) layer to learn spatial awareness weights based on the spatial layouts and the local structures of point clouds. Similar to CRF, Zhao et al.  proposed an Attention-based Score Refinement (ASR) module to post-process the segmentation results produced by the network. The initial segmentation result is refined by pooling the scores of neighbouring points with learned attention weights. This module can be easily integrated into existing deep networks to improve the final segmentation performance.
Local-global concatenation: Zhao et al.  proposed a permutation-invariant PS-Net to incorporate local structures and global context from point clouds. Edgeconv  and NetVLAD  are repeatedly stacked to capture the local information and scene-level global features.
Point Convolution Methods. These methods tend to propose effective convolution operations for point clouds. Hua et al.  proposed a point-wise convolution operator, where the neighboring points are binned into kernel cells and then convolved with kernel weights. Wang et al.  proposed a network called PCCN based on parametric continuous convolution layers. The kernel function of this layer is parameterized by MLPs and spans the continuous vector space. Hughes et al.  proposed a Kernel Point Fully Convolutional Network (KP-FCNN) based on Kernel Point Convolution (KPConv). Specifically, the convolution weights of KPConv are determined by the Euclidean distances to kernel points, and the number of kernel points is not fixed. The positions of the kernel points are formulated as an optimization problem of best coverage in a sphere space. Note that the radius neighbourhood is used to keep a consistent receptive field, while grid subsampling is used in each layer to achieve high robustness under varying densities of point clouds. In , Francis et al. provided rich ablation experiments and visualization results to show the impact of receptive field on the performance of aggregation-based methods. They also proposed a Dilated Point Convolution (DPC) operation to aggregate dilated neighbouring features, instead of the K nearest neighbours. This operation is demonstrated to be very effective in increasing the receptive field and can be easily integrated into existing aggregation-based networks.
To capture inherent context features from point clouds, Recurrent Neural Networks (RNN) have also been used for semantic segmentation of point clouds. Based on PointNet, Francis et al.  first transformed a block of points into multi-scale blocks and grid blocks to obtain input-level context. Then, the block-wise features extracted by PointNet are sequentially fed into Consolidation Units (CU) or Recurrent Consolidation Units (RCU) to obtain output-level context. Experimental results show that incorporating spatial context is important for the improvement of the segmentation performance. Huang et al.  proposed a lightweight local dependency modeling module, and utilized a slice pooling layer to convert unordered point feature sets into an ordered sequence of feature vectors. Ye et al.  first proposed a Pointwise Pyramid Pooling (3P) module to capture the coarse-to-fine local structure, and then utilized two-direction hierarchical RNNs to further obtain long-range spatial dependencies. RNN was then applied to achieve an end-to-end learning. However, these methods lose rich geometric features and density distribution from point clouds when aggregating the local neighbourhood features with global structure features . To alleviate the problems caused by the rigid and static pooling operations, Zhao et al.  proposed a Dynamic Aggregation Network (DAR-Net) to consider both global scene complexity and local geometric features. The inter-medium features are dynamically aggregated using a self-adapted receptive field and node weights. Liu et al.  proposed 3DCNN-DQN-RNN for efficient semantic parsing of large-scale point clouds. This network first learns the spatial distribution and color features using a 3D CNN network, DQN is further used to localize the class objects. The final concatenated feature vector is fed into a residual RNN to obtain the final segmentation results.
Graph-based Methods. To capture the underlying shapes and geometric structures of 3D point clouds, several methods resort to graph networks. Loic et al.  represented a point cloud as a set of interconnected simple shapes and superpoints, and used an attributed directed graph (i.e., superpoint graph) to capture the structure and context information. Then, the large-scale point cloud segmentation problem is spilt into three sub-problems, i.e., geometrically homogeneous partition, superpoint embedding, and contextual segmentation. To further improve the partition step, Loic and Mohamed  proposed a supervised framework to oversegment a point cloud into pure superpoints. This problem is formulated as a deep metric learning problem structured by an adjacency graph. In addition, a graph-structured contrastive loss is also proposed to help the recognition of borders between objects.
To better capture the local geometric relationships in high-dimensional space, Kang et al.  proposed a PyramNet based on Graph Embedding Module (GEM) and Pyramid Attention Network (PAN). The GEM module formulates a point cloud as a directed acyclic graph and utilzes a covariance matrix to replace the Euclidean distance for the construction of adjacent similarity matrix. Convolution kernels with four different sizes are used in the PAN module to extract features with different semantic intensities. In , Graph Attention Convolution (GAC) is proposed to selectively learn relevant features from a local neighbouring set. This operation is achieved by dynamically assigning attention weights to different neighbouring points and feature channels based on their spatial positions and feature differences. GAC can learn to capture discriminative features for segmentation, and has similar characteristics to the commonly used CRF model.
Compared to semantic segmentation, instance segmentation is more challenging as it requires more accurate and fine-grained reasoning of points. In particular, it not only needs to distinguish the points with different semantic meanings, but also separate instances with the same semantic meaning. Overall, existing methods can be divided into two groups: proposal-based methods and proposal-free methods. Several milestone methods are illustrated in Fig 11.
These methods convert the instance segmentation problem into two sub-tasks: 3D object detection and instance mask prediction. Hou et al.  proposed a 3D fully-convolutional Semantic Instance Segmentation (3D-SIS) network to achieve semantic instance segmentation on RGB-D scans. This network learns from both color and geometry features. Similar to 3D object detection, a 3D Region Proposal Network (3D-RPN) and a 3D Region of Interesting (3D-RoI) layer are used to predict bounding box locations, object class labels and instance masks. Following the analysis-by-synthesis strategy, Yi et al.  proposed a Generative Shape Proposal Network (GSPN) to generate high-objectness 3D proposals. These proposals are further refined by a Region-based PointNet (R-PointNet). The final label is obtained by predicting a per-point binary mask for each class label. Different from direct regression of 3D bounding boxes from point clouds, this method removes a large amount of meaningless proposals by enforcing geometric understanding. By extending 2D panoptic segmentation to 3D mapping, Gaku et al.  proposed an oneline volumetirc 3D mapping system to jointly achieve large-scale 3D reconstruction, semantic labeling, and instance segmentation. They first utilized 2D semantic and instance segmentation networks to obtain pixel-wise panoptic labels and then integrated these labels to the volumtric map. A fully-connected CRF is further used to achieve accurate segmentation. This semantic mapping system can achieve high-quality semantic mapping and discriminative object recognition. Yang et al.  proposed a single-stage, anchor-free and end-to-end trainable network called 3D-BoNet to achieve instance segmentation on point clouds. This method directly regress rough 3D bounding boxes for all potential instances, and then utilized a point-level binary classifier to obtain instance labels. Particularly, the bounding box generation task is formulated as an optimal assignment problem. In addition. a multi-criteria loss function is also proposed to regularize the generated bounding boxes. This method does not need any post-processing and is computationally efficient. Zhang et al.  proposed a network for instance segmentation of large-scale outdoor LiDAR point clouds. This method learns a feature representation on the bird’s-eye view of point clouds using self-attention blocks. The final instance labels are obtained based on the predicted horizontal center and the height limits.
Overall, proposal-based methods are intuitive and straightforward, and the instance segmentation results usually have good objectness. However, these methods require multi-stage training and pruning of redundant proposals. Therefore, they are usually time-consuming and computationally expensive.
Proposal-free methods [166, 167, 124, 34, 95, 93] do not have an object detection module. Instead, they usually consider instance segmentation as a subsequent clustering step after semantic segmentation. In particular, most existing methods are based on the assumption that points belonging to the same instance should have very similar features. Therefore, these methods mainly focus on discriminative feature learning and point grouping.
In a pioneering work, Wang et al. 
first introduced a Similarity Group Proposal Network (SGPN). This method first learns a feature and semantic map for each point, and then introduces a similarity matrix to represent the similarity between each paired features. To learn more discriminative features, they use a double-hinge loss to mutually adjust the similarity matrix and semantic segmentation results. Finally, a heuristic and non-maximal suppression method is adopted to merge similar points into instances. Since the construction of a similarity matrix requires large memory consumption, the scalability of this method is limited. Similarly, Liu et al. first leveraged submanifold sparse convolution  to predict semantic scores of each voxel and affinity between neighboring voxels. They then introduced a clustering algorithm to group points into instances based on the predicted affinity and the mesh topology. Further, Liang et al.  proposed a structure-aware loss for the learning of discriminative embeddings. This loss considers both the similarity of features and the geometric relations among points. An attention-based graph CNN is further used to adaptively refine the learned features by aggregating different information from neighbors.
Since the semantic category and instance label of a point are usually dependent on each other, several methods have been proposed to couple these two tasks into a single task. Wang et al.  integrated these two tasks by introducing an end-to-end and learnable Associatively Segmenting Instances and Semantics (ASIS) module. Experiments show that semantic features and instance features can mutually support each other to achieve an improved performance through this ASIS module. Similarly, Pham et al.  first introduced a Multi-Task Point-wise Network (MT-PNet) to assign a label to each point and regularized the embeddings in the feature space by introducing a discriminative loss . They then fused the predicted semantic labels and embeddings to a Multi-Value Conditional Random Field (MV-CRF) model for joint optimization. Finally, mean-field variational inference is used to produce semantic labels and instance labels. Hu et al. 
first proposed a Dynamic Region Growing (DRG) method to dynamically separate a point cloud into a set of disjoint patches, and then used an unsupervised K-means++ algorithm to group all these patches. Multi-scale patch segmentation is then performed with the guidance of contextual information between patches. Finally, these labeled patches are merged into object level to obtain final semantic and instance labels.
To achieve instance segmentation on full 3D scenes, Cathrin et al.  presented a hybrid 2D-3D network to jointly learn global consistent instance features from a BEV representation and local geometric features of point clouds. The learned features are then combined to achieve semantic and instance segmentation. Note that, rather than heuristic GroupMerging algorithms , a more flexible Meanshift  algorithm is used to group these points into instances. Alternatively, multi-task learning is also introduced for instance segmentation. Jean et al.  learned both the unique feature embedding of each instance and the directional information to the object’s center. Feature embedding loss and directional loss are proposed to adjust the learned feature embeddings in latent feature space. Mean-shift clustering and non-maximum suppression are adopted to group voxels into instances. This method achieves the state-of-the-art performance on the ScanNet  benchmark. Besides, the predicted directional information is particularly useful to determine the boundary of instances. Zhang et al.  introduced probabilistic embeddings to instance segmentation of point clouds. This method also incorporates uncertainty estimation and proposes a new loss function for the clustering step.
In summary, proposal-free methods do not require computationally expensive region-proposal components. However, the objectness of instance segments grouped by these methods is usually low since these methods do not explicitly detect object boundaries.
The difficulty for part segmentation of 3D shapes are twofold. First, shape parts with the same semantic label have a large geometric variation and ambiguity. Second, the method should be robust to noise and sampling.
VoxSegNet  is proposed to achieve fine-grained part segmentation on 3D voxelized data under a limited solution. A Spatial Dense Extraction (SDE) module (which consists of stacked atrous residual blocks) is proposed to extract multi-scale discriminative features from sparse volumetric data. The learned features are further re-weighted and fused by progressively applying an Attention Feature Aggregation (AFA) module. Evangelos et al.  combined FCNs and surface-based CRFs to achieve end-to-end 3D part segmentation. They first generated images from multiple views to achieve optimal surface coverage and fed these images into a 2D network to produce confidence maps. Then, these confidence maps are aggregated by a surface-based CRF, which is responsible for a consistent labeling of the entire scene. Yi et al.  introduced a Synchronized Spectral CNN (SyncSpecCNN) to perform convolution on irregular and non-isomorphic shape graphs. A spectral parameterization of dilated convolutional kernels and a spectral transformer network is introduced to solve the problem of multi-scale analysis in parts and information sharing across shapes.
Wang et al.  first performed shape segmentation on 3D meshes by introducing Shape Fully Convolutional Networks (SFCN) and taking three low-level geometric features as its input. They then utilized voting-based multi-label graph cuts to further refine the segmentation results. Zhu et al.  proposed a weakly-supervised CoSegNet for 3D shape co-segmentation. This network takes a collection of unsegmented 3D point cloud shapes as input, and produces shape part labels by iteratively minimizing a group consistency loss. Similar to CRF, a pre-trained part-refinement network is proposed to further refine and denoise part proposals. Chen et al.  proposed a Branched AutoEncoder network (BAE-NET) for unsupervised, one-shot and weakly supervised 3D shape co-segmentation. This method formulates the shape co-segmentation task as a representation learning problem and aims at finding the simplest part representations by minimizing the shape reconstruction loss. Based on the encoder-decoder architecture, each branch of this network can learn a compact representation for a specific part shape. The features learned from each branch and the point coordinate are then fed to the decoder to produce a binary value (which indicates whether the point belongs to this part). This method has good generalization ability and can process large 3D shape collections (up to 5000+ shapes). However, it is sensitive to initial parameters and does not incorporate shape semantics into the network, which hinders this method to obtain a robust and stable estimation in each iteration.
Table IV shows the results achieved by existing methods on public benchmark, including S3DIS , Semantic3D , ScanNet , and SemanticKITTI . The following issues need to be further investigated:
Point-based networks are the most frequently investigated methods. However, point representation naturally does not have the explicit neighbouring information, most existing point-based methods have to resort the expensive neighbor searching mechanism (e.g., KNN  or ball query ). This inherently limits the efficiency of these methods, as the neighbor searching mechanism requires both high computational cost and irregular memory access .
Learning from imbalanced data is still a challenging problem in point cloud segmentation. Although several approaches [206, 157, 78] have achieved a remarkable overall performance, their performance on minority classes is still limited. E.g., RandLA-Net  achieves an overall IoU of 76.0% on the reduced-8 subset of Semantic3D, but a very low IOU of 41.1% on the class of hardscape.
The majority of existing approaches [129, 131, 88, 17, 206] work on small point clouds (e.g., 1m1m with 4096 points). In practice, the point clouds acquired by depth sensors are usually immense and large-scale. Therefore, it is desirable to further investigate the problem of efficient segmentation of large-scale point clouds.
This paper has presented a contemporary survey of the state-of-the-art methods for 3D understanding, including 3D shape classification, 3D object detection & tracking, and 3D scene and object segmentation. A comprehensive taxonomy and performance comparison of these methods have been presented. Merits and demerits of various methods are also covered, with potential research directions being listed.
This work was partially supported by the National Natural Science Foundation of China (No. 61972435, 61602499), the Natural Science Foundation of Guangdong Province (2019A1515011271), the Shenzhen Technology and Innovation Committee, the Australian Research Council (Grants DP150100294 and DP150104251), and a China Scholarship Council (CSC) scholarship.
Justlookup: one millisecond deep feature extraction for point clouds by lookup tables. In ICME, pp. 326–331. Cited by: §2.2.1, TABLE I.
3DCNN-DQN-RNN: a deep reinforcement learning framework for semantic parsing of large-scale 3D point clouds. In ICCV, pp. 5678–5687. Cited by: §4.1.2.