The problem of 6 degrees of freedom (6D) object pose estimation is to determine the transformation from a local object coordinate system to a reference coordinate system (e.g., camera or robot coordinate). The transformation is composed of 3D location and 3D orientation. Robust and accurate 6D object pose estimation is of primary importance for many robotic applications such as grasping and dexterous manipulation. The recent success of convolutional neural networks (CNNs) in visual recognition has inspired methods that use deep networks for learning features from RGB images [15, 23, 30]. These learned features are used for inferring 6D object poses. Similarly, CNNs can also be applied to RGB-D images and treat depth information as an additional channel for feature learning [16, 33, 20, 3]. However, in some scenarios, color information may not be available, and depth information is not in the 2-dimensional matrix format (e.g., laser range finder data), which can be easily processed with CNN-based systems. Depth information can also be represented by a point cloud, which is an unordered set of points in a metric 3D space. In existing methods, point clouds are mainly used for pose refinement [26, 27, 32, 14]
or template matching with hand-crafted features extracted from point clouds[6, 12]. Using point clouds in the registration stage confines its usage scope and hand-crafted features are usually less effective compared to deep-learned features.
In this work, we investigate how to accurately estimate the 6D pose of a known object represented by a point cloud segment containing only geometric information using deep networks. Our approach is inspired by PointNet , a deep network for object classification and segmentation operating on point clouds. We adapt the system to the problem of pose estimation. PointNet provides a method to apply deep learning to unordered point sets, and it is a suitable architecture for our purpose. Our 6D pose regression method can be applied to any type of range data, e.g., data from laser range finders.
For developing our system, we investigate three open questions. The first is how to efficiently use depth information in a deep learning-based system. Although it has been shown by many applications that CNNs can extract powerful features from RGB-D information for specific tasks, due to the inherent difference between color and depth information, it is unclear whether this is an efficient way to treat depth information. We argue that a point cloud is a more suitable structure and should be used in the scope of both deep networks and geometry-based optimization.
The second question is whether translation and orientation should be estimated with separate networks or a single network in a supervised learning system. During a supervised learning process, a network learns the mapping from its input to the desired output guided by a loss function. Since the metric units for translation and orientation are different (i.e., meters and radians), we argue that regressing them using separate networks and loss functions is a more suitable choice. Our experiments show that an architecture with separate networks outperforms those with shared layers.
Another question is the choice of rotation representation and the loss function for measuring the distance between two rotations. Quaternions have been a popular choice for many learning-based systems [32, 31]. However, quaternions have the unit-norm constraint which imposes a limit on the network output range. We argue that axis-angle is a more suitable choice because it is a constraint-free representation. Concerning loss functions, L2 loss is a popular choice for measuring the distance between two rotations [31, 17]. We argue that the geodesic distance is a more suitable choice since it is well justified mathematically, and provides a clearer learning goal compared to the L2 loss. We show experimentally that our arguments are valid.
Our contributions are thus as follows:
We present a simple yet effective system that infers the 6D pose of known objects represented by point cloud segments containing geometric information. This system is based on PointNet and exploits the point cloud structure by utilizing it as the input to both deep networks and an iterative closest point (ICP) refinement process. To the best of our knowledge, this is the first deep learning architecture based system that regresses 6D poses from only unordered point sets.
We demonstrate that the proposed method outperforms state-of-the-art methods on a public dataset. Experimental results show that our system outperforms methods that use both color and depth information during pose inference stage. This evaluation result indicates that the proposed system is an efficient way to use depth information for pose estimation.
Ablation studies provide an evaluation of each system component. We show experimentally that each design choice has an impact on system performance.
Ii Related Work
Pose estimation has been well studied in the past using both color [15, 23, 30] and depth data [10, 28]. Since in this work we focus on investigating how to use geometric information during the pose inference stage, we mainly review works that use depth information.
is one of the first works that use hand-crafted features from depth information for pose estimation. It uses surface normals as part of its local patch features. This patch representation is adapted and used with random forest in[29, 28]. Surface normal is also used as an additional modality in . Another way of using depth information is to treat it as an extra image depth channel (RGBD) and feed it into a CNN [16, 1, 33, 20, 3], or random forest [2, 19, 22]
or a fully connected sparse autoencoder for feature extraction. Depth is also used to create point clouds, which are used for generating pose hypotheses with 3D-3D correspondences and ICP refinement in . Point clouds are used to facilitate Point-to-Point matching in [6, 12]. Geometry embeddings are extracted from point cloud segments with a deep network in . Approaches such as  use color information to provide an initial pose estimate, then refine it with ICP using depth information. Point cloud segments are also used for pose estimation in . However, they only report experimental data for pose estimation for a single angle as opposed to all three as we do. Furthermore, they formulate rotation estimation as classification and discretize the rotation angles to bins. It is not clear if this approach would scale up to three rotation angles. In our previous work , we only considered rotation regression for single objects. Here, we regress the full 6D pose in a multi-class setting.
For learning-based systems,  propose to predict translation and rotation with separate networks sequentially.  propose to regress translation and rotation with the same network. Quaternions are used as the rotation representation for regression [32, 31, 3]. Bui et al.  propose to use L2 loss function for rotation learning. Axis-angle representation is also used in . However, they only address estimating 3D rotation from RGB information, while we address both 3D rotation and 3D translation from point clouds. Works that use deep networks for extracting features and do nearest neighbor search pose retrieval or classification into discretized bins to obtain object poses [16, 33, 20, 1, 24] are not in the scope of this work and are not discussed in detail here.
Our work is most similar to DenseFusion  as we both use PointNet  to extract features from point cloud segments. However, there are two significant differences: first, during the pose estimation stage, we only use coordinate information from point clouds while DenseFusion also uses color information extracted by a CNN; second, we design regression targets and loss functions for rotation and translation individually while DenseFusion uses one regression target for the 6D pose.
Iii System Architecture
Figure 1 shows an overview of our system. The proposed system for object 6D pose estimation is a multi-class system, i.e., we use the same system to predict poses for objects from different classes. Hence, an object segment, as well as the corresponding class information, is required as input to the pose estimation networks. As semantic segmentation is a well-studied problem, we assume the object segment and class information is provided by an off-the-shelf method111Here, we use the semantic segmentation from . and focus on the object pose estimation from a point cloud segment in this work. A point cloud segment is created using depth and the target object segment. This segment is processed with Farthest Point Sampling (FPS)  to obtain a down-sampled segment with a consistent surface structure representation. This down-sampled segment and class information are combined as the input for two separate networks for rotation estimation in axis-angle representation and translation prediction through translation residual regression. The 6D pose is refined with a geometry-based optimization process to produce the final pose estimate.
Figure (a)a illustrates BaseNet, which is the basic building block of our system. BaseNet is an adapted version of PointNet . Given a point cloud with points as input, PointNet is invariant to all
possible permutations. Each point is processed independently using multi-layer perceptrons (MLPs) with shared weights. Compared to PointNet, we remove the spatial transformer blocks and adapt the dimension for the output layer to be
. For each input point with class information, a feature vector is learned with shared weights. These feature vectors are max-pooled to create a global representation of the input point cloud. Finally, we use a three-layer regression MLP on top of this global feature to predict the poses.
Figure (b)b shows a more detailed diagram of our system. We use two separate networks to handle translation and rotation estimation. Input to the rotation network is a point cloud with
points concatenated with the one-hot encoded class information. In total, the input is aby array where is the total number of classes. The output of the rotation network is the estimated rotation in axis-angle representation. The translation network takes normalized point coordinates concatenated with class information and estimates the translation residual. The full translation is obtained by adding back the coordinate mean.
Iv Supervised Learning for 6D Pose Regression
This section describes how the loss functions for 6D pose regression are formulated in our supervised learning framework. Given a set of points on the surface of a known object in the camera coordinate, the aim of pose estimation is to find a transformation that transforms from the object coordinates to the camera coordinates. This transformation is composed of a translation and a rotation. A translation consists of the displacements along the three coordinate axes. A rotation specifies the rotation around the three coordinate axes, and it has different representations such as axis-angle, Euler angles, quaternion, and rotation matrix. For supervised learning, suitable loss functions are required to measure the differences between predicted poses and ground truth poses. For rotation learning, we argue that the axis-angle representation is the best suited for the learning task. Geodesic distance is used as the loss function for rotation regression. For translation learning, we predict the residual of translation.
Iv-a Rotation estimation with axis-angle and geodesic distance loss
In the axis-angle representation, a vector represents a rotation of radians around the unit vector . Given an axis-angle representation , the corresponding rotation matrix is obtained via the exponential map
is the identity matrix and
is the skew-symmetric matrix
where and are the two rotation matrices corresponding to and , respectively.
This loss function directly measures the magnitude of rotation difference between and , so it is convenient to interpret. Furthermore, the network can make constraint-free predictions with axis-angle representations, in contrast to e.g. quaternion representations which require normalization.
Iv-B Translation residual estimation
To simplify the learning task by reducing the variance in regression space for translation prediction, the learning target is chosen to be the residual of translation. Given a translation residual, full translation prediction is obtained via
where is the mean of . -norm is used to measure the distance between prediction and ground truth , resulting in the translation loss function :
Iv-C Total loss function for 6D pose regression
The total loss is defined as the combination of the translation and the rotation loss:
where is a scaling factor. The total loss is used for training the pose estimation networks.
We evaluate the proposed system on the YCB-Video dataset  and compare the performance with the state-of-the-art methods PoseCNN  and DenseFusion . We also compare the performance on a subset of the object classes with a state-of-the-art RGB-based method DOPE . Besides prediction accuracy and performance under occlusions, we also investigate the impact of using different network structures, as well as the influence of different rotation representations. The implementation of our system is available on-line222https://github.com/GeeeG/CloudPose.
V-a Experiment setup
The YCB video dataset  contains 92 video sequences with total 133,827 frames of 21 objects selected from the YCB object set  with 6D pose annotations. We follow the official train/test split and use 80 video sequences for training. Testing is performed on the key frames chosen from the remaining 12 sequences . 80,000 frames of synthetic data are also provided by YCB-Video dataset as an extension to the training set. During training, Adam optimizer is used with a learning rate of . The batch size is . For the total loss, we use , which is given by the ratio between the expected error of translation and rotation at the end of the training . The number of points of the input point cloud segment is
. Batch normalization is applied to all layers. No dropout is used. All of our networks are trained forepochs. For refinement, we use the Point-to-Point ICP registration provided by Open3D  and refine for 10 iterations. The initial search radius is m and is reduced by after each iteration. For a fair comparison, all methods use object segmentation provided by PoseCNN during testing.
V-B Evaluation metrics
We use the average distance (AD) of model points and the average distance for a rotationally symmetric object (ADS) proposed in 
as evaluation metrics. Given a 3D model represented as a setwith points, ground truth rotation and translation , as well as estimated rotation and translation , the AD is defined as:
ADS is computed using closest point distance. It provides a distance measure that considers possible pose ambiguities caused by rotational symmetry:
A 6D pose estimate is considered to be correct if AD and ADS are smaller than a given threshold. We report the area under error threshold-accuracy curve (AUC) for AD and ADS. The maximum thresholds for both curves are set to m. Furthermore, we also provide ADS accuracy with threshold m (1cm) to illustrate the performance accuracy under a smaller error tolerance.
V-C Prediction accuracy
Evaluation results averaged for all 21 objects in the YCB-Video dataset are shown in Table I. PoseCNN  uses RGB information to provide an initial pose estimate (PC w/o ICP), then uses depth information with a highly customized ICP for pose refinement (PC). DenseFusion  (DF) uses both color and point cloud features extracted by deep networks to give per-pixel pose estimate for final pose voting, and iterative pose refinement is performed with an extra network module. Ours w/o ICP is the estimated pose from the proposed system architecture (Section III), and Ours is the result after ICP refinement. We also perform the ICP refinement on DF results (DF+ICP). For the overall performance in Table I, we highlight the best performance in bold font. Details regarding the data type used by pose regression networks and the post process are also presented.
Our method achieves state-of-the-art performance using only depth information. In terms of AD, we outperform both PC an DF. We observe that DF+ICP shows small improvement compared to DF. One possible reason is the sensitivity of ICP to the initial pose guess, if the method already performs well without refinement, ICP is able to provide further gains. If the initial guess is poor, ICP can even make the results worse. This result indicates that features learned from depth information represented by unordered point clouds are sufficient for accurately regressing 6D pose. Furthermore, this also shows that the proposed approach is an efficient way to use depth information in a deep learning framework for pose regression.
Performance for individual objects is shown in Table II. We use the trained network for six objects provided by the authors of DOPE  and report the results. The AD results are not available because the object coordinate frames used in the YCB object dataset , YCB video dataset for PoseCNN  and DOPE are different. As our method uses the frames from , and the transformation between  and  is not publicly available, we can not find the correspondence between model points required for AD. We also applied ICP to DOPE pose estimates, but the performance was not improved. A possible reason is the sensitivity of ICP to the initial pose estimate.
Some qualitative results are shown in Figure 17. Pose estimates from PC, DF and our method are used for projecting object models onto 2D images. More qualitative results are available in the supplementary video.
|PC w/o ICP ||✓||51.5||75.6||26.1|
|DF  + ICP333we apply ICP to DF results after iterative refinement.||✓||✓||✓||76.3||94.7||89.0|
|Ours w/o ICP||✓||76.0||91.3||80.9|
|DOPE ||PC ||DF ||Ours|
For a given target object in a frame, the occlusion factor of the object is defined as 
where is the number of pixels in the 2D ground truth segmentation, and is the number of pixels in the projection of the 3D object model onto the image plane using the camera intrinsic parameters and the ground truth 6D pose, when we assume the object would be fully visible. The occlusion factor of the YCB-Video dataset ranges from to . We divide this range into bins with a bin width of and report the prediction accuracy (ADS) with a threshold of cm. Figure 18 illustrates the results. It can be observed that our method (Ours) has competitive performance when the occlusion is lower than , then both ours and PC start to suffer as the amount of occlusion increases. One possible reason is that DF outputs per-pixel prediction with confidence scores, while ours and PC provide only one pose prediction. This per-pixel prediction may have helped to provide better performance when the amount of occlusion is higher than .
V-E Network architecture: ablation study
To investigate whether translation and rotation should be regressed with the same or separate networks, we compare the performance of different architectures. We alter the network architecture by incrementally sharing the layers between translation and rotation networks. Table III shows the result in terms of AD, ADS, and accuracies for translation and rotation under certain thresholds. None denotes the proposed architecture which regresses translation and rotation with two separate networks. The numbers in the first column denote the number of shared layers between translation and rotation BaseNet (Figure 4). We compare performance without ICP refinement. When sharing layers, the performance is worse than using two separate networks.
We also tested an architecture that shares all the layers while having the same amount of parameters as the proposed structure with a doubled layer width. The performance is similar to the architecture with the single width, and this verifies that the performance deterioration is not caused by insufficient network capacity. This result verifies that using separated networks for translation and rotation is a more suitable design choice.
|shared layers||AD||ADS||rot_err<10°||tran_err<1 cm|
V-F Rotation representation and loss function: ablation study
We investigate the impact of different rotation representations and loss functions. For comparing quaternion to axis-angle, we adapted our rotation network to have -dimensional output instead of . The output is normalized and then converted to the axis-angle representation. We use the same loss function as described in Section IV. For comparing L2 loss with Geodesic distance, we keep the rotation representation in axis-angle format and apply different loss functions. Table IV shows the accuracy of rotation prediction with different thresholds. With the same loss function, using axis-angle yields a better result than quaternion. This indicates that axis-angle is a better choice for rotation learning. With the same rotation representation, L2 loss slightly underperforms geodesic loss. Since geodesic distance also has a better mathematical justification, this makes it a better choice.
V-G Time performance
We measure the time performance on a Nvidia Titan X GPU. The system is implemented with Tensorflow. Pose estimation by a forward pass through our network takes 0.11 seconds for a single object. The 10 iterations of ICP refinement require an additional 0.3 seconds.
We propose a system for fast and accurate 6D pose estimation of known objects. We formulate the problem as a supervised learning problem and use two separate networks for rotation and translation regression, and use point clouds as input for the regression. We use axis-angle as rotation representation and geodesic distance as the loss function for rotation regression. Ablation studies show that these design choices outperform the commonly used quaternion representation and L2 loss. Experimental results show that the proposed system outperforms two state-of-the-art methods on a public benchmark.
To the best of our knowledge, this is the first deep learning system that regresses 6D object poses from only depth information represented by unordered point clouds. Features extracted from point clouds with deep networks can be used for accurately regressing object pose. Our pose regression system can be applied to range data from other sensors such as laser range finders. In the future work, we will investigate aspects such as pose estimation for rotational symmetry objects using only geometric information.
-  (2017) Pose guided RGBD feature learning for 3D object pose estimation. In ICCV, Cited by: §II, §II.
-  (2014) Learning 6D object pose estimation using 3D object coordinates. In ECCV, Cited by: §II.
-  (2018) When regression meets manifold learning for object recognition and pose estimation. In ICRA, Cited by: §I, §II, §II.
-  (2015) Benchmarking in manipulation research using the Yale-CMU-Berkeley object and model set. Robotics & Automation Magazine, IEEE 22 (3), pp. 36–52. Cited by: §V-A, §V-C.
-  (2016) Recovering 6D object pose and predicting next-best-view in the crowd. In CVPR, Cited by: §II.
-  (2000) Model globally, match locally: efficient and robust 3D object recognition. In CVPR, Cited by: §I, §II.
-  (1997) The farthest point strategy for progressive image sampling. IEEE Transactions on Image Processing 6 (9), pp. 1305–1315. Cited by: §III.
-  (2018) Occlusion resistant object rotation regression from point cloud segments. In ECCV 4th International Workshop on Recovering 6D Object Pose, Cited by: §II, §V-D.
International Journal of Computer Vision103 (3), pp. 267–305. Cited by: §IV-A, §IV-A.
-  (2011) Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes. In ICCV, Cited by: §II, §II.
-  (2012) Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes. In ACCV, Cited by: §II, §V-B.
-  (2016) Going further with point pair features. In ECCV, Cited by: §I, §II.
-  (2009) Metrics for 3D rotations: comparison and analysis. Journal of Mathematical Imaging and Vision 35 (2), pp. 155–164. Cited by: §IV-A.
-  (2018) iPose: instance-aware 6D pose estimation of partly occluded objects. In ACCV, Cited by: §I, §II.
-  (2017) SSD-6D: making RGB-based 3D detection and 6D pose estimation great again. In ICCV, Cited by: §I, §II.
-  (2016) Deep learning of local RGB-D pacthes for 3D object detection and 6D pose estimation. In ECCV, Cited by: §I, §II, §II.
-  (2017) Geometric loss functions for camera pose regression with deep learning. In CVPR, Cited by: §I.
-  (2015) PoseNet: a convolutional network for real-time 6-DOF camera relocalization. In ICCV, Cited by: §V-A.
-  (2015) Learning analysis-by-synthesis for 6D pose estimation in RGB-D images. In ICCV, Cited by: §I, §II.
-  (2018) A unified framework for multi-view multi-class object pose estimation. In ECCV, Cited by: §I, §II, §II.
-  (2017) 3D pose regression using convolutional neural networks. In CVPR workshops, Cited by: §II.
-  (2017) Global hypothesis generation for 6D object pose estimation. In CVPR, Cited by: §II.
-  (2018) Making deep heatmaps robust to partial occlusions for 3D object pose estimation. In ECCV, Cited by: §I, §II.
-  (2018) Frustum PointNets for 3D object detection from RGB-D data. In CVPR, Cited by: §II, §II.
-  (2017) PointNet: deep learning on point sets for 3D classification and segmentation. In CVPR, Cited by: §I, §II, §III.
-  (2009) Fast point feature histograms (FPFH) for 3D registration. In ICRA, Cited by: §I.
-  (2017) A learning-based variable size part extraction architecture for 6D object pose recovery in depth images. Image and Vision Computing 63 (C), pp. 38–50. Cited by: §I.
-  (2018) Latent-class hough forests for 6 DoF object pose estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence 40 (1), pp. 119–132. Cited by: §II, §II.
-  (2014) Latent-class hough forests for 3D object detection and pose estimation. In ECCV, Cited by: §II.
-  (2018) Deep object pose estimation for semantic robotic grasping of household objects. In CoRL, Cited by: §I, §II, §V-C, TABLE II, §V.
-  (2019) DenseFusion: 6D object pose estimation by iterative dense fusion. In CVPR, Cited by: §I, §II, §II, §II, Fig. 17, Fig. 18, §V-C, TABLE I, TABLE II, §V.
-  (2018) PoseCNN: a convolutional neural network for 6D object pose estimation in cluttered scenes. In RSS, Cited by: §I, §I, §II, §II, Fig. 17, Fig. 18, §V-A, §V-C, §V-C, TABLE I, TABLE II, §V, footnote 1.
-  (2017) 3D object instance recognition and pose estimation using triplet loss with dynamic margin. In IROS, Cited by: §I, §II, §II.
-  (2018) Open3D: a modern library for 3D data processing. arXiv:1801.09847. Cited by: §V-A.