A point cloud is a set of 3D points that can be used to represent a 3D surface. Each point is usually associated with one single color along with other attributes. The point cloud can be used in many applications involving the rendering of 3D objects 
such as 3D immersive telepresence and 6 degree-of-freedom virtual reality. However, the point cloud with only one single color is essentially not realistic since the colors of the real world objects may vary significantly along with the change of the view angles. Recently, 8i captures several plenoptic point clouds with different colors in different view angles. These point clouds are more realistic but also much larger, and thus bring more burdens to the communication and storage. For example, the or attributes in the plenoptic point clouds make the attributes of the point clouds or times larger. Therefore, there is an urgent need to compress them efficiently.
To transmit or store a point cloud, we need to signal the geometry or position information as well as attribute information. Octree and its variations  are typical methods to compress the geometry. Some methods also introduce plane  or mapping  to compress the geometry more efficiently. However, as the main difference between the plenoptic point cloud and the general point cloud with one single color is the attribute information, we put more focuses on the review of the point cloud attribute compression in this paper.
The first group of works focusing on attribute compression is the transform-based method. Zhang et al. 
first proposed using Graph Fourier Transform (GFT) to exploit the correlations among the already encoded geometry information. However, deriving the transform kernel requires solving a very complex eigenproblem. Therefore, Queiroz and Chou introduced the Region-based Adaptive Hierarchical Transform (RAHT) to obtain a better balance between the complexity and the performance. This work is adopted in the geometry-based point cloud compression (G-PCC) standard  and is the recommended algorithm to compress the static dense point cloud. The second group of works focusing on attribute compression is the mapping-based method. As a representative, Mammou et al.  proposed a video-based point cloud compression (V-PCC) method to project the point cloud to 2-D videos patch by patch and compress them using high efficiency video coding (HEVC) . Due to the high efficiency of the 2-D video compression standard, the coding efficiency of the V-PCC is very high. It is shown in one recent work that the static dense point cloud can be also coded efficiently using V-PCC . The third group of works is the prediction-based method mainly designed for the sparse point cloud attribute. Mammou et al.  introduced a layer-based prediction to predict the point cloud from its coarse representation. Kathariya et al.  proposed using kd-tree to divide the point cloud into various layers to further improve the compression performance.
In terms of the compression of the plenoptic point cloud attributes, Sandri et al.  extended the RAHT to further exploit the correlations among multiple attributes. They used Discrete Cosine Transform (DCT) or Kahunen-Loeve Transform (KLT) to utilize the correlations among various attributes. The RAHT-DCT or RAHT-KLT brings much better compression performance compared with the RAHT. However, using only transform is unable to fully utilize the correlations among various views. We believe a video-based solution with prediction, transform, quantization, and entropy coding is a better way to exploit the correlations.
Therefore, in this paper, we first extend the V-PCC to support the plenoptic point cloud by generating multiple attribute videos. Under the current V-PCC framework, these attribute videos are encoded independently. Then based on the observation that these videos from multiple views have very high correlations, we propose encoding them using multiview high efficiency video coding (MV-HEVC)  to utilize the correlations among various attributes. Furthermore, as the unoccupied pixels will have no influences on the reconstructed quality of the plenoptic point cloud, we propose a block-based group padding method that unifies the unoccupied attribute pixels from different views to reduce their bit cost.
The rest of this paper is organized as follows. In Section 2, we will introduce the proposed video-based plenoptic point cloud compression framework. The block-based padding for the unoccupied pixels will be introduced in Section 3. Section 4 will describe the experimental results in detail. Section 5 will conclude the paper.
2 Video-based plenoptic point cloud compression framework
Under the V-PCC, a point cloud is first divided into several patches by projecting to its bounding box. Each patch is generated by clustering the neighboring points with similar normals together. In this way, the generated patches will have fewer variances in the geometry and attributes, and thus can be coded efficiently. After the patches are generated, the V-PCC uses a simple packing strategy to organize the patches into frames. The patch location is determined through an exhaustive search in a raster scan order. In addition, patch rotation is supported to allow more flexible packing to improve compression performance. After packing, the padding process aims to fill any empty space between the patches to make the generated frames more suitable for video coding. The geometry and attribute videos will finally go through HEVC to generate the bitstream. Note that each static point cloud is projected to two frames to handle the occlusion so as to achieve a better balance between the number of projected points and the compression performance.
To use the V-PCC to compress the plenoptic point cloud, we follow the above processes to generate the geometry and attribute videos. The main difference is that there will be multiple attribute videos generated since each point has multiple attributes. The multiple attribute videos will then be encoded using HEVC to be compressed efficiently. Under this method, we will show the performance of the video-based plenoptic point cloud compression framework without using the correlations among various views.
Fig. 1 gives a typical example of the attribute frames generated from multiple view angles. We can see that the generated attribute frames from various view angles are very similar despite some pixel differences. Therefore, we propose using MV-HEVC to exploit the correlations from multiple views to compress them more efficiently. As we have mentioned, each static point cloud can be projected to two frames. Therefore, using a -view static point cloud as an example, we utilize the multiview encoding structure as shown in Fig. 2.
In Fig. 2, the indices from to indicate the index of each view. The sub-indices and are the two frames projected from one point cloud. The arrows among various squares in the figure indicate the reference relationships among various views and frames. As shown in Fig. 2, we organize a hierarchical coding structure with group of pictures (GOP) size in the view direction. In this way, all the views are divided into hierarchical levels. All the views are coded as B frames using bi-directional prediction to exploit the correlations among various views. In the temporal domain, as the two frames projected from one static point cloud has much larger correlations than that among various views, the second frame will only reference the first frame with the same view index under the proposed reference structure.
In addition to the reference relationship, the quantization parameters (QPs) of various views and frames are also important to the compression performance. Currently, we set the QPs according to the following basic rule. First, the higher the hierarchical level is, the larger the QP is. In the view direction, we set the QPs of each level as . Second, the second non-reference frame uses a higher QP than the first reference frame. In the temporal direction, we set the QP of as . The detailed setting of the QPs of all the frames and views are shown in Table 1. In this work, the proposed coding structure is used to compress static plenoptic point clouds. This coding structure can be easily applied to dynamic plenoptic point clouds by propagating the coding structure in the temporal domain.
3 Block-based group padding for unoccupied pixels
In the V-PCC, the unoccupied pixels are padded in a way that they will cost bits as few as possible since they have no influences on the reconstructed point cloud quality. Similarly, in the plenoptic point cloud, we also need to minimize the bit cost of the unoccupied pixels. In the V-PCC, several padding methods   are proposed to minimize the bit cost across the temporal direction. Especially, all the unoccupied pixels are padded using the average of the first frame and the second frame to minimize the bit cost of the unoccupied pixels in the second frame. However, since the pixel differences between the pixels in various views are much larger than that in the temporal domain, this method is unable to deal with the unoccupied pixels in the view directions.
Fig. 3 shows a typical example of the projected attribute with the unoccupied pixels set as black. We can see that some large continuous unoccupied pixels exist as indicated by the green rectangle. Those pixels can be padded properly to increase the correlations across various views to save some bits. However, there are also some isolated unoccupied pixels as indicated by the small red square. If we pad those pixels across the view direction, the blocks including both occupied and unoccupied pixels will become less continuous in the spatial domain. This will make those blocks unable to find the corresponding block in the reference frame and lead to a serious bitrate increase.
Therefore, in this paper, we propose a block-based group padding for unoccupied pixels. For each pixel, we first find a block with the pixel as the center pixel. Only when all the pixels in the block are unoccupied, we will pad the pixel. In this way, the isolated unoccupied pixels will not be padded and we can still keep the spatial continuity for the blocks including both occupied and unoccupied pixels. For the large continuous unoccupied pixels, we can reduce the prediction residue so as to compress them more efficiently. In our current implementation, the block size is set to .
After the detection of the continuous unoccupied pixels, we will pad them using the average value of all the views in both frame and frame ,
where is number of views for the plenoptic point cloud, is the frame index, is the view index, and is the value to be padded for each position. After the above padding scheme, in both the view direction and temporal direction, we can obtain a very good prediction for the continuous unoccupied pixels, and thus the bitrate of the unoccupied pixels can be significantly reduced.
|Name||Points||Cameras||Geometry bit depth||Attribute bit depth|
|Color bits||Luma PSNR||Color bits||Luma PSNR||BD-rate|
4 Experimental results
The proposed algorithms are implemented in the V-PCC reference software TMC-7.0  to compare with the state-of-the-art method RAHT-KLT . We test all the static plenoptic point clouds defined in . The characteristics of all the tested static plenoptic point clouds are shown in Table 2. When verifying the performance of the proposed extension of V-PCC, we follow the V-PCC common test condition  to test various bitrates from r1 (low bitrate) to r5 (high bitrate). We use the all intra configuration to generate the results since we have only one static plenoptic point cloud to be compressed. When verifying the performance of the proposed multiview-based solution, we use the same QP settings for the I frames as the V-PCC-based method. The QPs of the other views and frames are set according to Table 1
. Note that the average peak signal to noise ratio (PSNR) of all the views is used as the quality metric for the attribute.
We first give a performance comparison between our multiview solution with the proposed padding method and the state-of-the-art method RAHT-KLT as shown in Table 3. We can see that the proposed multiview solution can lead to performance improvements on average compared with the state-of-the-art method. Through better utilizing the correlations among various views, the proposed method leads to a better compression performance compared with RAHT-KLT. The experimental results demonstrate the effectiveness of the proposed algorithm. In addition, we can see that the proposed algorithm always leads to a better R-D performance in low bitrate case. While in high bitrate case, the proposed algorithm brings similar or even slightly worse R-D performance.
After an overall comparison with the state-of-the-art method, we then verify the performance of the proposed methods one by one. Table 4 gives a comparison between the proposed multi-view solution compared with the V-PCC solution without using the correlations among multiple views. We can see that through utilizing the correlations among various views, we can achieve , , and bitrate savings on average for the Luma, Cb, and Cr components with respect to the attribute bits, respectively. Although the proposed algorithm mainly targets the attribute compression, the proposed algorithm can reduce the total bits so as to bring performance improvements for both the geometry and attribute with respect to the total bits. We can achieve and bitrate savings on average for the geometry under D1 and D2 measurements. In addition, we can achieve an average of , , and performance improvements for the Luma, Cb, and Cr components, respectively. The experimental results demonstrate that the proposed multiview compression framework can substantially exploit the correlations among various views so as to improve the performance.
Table 5 shows the performance of the proposed block-padding method under our proposed multiview-based solution. We can see that the proposed block-based padding can provide an average of , , and performance improvements with respect to only the attribute bits for the Luma, Cb, and Cr components, respectively. With respect to the total bitrate, the proposed algorithm can bring over bitrate savings on average for both the geometry and attribute. The experimental results demonstrate that through making the unoccupied pixel with higher correlations, the block-based padding algorithm can bring obvious bitrate saving for the unoccupied pixels so as to improve the overall performance.
|(a) Loot R-D curve||(b) Longdress R-D curve|
Two examples of the R-D curves for all the above-mentioned methods are shown in Fig. 4. We can see that the V-PCC without considering the view correlations shows the worst performance. The multiview-video solution shows a better performance than the RAHT-KLT, especially in the low bitrate case. The proposed group padding algorithm can further improve the performance by saving the bit cost of the unoccupied pixels.
5 Conclusion and future work
In this paper, we propose video-based plenoptic point cloud compression methods to compress the plenoptic point cloud more efficiently. We first extend the current video-based point cloud compression (V-PCC) to support the plenoptic point cloud by generating multiple attribute videos. Then based on the observation that these videos from multiple views have very high correlations, we propose encoding them using multiview high efficiency video coding. We further propose a block-based group padding method that unifies the unoccupied attribute pixels from different views to reduce their bit cost. The proposed algorithms are implemented in the V-PCC reference software. The experimental results show that the proposed algorithms can bring significant bitrate savings compared with the state-of-the-art method for plenoptic point cloud compression.
-  C. Tulvan, R. Mekuria, Z. Li, and S. Laserre, “Use Cases for Point Cloud Compression (PCC),” ISO/IEC JTC1/SC29/WG11 MPEG2015/N16331, Geneva, CH, Jun. 2016.
-  M. Krivokuća, P. A. Chou, and P. Savill, “8i Voxelized Surface Light Field (8iVSLF) Dataset,” ISO/IEC JTC1/SC29/WG11 m42914, Ljubljana, Jul. 2018.
-  R. Schnabel and R. Klein, “Octree-based Point-Cloud Compression,” IEEE VGTC conference on Point-Based Graphics, vol. 6, pp. 111–120, 2006.
-  B. Kathariya, L. Li, Z. Li, J. Alvarez, and J. Chen, “Scalable Point Cloud Geometry Coding with Binary Tree Embedded Quadtree,” in 2018 IEEE International Conference on Multimedia and Expo (ICME), 2018, pp. 1–6.
-  L. He, W. Zhu, and Y. Xu, “Best-Effort Projection based Attribute Compression for 3D Point Cloud,” in 2017 23rd Asia-Pacific Conference on Communications (APCC), Dec. 2017, pp. 1–6.
-  C. Zhang, D. Florêncio, and C. Loop, “Point Cloud Attribute Compression with Graph Transform,” in 2014 IEEE International Conference on Image Processing (ICIP), Oct. 2014, pp. 2066–2070.
-  R. L. de Queiroz and P. A. Chou, “Compression of 3D Point Clouds Using a Region-Adaptive Hierarchical Transform,” IEEE Transactions on Image Processing, vol. 25, no. 8, pp. 3947–3956, Aug. 2016.
-  S. Schwarz, M. Preda, V. Baroncini, M. Budagavi, P. Cesar, P. A. Chou, R. A. Cohen, M. Krivokuća, S. Lasserre, Z. Li, J. Llach, K. Mammou, R. Mekuria, O. Nakagami, E. Siahaan, A. Tabatabai, A. M. Tourapis, and V. Zakharchenko, “Emerging MPEG Standards for Point Cloud Compression,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 9, no. 1, pp. 133–148, Mar. 2019.
-  G. J. Sullivan, J. Ohm, W. Han, and T. Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Standard,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, no. 12, pp. 1649–1668, Dec. 2012.
-  M. Gonçalves, L. Agostini, D. Palomino, M. Porto, and G. Correa, “Encoding efficiency and computational cost assessment of state-of-the-art point cloud codecs,” in 2019 IEEE International Conference on Image Processing (ICIP), Sep. 2019, pp. 3726–3730.
-  B. Kathariya, V. Zakharchenko, Z. Li, and J. Chen, “Level-of-Detail Generation Using Binary-Tree for Lifting Scheme in LiDAR Point Cloud Attributes Coding,” in 2019 Data Compression Conference (DCC), March 2019, pp. 580–580.
-  G. Sandri, R. L. de Queiroz, and P. A. Chou, “Compression of plenoptic point clouds,” IEEE Transactions on Image Processing, vol. 28, no. 3, pp. 1419–1427, March 2019.
-  M. M. Hannuksela, Y. Yan, X. Huang, and H. Li, “Overview of the multiview high efficiency video coding (mv-hevc) standard,” in 2015 IEEE International Conference on Image Processing (ICIP), Sep. 2015, pp. 2154–2158.
-  K. Mammou, J. Kim, V. Valentin, F. Robinet, A. Tourapis, and Y. Su, “CE2.12 Related: Sparse Linear Model Based Padding Method for the Texture Images,” Document ISO/IEC JTC1/SC29/WG11 m44837, Macau, CH, Oct. 2018.
-  S. Rhyu, Y. Oh, and J. Woo, “PCC CE2.13 Report on Texture and Depth Padding Improvement,” Document ISO/IEC JTC1/SC29/WG11 m43667, Ljubljana, SI, Jul. 2018.
-  Video-based point cloud compression test model, TMC-7.0. [Online]. Available: http://mpegx.int-evry.fr/software/MPEG/PCC/mpeg-pcc-dmetric.git
-  3DG, “Common test conditions for point cloud compression,” Document ISO/IEC JTC1/SC29/WG11 N18665, Gothenburg, SE, Jul. 2019.