Unsupervised Learning of Fine Structure Generation for 3D Point Clouds by 2D Projection Matching

08/08/2021
by   Chen Chao, et al.
0

Learning to generate 3D point clouds without 3D supervision is an important but challenging problem. Current solutions leverage various differentiable renderers to project the generated 3D point clouds onto a 2D image plane, and train deep neural networks using the per-pixel difference with 2D ground truth images. However, these solutions are still struggling to fully recover fine structures of 3D shapes, such as thin tubes or planes. To resolve this issue, we propose an unsupervised approach for 3D point cloud generation with fine structures. Specifically, we cast 3D point cloud learning as a 2D projection matching problem. Rather than using entire 2D silhouette images as a regular pixel supervision, we introduce structure adaptive sampling to randomly sample 2D points within the silhouettes as an irregular point supervision, which alleviates the consistency issue of sampling from different view angles. Our method pushes the neural network to generate a 3D point cloud whose 2D projections match the irregular point supervision from different view angles. Our 2D projection matching approach enables the neural network to learn more accurate structure information than using the per-pixel difference, especially for fine and thin 3D structures. Our method can recover fine 3D structures from 2D silhouette images at different resolutions, and is robust to different sampling methods and point number in irregular point supervision. Our method outperforms others under widely used benchmarks. Our code, data and models are available at https://github.com/chenchao15/2D_projection_matching.

READ FULL TEXT
research
08/01/2021

SSPU-Net: Self-Supervised Point Cloud Upsampling via Differentiable Rendering

Point clouds obtained from 3D sensors are usually sparse. Existing metho...
research
12/01/2022

Leveraging Single-View Images for Unsupervised 3D Point Cloud Completion

Point clouds captured by scanning devices are often incomplete due to oc...
research
12/08/2019

SampleNet: Differentiable Point Cloud Sampling

There is a growing number of tasks that work directly on point clouds. A...
research
07/12/2020

DRWR: A Differentiable Renderer without Rendering for Unsupervised 3D Structure Learning from Silhouette Images

Differentiable renderers have been used successfully for unsupervised 3D...
research
12/05/2020

ParaNet: Deep Regular Representation for 3D Point Clouds

Although convolutional neural networks have achieved remarkable success ...
research
11/05/2019

A GMM based algorithm to generate point-cloud and its application to neuroimaging

Recent years have witnessed the emergence of 3D medical imaging techniqu...
research
08/22/2023

Learning a More Continuous Zero Level Set in Unsigned Distance Fields through Level Set Projection

Latest methods represent shapes with open surfaces using unsigned distan...

Please sign up or login with your details

Forgot password? Click here to reset