From Image Collections to Point Clouds with Self-supervised Shape and Pose Networks

05/05/2020
by   K L Navaneet, et al.
18

Reconstructing 3D models from 2D images is one of the fundamental problems in computer vision. In this work, we propose a deep learning technique for 3D object reconstruction from a single image. Contrary to recent works that either use 3D supervision or multi-view supervision, we use only single view images with no pose information during training as well. This makes our approach more practical requiring only an image collection of an object category and the corresponding silhouettes. We learn both 3D point cloud reconstruction and pose estimation networks in a self-supervised manner, making use of differentiable point cloud renderer to train with 2D supervision. A key novelty of the proposed technique is to impose 3D geometric reasoning into predicted 3D point clouds by rotating them with randomly sampled poses and then enforcing cycle consistency on both 3D reconstructions and poses. In addition, using single-view supervision allows us to do test-time optimization on a given test image. Experiments on the synthetic ShapeNet and real-world Pix3D datasets demonstrate that our approach, despite using less supervision, can achieve competitive performance compared to pose-supervised and multi-view supervised approaches.

READ FULL TEXT

page 7

page 13

page 15

research
08/01/2021

SSPU-Net: Self-Supervised Point Cloud Upsampling via Differentiable Rendering

Point clouds obtained from 3D sensors are usually sparse. Existing metho...
research
01/11/2018

Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction

We present a framework for learning single-view shape and pose predictio...
research
05/15/2023

AutoRecon: Automated 3D Object Discovery and Reconstruction

A fully automated object reconstruction pipeline is crucial for digital ...
research
10/22/2018

Unsupervised Learning of Shape and Pose with Differentiable Point Clouds

We address the problem of learning accurate 3D shape and camera pose fro...
research
11/24/2022

SfM-TTR: Using Structure from Motion for Test-Time Refinement of Single-View Depth Networks

Estimating a dense depth map from a single view is geometrically ill-pos...
research
01/14/2020

Improving Semantic Analysis on Point Clouds via Auxiliary Supervision of Local Geometric Priors

Existing deep learning algorithms for point cloud analysis mainly concer...
research
11/28/2018

CAPNet: Continuous Approximation Projection For 3D Point Cloud Reconstruction Using 2D Supervision

Knowledge of 3D properties of objects is a necessity in order to build e...

Please sign up or login with your details

Forgot password? Click here to reset