Log In Sign Up

AUTO3D: Novel view synthesis through unsupervisely learned variational viewpoint and global 3D representation

by   Xiaofeng Liu, et al.

This paper targets on learning-based novel view synthesis from a single or limited 2D images without the pose supervision. In the viewer-centered coordinates, we construct an end-to-end trainable conditional variational framework to disentangle the unsupervisely learned relative-pose/rotation and implicit global 3D representation (shape, texture and the origin of viewer-centered coordinates, etc.). The global appearance of the 3D object is given by several appearance-describing images taken from any number of viewpoints. Our spatial correlation module extracts a global 3D representation from the appearance-describing images in a permutation invariant manner. Our system can achieve implicitly 3D understanding without explicitly 3D reconstruction. With an unsupervisely learned viewer-centered relative-pose/rotation code, the decoder can hallucinate the novel view continuously by sampling the relative-pose in a prior distribution. In various applications, we demonstrate that our model can achieve comparable or even better results than pose/3D model-supervised learning-based novel view synthesis (NVS) methods with any number of input views.


page 9

page 11


Novel View Synthesis from a Single Image via Unsupervised learning

View synthesis aims to generate novel views from one or more given sourc...

Unsupervised Continuous Object Representation Networks for Novel View Synthesis

Novel View Synthesis (NVS) is concerned with the generation of novel vie...

im2nerf: Image to Neural Radiance Field in the Wild

We propose im2nerf, a learning framework that predicts a continuous neur...

Pixels, voxels, and views: A study of shape representations for single view 3D object shape prediction

The goal of this paper is to compare surface-based and volumetric 3D obj...

Learning Pose Specific Representations by Predicting Different Views

The labeled data required to learn pose estimation for articulated objec...

A Divide et Impera Approach for 3D Shape Reconstruction from Multiple Views

Estimating the 3D shape of an object from a single or multiple images ha...

The Devil is in the Pose: Ambiguity-free 3D Rotation-invariant Learning via Pose-aware Convolution

Rotation-invariant (RI) 3D deep learning methods suffer performance degr...