DeepAI
Log In Sign Up

AUTO3D: Novel view synthesis through unsupervisely learned variational viewpoint and global 3D representation

07/13/2020
by   Xiaofeng Liu, et al.
0

This paper targets on learning-based novel view synthesis from a single or limited 2D images without the pose supervision. In the viewer-centered coordinates, we construct an end-to-end trainable conditional variational framework to disentangle the unsupervisely learned relative-pose/rotation and implicit global 3D representation (shape, texture and the origin of viewer-centered coordinates, etc.). The global appearance of the 3D object is given by several appearance-describing images taken from any number of viewpoints. Our spatial correlation module extracts a global 3D representation from the appearance-describing images in a permutation invariant manner. Our system can achieve implicitly 3D understanding without explicitly 3D reconstruction. With an unsupervisely learned viewer-centered relative-pose/rotation code, the decoder can hallucinate the novel view continuously by sampling the relative-pose in a prior distribution. In various applications, we demonstrate that our model can achieve comparable or even better results than pose/3D model-supervised learning-based novel view synthesis (NVS) methods with any number of input views.

READ FULL TEXT

page 9

page 11

10/29/2021

Novel View Synthesis from a Single Image via Unsupervised learning

View synthesis aims to generate novel views from one or more given sourc...
07/30/2020

Unsupervised Continuous Object Representation Networks for Novel View Synthesis

Novel View Synthesis (NVS) is concerned with the generation of novel vie...
09/08/2022

im2nerf: Image to Neural Radiance Field in the Wild

We propose im2nerf, a learning framework that predicts a continuous neur...
04/17/2018

Pixels, voxels, and views: A study of shape representations for single view 3D object shape prediction

The goal of this paper is to compare surface-based and volumetric 3D obj...
04/10/2018

Learning Pose Specific Representations by Predicting Different Views

The labeled data required to learn pose estimation for articulated objec...
11/17/2020

A Divide et Impera Approach for 3D Shape Reconstruction from Multiple Views

Estimating the 3D shape of an object from a single or multiple images ha...
05/30/2022

The Devil is in the Pose: Ambiguity-free 3D Rotation-invariant Learning via Pose-aware Convolution

Rotation-invariant (RI) 3D deep learning methods suffer performance degr...