Fully Understanding Generic Objects: Modeling, Segmentation, and Reconstruction

by   Feng Liu, et al.

Inferring 3D structure of a generic object from a 2D image is a long-standing objective of computer vision. Conventional approaches either learn completely from CAD-generated synthetic data, which have difficulty in inference from real images, or generate 2.5D depth image via intrinsic decomposition, which is limited compared to the full 3D reconstruction. One fundamental challenge lies in how to leverage numerous real 2D images without any 3D ground truth. To address this issue, we take an alternative approach with semi-supervised learning. That is, for a 2D image of a generic object, we decompose it into latent representations of category, shape and albedo, lighting and camera projection matrix, decode the representations to segmented 3D shape and albedo respectively, and fuse these components to render an image well approximating the input image. Using a category-adaptive 3D joint occupancy field (JOF), we show that the complete shape and albedo modeling enables us to leverage real 2D images in both modeling and model fitting. The effectiveness of our approach is demonstrated through superior 3D reconstruction from a single image, being either synthetic or real, and shape segmentation.



page 1

page 3

page 7

page 8


Voxel-based 3D Detection and Reconstruction of Multiple Objects from a Single Image

Inferring 3D locations and shapes of multiple objects from a single 2D i...

MarrNet: 3D Shape Reconstruction via 2.5D Sketches

3D object reconstruction from a single image is a highly under-determine...

ShaRF: Shape-conditioned Radiance Fields from a Single View

We present a method for estimating neural scenes representations of obje...

Self-Supervised Intrinsic Image Decomposition

Intrinsic decomposition from a single image is a highly challenging task...

3D Shape Reconstruction from a Single 2D Image via 2D-3D Self-Consistency

Aiming at inferring 3D shapes from 2D images, 3D shape reconstruction ha...

De-rendering 3D Objects in the Wild

With increasing focus on augmented and virtual reality applications (XR)...

3D Interpreter Networks for Viewer-Centered Wireframe Modeling

Understanding 3D object structure from a single image is an important bu...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.