DeepAI AI Chat
Log In Sign Up

An Effective Loss Function for Generating 3D Models from Single 2D Image without Rendering

by   Nikola Zubić, et al.

Differentiable rendering is a very successful technique that applies to a Single-View 3D Reconstruction. Current renderers use losses based on pixels between a rendered image of some 3D reconstructed object and ground-truth images from given matched viewpoints to optimise parameters of the 3D shape. These models require a rendering step, along with visibility handling and evaluation of the shading model. The main goal of this paper is to demonstrate that we can avoid these steps and still get reconstruction results as other state-of-the-art models that are equal or even better than existing category-specific reconstruction methods. First, we use the same CNN architecture for the prediction of a point cloud shape and pose prediction like the one used by Insafutdinov & Dosovitskiy. Secondly, we propose the novel effective loss function that evaluates how well the projections of reconstructed 3D point clouds cover the ground truth object's silhouette. Then we use Poisson Surface Reconstruction to transform the reconstructed point cloud into a 3D mesh. Finally, we perform a GAN-based texture mapping on a particular 3D mesh and produce a textured 3D mesh from a single 2D image. We evaluate our method on different datasets (including ShapeNet, CUB-200-2011, and Pascal3D+) and achieve state-of-the-art results, outperforming all the other supervised and unsupervised methods and 3D representations, all in terms of performance, accuracy, and training time.


page 3

page 5

page 6

page 9


DRWR: A Differentiable Renderer without Rendering for Unsupervised 3D Structure Learning from Silhouette Images

Differentiable renderers have been used successfully for unsupervised 3D...

Learning Nonparametric Human Mesh Reconstruction from a Single Image without Ground Truth Meshes

Nonparametric approaches have shown promising results on reconstructing ...

Visual Enhanced 3D Point Cloud Reconstruction from A Single Image

Solving the challenging problem of 3D object reconstruction from a singl...

Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction

Rendering is the process of generating 2D images from 3D assets, simulat...

C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion

We propose C3DPO, a method for extracting 3D models of deformable object...

3D Colored Shape Reconstruction from a Single RGB Image through Diffusion

We propose a novel 3d colored shape reconstruction method from a single ...

Predicting the Next Best View for 3D Mesh Refinement

3D reconstruction is a core task in many applications such as robot navi...