3D Colored Shape Reconstruction from a Single RGB Image through Diffusion

02/11/2023
by   Bo Li, et al.
0

We propose a novel 3d colored shape reconstruction method from a single RGB image through diffusion model. Diffusion models have shown great development potentials for high-quality 3D shape generation. However, most existing work based on diffusion models only focus on geometric shape generation, they cannot either accomplish 3D reconstruction from a single image, or produce 3D geometric shape with color information. In this work, we propose to reconstruct a 3D colored shape from a single RGB image through a novel conditional diffusion model. The reverse process of the proposed diffusion model is consisted of three modules, shape prediction module, color prediction module and NeRF-like rendering module. In shape prediction module, the reference RGB image is first encoded into a high-level shape feature and then the shape feature is utilized as a condition to predict the reverse geometric noise in diffusion model. Then the color of each 3D point updated in shape prediction module is predicted by color prediction module. Finally, a NeRF-like rendering module is designed to render the colored point cloud predicted by the former two modules to 2D image space to guide the training conditioned only on a reference image. As far as the authors know, the proposed method is the first diffusion model for 3D colored shape reconstruction from a single RGB image. Experimental results demonstrate that the proposed method achieves competitive performance on colored 3D shape reconstruction, and the ablation study validates the positive role of the color prediction module in improving the reconstruction quality of 3D geometric point cloud.

READ FULL TEXT

page 6

page 7

research
08/15/2023

CCD-3DR: Consistent Conditioning in Diffusion for Single-Image 3D Reconstruction

In this paper, we present a novel shape reconstruction method leveraging...
research
02/21/2023

PC^2: Projection-Conditioned Point Cloud Diffusion for Single-Image 3D Reconstruction

Reconstructing the 3D shape of an object from a single RGB image is a lo...
research
08/31/2023

BuilDiff: 3D Building Shape Generation using Single-Image Conditional Point Cloud Diffusion Models

3D building generation with low data acquisition costs, such as single i...
research
03/18/2023

3DQD: Generalized Deep 3D Shape Prior via Part-Discretized Diffusion Process

We develop a generalized 3D shape generation prior model, tailored for m...
research
04/17/2018

Im2Avatar: Colorful 3D Reconstruction from a Single Image

Existing works on single-image 3D reconstruction mainly focus on shape r...
research
06/21/2017

Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction

Conventional methods of 3D object generative modeling learn volumetric p...
research
03/14/2023

Point Cloud Diffusion Models for Automatic Implant Generation

Advances in 3D printing of biocompatible materials make patient-specific...

Please sign up or login with your details

Forgot password? Click here to reset