SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation

12/08/2022
by   Yen-Chi Cheng, et al.
0

In this work, we present a novel framework built to simplify 3D asset generation for amateur users. To enable interactive generation, our method supports a variety of input modalities that can be easily provided by a human, including images, text, partially observed shapes and combinations of these, further allowing to adjust the strength of each input. At the core of our approach is an encoder-decoder, compressing 3D shapes into a compact latent representation, upon which a diffusion model is learned. To enable a variety of multi-modal inputs, we employ task-specific encoders with dropout followed by a cross-attention mechanism. Due to its flexibility, our model naturally supports a variety of tasks, outperforming prior works on shape completion, image-based 3D reconstruction, and text-to-3D. Most interestingly, our model can combine all these tasks into one swiss-army-knife tool, enabling the user to perform shape generation using incomplete shapes, images, and textual descriptions at the same time, providing the relative weights for each input and facilitating interactivity. Despite our approach being shape-only, we further show an efficient method to texture the generated shape using large-scale text-to-image models.

READ FULL TEXT

page 1

page 5

page 7

page 8

page 12

page 13

page 15

page 16

research
03/05/2023

Text2Face: A Multi-Modal 3D Face Model

We present the first 3D morphable modelling approach, whereby 3D face sh...
research
12/06/2022

Diffusion-SDF: Text-to-Shape via Voxelized Diffusion

With the rising industrial attention to 3D virtual modeling technology, ...
research
03/17/2022

AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation

Powerful priors allow us to perform inference with insufficient informat...
research
09/09/2022

ISS: Image as Stetting Stone for Text-Guided 3D Shape Generation

Text-guided 3D shape generation remains challenging due to the absence o...
research
06/28/2023

DiffComplete: Diffusion-based Generative 3D Shape Completion

We introduce a new diffusion-based approach for shape completion on 3D r...
research
07/19/2022

ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Model

We present ShapeCrafter, a neural network for recursive text-conditioned...
research
07/24/2022

Cross-Modal 3D Shape Generation and Manipulation

Creating and editing the shape and color of 3D objects require tremendou...

Please sign up or login with your details

Forgot password? Click here to reset