Consistent View Synthesis with Pose-Guided Diffusion Models

03/30/2023
by   Hung-Yu Tseng, et al.
0

Novel view synthesis from a single image has been a cornerstone problem for many Virtual Reality applications that provide immersive experiences. However, most existing techniques can only synthesize novel views within a limited range of camera motion or fail to generate consistent and high-quality novel views under significant camera movement. In this work, we propose a pose-guided diffusion model to generate a consistent long-term video of novel views from a single image. We design an attention layer that uses epipolar lines as constraints to facilitate the association between different viewpoints. Experimental results on synthetic and real-world datasets demonstrate the effectiveness of the proposed diffusion model against state-of-the-art transformer-based and GAN-based approaches.

READ FULL TEXT

page 1

page 3

page 4

page 5

page 6

page 8

page 13

page 14

research
04/05/2023

Generative Novel View Synthesis with 3D-Aware Diffusion Models

We present a diffusion-based model for 3D-aware generative novel view sy...
research
02/20/2023

NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion

Novel view synthesis from a single image requires inferring occluded reg...
research
09/19/2023

ReShader: View-Dependent Highlights for Single Image View-Synthesis

In recent years, novel view synthesis from a single image has seen signi...
research
11/29/2022

NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360° Views

Virtual reality and augmented reality (XR) bring increasing demand for 3...
research
08/20/2023

Make-It-4D: Synthesizing a Consistent Long-Term Dynamic Scene Video from a Single Image

We study the problem of synthesizing a long-term dynamic video from only...
research
10/24/2022

EpipolarNVS: leveraging on Epipolar geometry for single-image Novel View Synthesis

Novel-view synthesis (NVS) can be tackled through different approaches, ...
research
04/15/2021

Geometry-Free View Synthesis: Transformers and no 3D Priors

Is a geometric model required to synthesize novel views from a single im...

Please sign up or login with your details

Forgot password? Click here to reset