SHERF: Generalizable Human NeRF from a Single Image

03/22/2023
by   Shoukang Hu, et al.
0

Existing Human NeRF methods for reconstructing 3D humans typically rely on multiple 2D images from multi-view cameras or monocular videos captured from fixed camera views. However, in real-world scenarios, human images are often captured from random camera angles, presenting challenges for high-quality 3D human reconstruction. In this paper, we propose SHERF, the first generalizable Human NeRF model for recovering animatable 3D humans from a single input image. SHERF extracts and encodes 3D human representations in canonical space, enabling rendering and animation from free views and poses. To achieve high-fidelity novel view and pose synthesis, the encoded 3D human representations should capture both global appearance and local fine-grained textures. To this end, we propose a bank of 3D-aware hierarchical features, including global, point-level, and pixel-aligned features, to facilitate informative encoding. Global features enhance the information extracted from the single input image and complement the information missing from the partial 2D observation. Point-level features provide strong clues of 3D human structure, while pixel-aligned features preserve more fine-grained details. To effectively integrate the 3D-aware hierarchical feature bank, we design a feature fusion transformer. Extensive experiments on THuman, RenderPeople, ZJU_MoCap, and HuMMan datasets demonstrate that SHERF achieves state-of-the-art performance, with better generalizability for novel view and pose synthesis.

READ FULL TEXT

page 1

page 6

page 14

page 15

page 16

page 17

research
07/18/2023

PixelHuman: Animatable Neural Radiance Fields from Few Images

In this paper, we propose PixelHuman, a novel human rendering model that...
research
02/10/2022

PVSeRF: Joint Pixel-, Voxel- and Surface-Aligned Radiance Field for Single-Image Novel View Synthesis

We present PVSeRF, a learning framework that reconstructs neural radianc...
research
07/23/2023

TransHuman: A Transformer-based Human Representation for Generalizable Neural Human Rendering

In this paper, we focus on the task of generalizable neural human render...
research
07/12/2022

Vision Transformer for NeRF-Based View Synthesis from a Single Input Image

Although neural radiance fields (NeRF) have shown impressive advances fo...
research
01/20/2022

HDhuman: High-quality Human Performance Capture with Sparse Views

In this paper, we introduce HDhuman, a method that addresses the challen...
research
12/13/2022

Structured 3D Features for Reconstructing Relightable and Animatable Avatars

We introduce Structured 3D Features, a model based on a novel implicit 3...
research
12/08/2021

Garment4D: Garment Reconstruction from Point Cloud Sequences

Learning to reconstruct 3D garments is important for dressing 3D human b...

Please sign up or login with your details

Forgot password? Click here to reset