DeepAI
Log In Sign Up

SPEC: Seeing People in the Wild with an Estimated Camera

10/01/2021
by   Muhammed Kocabas, et al.
2

Due to the lack of camera parameter information for in-the-wild images, existing 3D human pose and shape (HPS) estimation methods make several simplifying assumptions: weak-perspective projection, large constant focal length, and zero camera rotation. These assumptions often do not hold and we show, quantitatively and qualitatively, that they cause errors in the reconstructed 3D shape and pose. To address this, we introduce SPEC, the first in-the-wild 3D HPS method that estimates the perspective camera from a single image and employs this to reconstruct 3D human bodies more accurately. 3D human bodies. First, we train a neural network to estimate the field of view, camera pitch, and roll given an input image. We employ novel losses that improve the calibration accuracy over previous work. We then train a novel network that concatenates the camera calibration to the image features and uses these together to regress 3D body shape and pose. SPEC is more accurate than the prior art on the standard benchmark (3DPW) as well as two new datasets with more challenging camera views and varying focal lengths. Specifically, we create a new photorealistic synthetic dataset (SPEC-SYN) with ground truth 3D bodies and a novel in-the-wild dataset (SPEC-MTP) with calibration and high-quality reference bodies. Both qualitative and quantitative analysis confirm that knowing camera parameters during inference regresses better human bodies. Code and datasets are available for research purposes at https://spec.is.tue.mpg.de.

READ FULL TEXT

page 8

page 14

page 15

page 16

page 17

page 18

page 21

page 22

01/20/2022

Estimating Egocentric 3D Human Pose in the Wild with External Weak Supervision

Egocentric 3D human pose estimation with a single fisheye camera has dra...
07/26/2022

A Reliable Online Method for Joint Estimation of Focal Length and Camera Rotation

Linear perspectivecues deriving from regularities of the built environme...
08/01/2022

CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation

Top-down methods dominate the field of 3D human pose and shape estimatio...
12/31/2021

Learning to Predict 3D Lane Shape and Camera Pose from a Single Image via Geometry Constraints

Detecting 3D lanes from the camera is a rising problem for autonomous ve...
09/14/2020

Beyond Weak Perspective for Monocular 3D Human Pose Estimation

We consider the task of 3D joints location and orientation prediction fr...
11/27/2020

PCLs: Geometry-aware Neural Reconstruction of 3D Pose with Perspective Crop Layers

Local processing is an essential feature of CNNs and other neural networ...
07/12/2021

Multi-view Image-based Hand Geometry Refinement using Differentiable Monte Carlo Ray Tracing

The amount and quality of datasets and tools available in the research f...