End-to-End Learning of Multi-category 3D Pose and Shape Estimation

by   Yigit Baran Can, et al.

In this paper, we study the representation of the shape and pose of objects using their keypoints. Therefore, we propose an end-to-end method that simultaneously detects 2D keypoints from an image and lifts them to 3D. The proposed method learns both 2D detection and 3D lifting only from 2D keypoints annotations. In this regard, a novel method that explicitly disentangles the pose and 3D shape by means of augmentation-based cyclic self-supervision is proposed, for the first time. In addition of being end-to-end in image to 3D learning, our method also handles objects from multiple categories using a single neural network. We use a Transformer-based architecture to detect the keypoints, as well as to summarize the visual context of the image. This visual context information is then used while lifting the keypoints to 3D, so as to allow the context-based reasoning for better performance. While lifting, our method learns a small set of basis shapes and their sparse non-negative coefficients to represent the 3D shape in canonical frame. Our method can handle occlusions as well as wide variety of object classes. Our experiments on three benchmarks demonstrate that our method performs better than the state-of-the-art. Our source code will be made publicly available.


page 2

page 8


Learning Canonical Shape Space for Category-Level 6D Object Pose and Size Estimation

We present a novel approach to category-level 6D object pose and size es...

Piecewise Planar Hulls for Semi-Supervised Learning of 3D Shape and Pose from 2D Images

We study the problem of estimating 3D shape and pose of an object in ter...

Learning 3D Object Categories by Looking Around Them

Traditional approaches for learning 3D object categories use either synt...

PSVT: End-to-End Multi-person 3D Pose and Shape Estimation with Progressive Video Transformers

Existing methods of multi-person video 3D human Pose and Shape Estimatio...

GeoGraph: Learning graph-based multi-view object detection with geometric cues end-to-end

In this paper we propose an end-to-end learnable approach that detects s...

RayTran: 3D pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformers

We propose a transformer-based neural network architecture for multi-obj...

Optimized edge-based grasping method for a cluttered environment

This paper looks into the problem of grasping region localization along ...

Please sign up or login with your details

Forgot password? Click here to reset