Semi-supervised Dense Keypointsusing Unlabeled Multiview Images

by   Zhixuan Yu, et al.

This paper presents a new end-to-end semi-supervised framework to learn a dense keypoint detector using unlabeled multiview images. A key challenge lies in finding the exact correspondences between the dense keypoints in multiple views since the inverse of keypoint mapping can be neither analytically derived nor differentiated. This limits applying existing multiview supervision approaches on sparse keypoint detection that rely on the exact correspondences. To address this challenge, we derive a new probabilistic epipolar constraint that encodes the two desired properties. (1) Soft correspondence: we define a matchability, which measures a likelihood of a point matching to the other image's corresponding point, thus relaxing the exact correspondences' requirement. (2) Geometric consistency: every point in the continuous correspondence fields must satisfy the multiview consistency collectively. We formulate a probabilistic epipolar constraint using a weighted average of epipolar errors through the matchability thereby generalizing the point-to-point geometric error to the field-to-field geometric error. This generalization facilitates learning a geometrically coherent dense keypoint detection model by utilizing a large number of unlabeled multiview images. Additionally, to prevent degenerative cases, we employ a distillation-based regularization by using a pretrained model. Finally, we design a new neural network architecture, made of twin networks, that effectively minimizes the probabilistic epipolar errors of all possible correspondences between two view images by building affinity matrices. Our method shows superior performance compared to existing methods, including non-differentiable bootstrapping in terms of keypoint accuracy, multiview consistency, and 3D reconstruction accuracy.


page 1

page 2

page 3

page 4

page 5

page 6

page 7

page 8


MONET: Multiview Semi-supervised Keypoint via Epipolar Divergence

This paper presents MONET---an end-to-end semi-supervised learning frame...

Multiview Supervision By Registration

This paper presents a semi-supervised learning framework to train a keyp...

Warp Consistency for Unsupervised Learning of Dense Correspondences

The key challenge in learning dense correspondences lies in the lack of ...

Estimation of Large Motion in Lung CT by Integrating Regularized Keypoint Correspondences into Dense Deformable Registration

We present a novel algorithm for the registration of pulmonary CT scans....

Keypoint Transfer for Fast Whole-Body Segmentation

We introduce an approach for image segmentation based on sparse correspo...

Canonical Surface Mapping via Geometric Cycle Consistency

We explore the task of Canonical Surface Mapping (CSM). Specifically, gi...

Dense 3D Face Correspondence

We present an algorithm that automatically establishes dense corresponde...

Please sign up or login with your details

Forgot password? Click here to reset