FrameNet: Learning Local Canonical Frames of 3D Surfaces from a Single RGB Image

03/29/2019
by   Jingwei Huang, et al.
4

In this work, we introduce the novel problem of identifying dense canonical 3D coordinate frames from a single RGB image. We observe that each pixel in an image corresponds to a surface in the underlying 3D geometry, where a canonical frame can be identified as represented by three orthogonal axes, one along its normal direction and two in its tangent plane. We propose an algorithm to predict these axes from RGB. Our first insight is that canonical frames computed automatically with recently introduced direction field synthesis methods can provide training data for the task. Our second insight is that networks designed for surface normal prediction provide better results when trained jointly to predict canonical frames, and even better when trained to also predict 2D projections of canonical frames. We conjecture this is because projections of canonical tangent directions often align with local gradients in images, and because those directions are tightly linked to 3D canonical frames through projective geometry and orthogonality constraints. In our experiments, we find that our method predicts 3D canonical frames that can be used in applications ranging from surface normal estimation, feature matching, and augmented reality.

READ FULL TEXT

page 2

page 3

page 5

page 7

page 8

page 11

page 12

page 13

research
12/05/2022

ObjectMatch: Robust Registration using Canonical Object Correspondences

We present ObjectMatch, a semantic and object-centric camera pose estima...
research
12/15/2020

Seeing Behind Objects for 3D Multi-Object Tracking in RGB-D Sequences

Multi-object tracking from RGB-D video sequences is a challenging proble...
research
11/25/2020

DRACO: Weakly Supervised Dense Reconstruction And Canonicalization of Objects

We present DRACO, a method for Dense Reconstruction And Canonicalization...
research
04/14/2020

Footprints and Free Space from a Single Color Image

Understanding the shape of a scene from a single color image is a formid...
research
06/05/2020

Novel Object Viewpoint Estimation through Reconstruction Alignment

The goal of this paper is to estimate the viewpoint for a novel object. ...
research
10/08/2019

Canonical extensions of locally compact frames

Canonical extension of finitary ordered structures such as lattices, pos...
research
06/09/2021

Learning normal form autoencoders for data-driven discovery of universal,parameter-dependent governing equations

Complex systems manifest a small number of instabilities and bifurcation...

Please sign up or login with your details

Forgot password? Click here to reset