Recognizing Scenes from Novel Viewpoints

by   Shengyi Qian, et al.

Humans can perceive scenes in 3D from a handful of 2D views. For AI agents, the ability to recognize a scene from any viewpoint given only a few images enables them to efficiently interact with the scene and its objects. In this work, we attempt to endow machines with this ability. We propose a model which takes as input a few RGB images of a new scene and recognizes the scene from novel viewpoints by segmenting it into semantic categories. All this without access to the RGB images from those views. We pair 2D scene recognition with an implicit 3D representation and learn from multi-view 2D annotations of hundreds of scenes without any 3D supervision beyond camera poses. We experiment on challenging datasets and demonstrate our model's ability to jointly capture semantics and geometry of novel scenes with diverse layouts, object types and shapes.


page 1

page 6

page 7

page 8

page 13

page 14

page 15


Neural Scene Graphs for Dynamic Scenes

Recent implicit neural rendering methods have demonstrated that it is po...

Implicit Shape Model Trees: Recognition of 3-D Indoor Scenes and Prediction of Object Poses for Mobile Robots

For a mobile robot, we present an approach to recognize scenes in arrang...

Indoor Scene Recognition in 3D

Recognising in what type of environment one is located is an important p...

3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera

A comprehensive semantic understanding of a scene is important for many ...

A Number Sense as an Emergent Property of the Manipulating Brain

The ability to understand and manipulate numbers and quantities emerges ...

Semi-Supervised Learning of Multi-Object 3D Scene Representations

Representing scenes at the granularity of objects is a prerequisite for ...

Scenarios: A New Representation for Complex Scene Understanding

The ability for computational agents to reason about the high-level cont...

Please sign up or login with your details

Forgot password? Click here to reset