Log In Sign Up

VTNet: Visual Transformer Network for Object Goal Navigation

by   Heming Du, et al.

Object goal navigation aims to steer an agent towards a target object based on observations of the agent. It is of pivotal importance to design effective visual representations of the observed scene in determining navigation actions. In this paper, we introduce a Visual Transformer Network (VTNet) for learning informative visual representation in navigation. VTNet is a highly effective structure that embodies two key properties for visual representations: First, the relationships among all the object instances in a scene are exploited; Second, the spatial locations of objects and image regions are emphasized so that directional navigation signals can be learned. Furthermore, we also develop a pre-training scheme to associate the visual representations with navigation signals, and thus facilitate navigation policy learning. In a nutshell, VTNet embeds object and region features with their location cues as spatial-aware descriptors and then incorporates all the encoded descriptors through attention operations to achieve informative representation for navigation. Given such visual representations, agents are able to explore the correlations between visual observations and navigation actions. For example, an agent would prioritize "turning right" over "turning left" when the visual representation emphasizes on the right side of activation map. Experiments in the artificial environment AI2-Thor demonstrate that VTNet significantly outperforms state-of-the-art methods in unseen testing environments.


page 2

page 4

page 13

page 14

page 16


Agent-Centric Relation Graph for Object Visual Navigation

Object visual navigation aims to steer an agent towards a target object ...

Teaching Agents how to Map: Spatial Reasoning for Multi-Object Navigation

In the context of visual navigation, the capacity to map a novel environ...

Situational Fusion of Visual Representation for Visual Navigation

A complex visual navigation task puts an agent in different situations w...

Object-oriented Targets for Visual Navigation using Rich Semantic Representations

When searching for an object humans navigate through a scene using seman...

Pushing it out of the Way: Interactive Visual Navigation

We have observed significant progress in visual navigation for embodied ...

Learning Object Relation Graph and Tentative Policy for Visual Navigation

Target-driven visual navigation aims at navigating an agent towards a gi...

Visual Pre-training for Navigation: What Can We Learn from Noise?

A powerful paradigm for sensorimotor control is to predict actions from ...