End-to-End Egospheric Spatial Memory

02/15/2021
by   Daniel Lenton, et al.
8

Spatial memory, or the ability to remember and recall specific locations and objects, is central to autonomous agents' ability to carry out tasks in real environments. However, most existing artificial memory modules have difficulty recalling information over long time periods and are not very adept at storing spatial information. We propose a parameter-free module, Egospheric Spatial Memory (ESM), which encodes the memory in an ego-sphere around the agent, enabling expressive 3D representations. ESM can be trained end-to-end via either imitation or reinforcement learning, and improves both training efficiency and final performance against other memory baselines on both drone and manipulator visuomotor control tasks. The explicit egocentric geometry also enables us to seamlessly combine the learned controller with other non-learned modalities, such as local obstacle avoidance. We further show applications to semantic segmentation on the ScanNet dataset, where ESM naturally combines image-level and map-level inference modalities. Through our broad set of experiments, we show that ESM provides a general computation graph for embodied spatial reasoning, and the module forms a bridge between real-time mapping systems and differentiable memory architectures.

READ FULL TEXT

page 6

page 7

page 8

page 9

page 15

page 16

page 17

page 20

research
07/31/2018

Egocentric Spatial Memory

Egocentric spatial memory (ESM) defines a memory system with encoding, s...
research
07/13/2022

Trans4Map: Revisiting Holistic Top-down Mapping from Egocentric Images to Allocentric Semantics with Vision Transformers

Humans have an innate ability to sense their surroundings, as they can e...
research
09/24/2021

CLIPort: What and Where Pathways for Robotic Manipulation

How can we imbue robots with the ability to manipulate objects precisely...
research
07/26/2016

Region-based semantic segmentation with end-to-end training

We propose a novel method for semantic segmentation, the task of labelin...
research
05/31/2018

Following High-level Navigation Instructions on a Simulated Quadcopter with Imitation Learning

We introduce a method for following high-level navigation instructions b...
research
02/17/2022

Shift-Memory Network for Temporal Scene Segmentation

Semantic segmentation has achieved great accuracy in understanding spati...
research
09/16/2021

End-to-End Partially Observable Visual Navigation in a Diverse Environment

How can a robot navigate successfully in a rich and diverse environment,...

Please sign up or login with your details

Forgot password? Click here to reset