Walking with MIND: Mental Imagery eNhanceD Embodied QA

by   Juncheng Li, et al.

The EmbodiedQA is a task of training an embodied agent by intelligently navigating in a simulated environment and gathering visual information to answer questions. Existing approaches fail to explicitly model the mental imagery function of the agent, while the mental imagery is crucial to embodied cognition, and has a close relation to many high-level meta-skills such as generalization and interpretation. In this paper, we propose a novel Mental Imagery eNhanceD (MIND) module for the embodied agent, as well as a relevant deep reinforcement framework for training. The MIND module can not only model the dynamics of the environment (e.g. 'what might happen if the agent passes through a door') but also help the agent to create a better understanding of the environment (e.g. 'The refrigerator is usually in the kitchen'). Such knowledge makes the agent a faster and better learner in locating a feasible policy with only a few trails. Furthermore, the MIND module can generate mental images that are treated as short-term subgoals by our proposed deep reinforcement framework. These mental images facilitate policy learning since short-term subgoals are easy to achieve and reusable. This yields better planning efficiency than other algorithms that learn a policy directly from primitive actions. Finally, the mental images visualize the agent's intentions in a way that human can understand, and this endows our agent's actions with more interpretability. The experimental results and further analysis prove that the agent with the MIND module is superior to its counterparts not only in EQA performance but in many other aspects such as route planning, behavioral interpretation, and the ability to generalize from a few examples.


page 1

page 4

page 5

page 8


Simulated Mental Imagery for Robotic Task Planning

Traditional AI-planning methods for task planning in robotics require sy...

How to mind your own business

I was walking past the mental hospital the other day, and all the patien...

A Model of Spatial Thinking for Computational Intelligence

Trying to be effective (no matter who exactly and in what field) a perso...

Efficient Reinforcement Learning with a Mind-Game for Full-Length StarCraft II

StarCraft II provides an extremely challenging platform for reinforcemen...

MIRA: Mental Imagery for Robotic Affordances

Humans form mental images of 3D scenes to support counterfactual imagina...

Interactive Agent Modeling by Learning to Probe

The ability of modeling the other agents, such as understanding their in...

Deep Interpretable Models of Theory of Mind For Human-Agent Teaming

When developing AI systems that interact with humans, it is essential to...

Please sign up or login with your details

Forgot password? Click here to reset