MIRA: Mental Imagery for Robotic Affordances

by   Lin Yen-Chen, et al.

Humans form mental images of 3D scenes to support counterfactual imagination, planning, and motor control. Our abilities to predict the appearance and affordance of the scene from previously unobserved viewpoints aid us in performing manipulation tasks (e.g., 6-DoF kitting) with a level of ease that is currently out of reach for existing robot learning frameworks. In this work, we aim to build artificial systems that can analogously plan actions on top of imagined images. To this end, we introduce Mental Imagery for Robotic Affordances (MIRA), an action reasoning framework that optimizes actions with novel-view synthesis and affordance prediction in the loop. Given a set of 2D RGB images, MIRA builds a consistent 3D scene representation, through which we synthesize novel orthographic views amenable to pixel-wise affordances prediction for action optimization. We illustrate how this optimization process enables us to generalize to unseen out-of-plane rotations for 6-DoF robotic manipulation tasks given a limited number of demonstrations, paving the way toward machines that autonomously learn to understand the world around them for planning actions.


page 2

page 6

page 9


Simulated Mental Imagery for Robotic Task Planning

Traditional AI-planning methods for task planning in robotics require sy...

3D Neural Scene Representations for Visuomotor Control

Humans have a strong intuitive understanding of the 3D environment aroun...

Learning to Scaffold the Development of Robotic Manipulation Skills

Learning contact-rich, robotic manipulation skills is a challenging prob...

URoboSim – An Episodic Simulation Framework for Prospective Reasoning in Robotic Agents

Anticipating what might happen as a result of an action is an essential ...

Walking with MIND: Mental Imagery eNhanceD Embodied QA

The EmbodiedQA is a task of training an embodied agent by intelligently ...

Advantage of prediction and mental imagery for goal‐directed behaviour in agents and robots

Mental imagery and planning are important aspects of cognitive behaviour...

Please sign up or login with your details

Forgot password? Click here to reset