Log In Sign Up

Spatial Intention Maps for Multi-Agent Mobile Manipulation

by   Jimmy Wu, et al.

The ability to communicate intention enables decentralized multi-agent robots to collaborate while performing physical tasks. In this work, we present spatial intention maps, a new intention representation for multi-agent vision-based deep reinforcement learning that improves coordination between decentralized mobile manipulators. In this representation, each agent's intention is provided to other agents, and rendered into an overhead 2D map aligned with visual observations. This synergizes with the recently proposed spatial action maps framework, in which state and action representations are spatially aligned, providing inductive biases that encourage emergent cooperative behaviors requiring spatial coordination, such as passing objects to each other or avoiding collisions. Experiments across a variety of multi-agent environments, including heterogeneous robot teams with different abilities (lifting, pushing, or throwing), show that incorporating spatial intention maps improves performance for different mobile manipulation tasks while significantly enhancing cooperative behaviors.


page 1

page 2

page 3

page 4

page 5

page 9


CH-MARL: A Multimodal Benchmark for Cooperative, Heterogeneous Multi-Agent Reinforcement Learning

We propose a multimodal (vision-and-language) benchmark for cooperative ...

Joint Attention for Multi-Agent Coordination and Social Learning

Joint attention - the ability to purposefully coordinate attention with ...

HAMMER: Multi-Level Coordination of Reinforcement Learning Agents via Learned Messaging

Cooperative multi-agent reinforcement learning (MARL) has achieved signi...

Intention Recognition for Multiple Agents

Intention recognition is an important step to facilitate collaboration i...

Spatial Action Maps for Mobile Manipulation

This paper proposes a new action representation for learning to perform ...

Genetic agent approach for improving on-the-fly web map generalization

The utilization of web mapping becomes increasingly important in the dom...

Code Repositories


Learning multi-agent robotic mobile manipulation with deep reinforcement learning

view repo