Towards Disturbance-Free Visual Mobile Manipulation

by   Tianwei Ni, et al.

Embodied AI has shown promising results on an abundance of robotic tasks in simulation, including visual navigation and manipulation. The prior work generally pursues high success rates with shortest paths while largely ignoring the problems caused by collision during interaction. This lack of prioritization is understandable: in simulated environments there is no inherent cost to breaking virtual objects. As a result, well-trained agents frequently have catastrophic collision with objects despite final success. In the robotics community, where the cost of collision is large, collision avoidance is a long-standing and crucial topic to ensure that robots can be safely deployed in the real world. In this work, we take the first step towards collision/disturbance-free embodied AI agents for visual mobile manipulation, facilitating safe deployment in real robots. We develop a new disturbance-avoidance methodology at the heart of which is the auxiliary task of disturbance prediction. When combined with a disturbance penalty, our auxiliary task greatly enhances sample efficiency and final performance by knowledge distillation of disturbance into the agent. Our experiments on ManipulaTHOR show that, on testing scenes with novel objects, our method improves the success rate from 61.7 disturbance from 29.8 studies show the value of our pipelined approach. Project site is at


page 1

page 4

page 7

page 17


Deadlock-Free Collision Avoidance for Nonholonomic Robots

We present a method for deadlock-free and collision-free navigation in a...

Large-Scale Package Manipulation via Learned Metrics of Pick Success

Automating warehouse operations can reduce logistics overhead costs, ult...

Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learning

Developing a safe and efficient collision avoidance policy for multiple ...

DIViS: Domain Invariant Visual Servoing for Collision-Free Goal Reaching

Robots should understand both semantics and physics to be functional in ...

Flies as Ship Captains? Digital Evolution Unravels Selective Pressures to Avoid Collision in Drosophila

Flies that walk in a covered planar arena on straight paths avoid collid...

Safe Real-World Reinforcement Learning for Mobile Agent Obstacle Avoidance

Collision avoidance is key for mobile robots and agents to operate safel...

Distributed Swarm Collision Avoidance Based on Angular Calculations

Collision avoidance is one of the most important topics in the robotics ...

Please sign up or login with your details

Forgot password? Click here to reset