Batch Exploration with Examples for Scalable Robotic Reinforcement Learning

10/22/2020
by   Annie S. Chen, et al.
6

Learning from diverse offline datasets is a promising path towards learning general purpose robotic agents. However, a core challenge in this paradigm lies in collecting large amounts of meaningful data, while not depending on a human in the loop for data collection. One way to address this challenge is through task-agnostic exploration, where an agent attempts to explore without a task-specific reward function, and collect data that can be useful for any downstream task. While these approaches have shown some promise in simple domains, they often struggle to explore the relevant regions of the state space in more challenging settings, such as vision based robotic manipulation. This challenge stems from an objective that encourages exploring everything in a potentially vast state space. To mitigate this challenge, we propose to focus exploration on the important parts of the state space using weak human supervision. Concretely, we propose an exploration technique, Batch Exploration with Examples (BEE), that explores relevant regions of the state-space, guided by a modest number of human provided images of important states. These human provided images only need to be collected once at the beginning of data collection and can be collected in a matter of minutes, allowing us to scalably collect diverse datasets, which can then be combined with any batch RL algorithm. We find that BEE is able to tackle challenging vision-based manipulation tasks both in simulation and on a real Franka robot, and observe that compared to task-agnostic and weakly-supervised exploration techniques, it (1) interacts more than twice as often with relevant objects, and (2) improves downstream task performance when used in conjunction with offline RL.

READ FULL TEXT

page 1

page 4

page 5

page 7

research
01/27/2022

The Challenges of Exploration for Offline Reinforcement Learning

Offline Reinforcement Learning (ORL) enablesus to separately study the t...
research
05/10/2021

Efficient Self-Supervised Data Collection for Offline Robot Learning

A practical approach to robot reinforcement learning is to first collect...
research
09/16/2021

Conservative Data Sharing for Multi-Task Offline Reinforcement Learning

Offline reinforcement learning (RL) algorithms have shown promising resu...
research
07/14/2020

Goal-Aware Prediction: Learning to Model What Matters

Learned dynamics models combined with both planning and policy learning ...
research
04/06/2020

Weakly-Supervised Reinforcement Learning for Controllable Behavior

Reinforcement learning (RL) is a powerful framework for learning to take...
research
03/10/2020

Active Reward Learning for Co-Robotic Vision Based Exploration in Bandwidth Limited Environments

We present a novel POMDP problem formulation for a robot that must auton...
research
10/23/2022

Learning General World Models in a Handful of Reward-Free Deployments

Building generally capable agents is a grand challenge for deep reinforc...

Please sign up or login with your details

Forgot password? Click here to reset