DeepAI AI Chat
Log In Sign Up

Dynamical Distance Learning for Unsupervised and Semi-Supervised Skill Discovery

by   Kristian Hartikainen, et al.

Reinforcement learning requires manual specification of a reward function to learn a task. While in principle this reward function only needs to specify the task goal, in practice reinforcement learning can be very time-consuming or even infeasible unless the reward function is shaped so as to provide a smooth gradient towards a successful outcome. This shaping is difficult to specify by hand, particularly when the task is learned from raw observations, such as images. In this paper, we study how we can automatically learn dynamical distances: a measure of the expected number of time steps to reach a given goal state from any other state. These dynamical distances can be used to provide well-shaped reward functions for reaching new goals, making it possible to learn complex tasks efficiently. We also show that dynamical distances can be used in a semi-supervised regime, where unsupervised interaction with the environment is used to learn the dynamical distances, while a small amount of preference supervision is used to determine the task goal, without any manually engineered reward function or goal examples. We evaluate our method both in simulation and on a real-world robot. We show that our method can learn locomotion skills in simulation without any supervision. We also show that it can learn to turn a valve with a real-world 9-DoF hand, using raw image observations and ten preference labels, without any other supervision. Videos of the learned skills can be found on the project website:


page 2

page 8

page 13


Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning

Reinforcement learning provides a general framework for learning robotic...

End-to-End Robotic Reinforcement Learning without Reward Engineering

The combination of deep neural network models and reinforcement learning...

Model-Based Visual Planning with Self-Supervised Functional Distances

A generalist robot must be able to complete a variety of tasks in its en...

Automatic Evaluation of Excavator Operators using Learned Reward Functions

Training novice users to operate an excavator for learning different ski...

Learning Arbitrary-Goal Fabric Folding with One Hour of Real Robot Experience

Manipulating deformable objects, such as fabric, is a long standing prob...

Unsupervised Perceptual Rewards for Imitation Learning

Reward function design and exploration time are arguably the biggest obs...

Emergence of Locomotion Behaviours in Rich Environments

The reinforcement learning paradigm allows, in principle, for complex be...