Data-efficient visuomotor policy training using reinforcement learning and generative models

by   Ali Ghadirzadeh, et al.
KTH Royal Institute of Technology

We present a data-efficient framework for solving deep visuomotor sequential decision-making problems which exploits the combination of reinforcement learning (RL) with the latent variable generative models. Our framework trains deep visuomotor policies by introducing an action latent variable such that the feed-forward policy search can be divided into two parts: (1) training a sub-policy that outputs a distribution over the action latent variable given a state of the system, and (2) training a generative model that outputs a sequence of motor actions given a latent action representation. Our approach enables safe exploration and alleviates the data-inefficiency problem as it exploits prior knowledge about valid sequences of motor actions. Moreover, by evaluating the quality of the generative models we are able to predict the performance of the RL policy training prior to the actual training on the physical robot. We achieve this by defining two novel measures, disentanglement and local linearity, for assessing the quality of generative models' latent spaces, and complementing them with the existing measures for evaluation of generative models. We demonstrate the efficiency of our approach on a picking task using several different generative models and determine which of their properties have the most influence on the final policy training.


page 1

page 2

page 3

page 4


Training and Evaluation of Deep Policies using Reinforcement Learning and Generative Models

We present a data-efficient framework for solving sequential decision-ma...

Decision Stacks: Flexible Reinforcement Learning via Modular Generative Models

Reinforcement learning presents an attractive paradigm to reason about s...

Generative Multiple-Instance Learning Models For Quantitative Electromyography

We present a comprehensive study of the use of generative modeling appro...

Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models

Defining action spaces for conversational agents and optimizing their de...

Causal Graphs Underlying Generative Models: Path to Learning with Limited Data

Training generative models that capture rich semantics of the data and i...

Evaluating Disentanglement in Generative Models Without Knowledge of Latent Factors

Probabilistic generative models provide a flexible and systematic framew...

VPE: Variational Policy Embedding for Transfer Reinforcement Learning

Reinforcement Learning methods are capable of solving complex problems, ...

Please sign up or login with your details

Forgot password? Click here to reset