REPAINT: Knowledge Transfer in Deep Actor-Critic Reinforcement Learning

by   Yunzhe Tao, et al.

Accelerating the learning processes for complex tasks by leveraging previously learned tasks has been one of the most challenging problems in reinforcement learning, especially when the similarity between source and target tasks is low or unknown. In this work, we propose a REPresentation-And-INstance Transfer algorithm (REPAINT) for deep actor-critic reinforcement learning paradigm. In representation transfer, we adopt a kickstarted training method using a pre-trained teacher policy by introducing an auxiliary cross-entropy loss. In instance transfer, we develop a sampling approach, i.e., advantage-based experience replay, on transitions collected following the teacher policy, where only the samples with high advantage estimates are retained for policy update. We consider both learning an unseen target task by transferring from previously learned teacher tasks and learning a partially unseen task composed of multiple sub-tasks by transferring from a pre-learned teacher sub-task. In several benchmark experiments, REPAINT significantly reduces the total training time and improves the asymptotic performance compared to training with no prior knowledge and other baselines.


Asynchronous Advantage Actor-Critic Agent for Starcraft II

Deep reinforcement learning, and especially the Asynchronous Advantage A...

Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past

Soft Actor-Critic (SAC) is an off-policy actor-critic deep reinforcement...

Distillation Strategies for Proximal Policy Optimization

Vision-based deep reinforcement learning (RL), similar to deep learning,...

Actor Critic with Differentially Private Critic

Reinforcement learning algorithms are known to be sample inefficient, an...

AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers

The exploration mechanism used by a Deep Reinforcement Learning (RL) age...

Deep Learnable Strategy Templates for Multi-Issue Bilateral Negotiation

We study how to exploit the notion of strategy templates to learn strate...

Accelerating Reinforcement Learning by Composing Solutions of Automatically Identified Subtasks

This paper discusses a system that accelerates reinforcement learning by...