Sample-Efficient Automated Deep Reinforcement Learning

by   Jörg K. H. Franke, et al.

Despite significant progress in challenging problems across various domains, applying state-of-the-art deep reinforcement learning (RL) algorithms remains challenging due to their sensitivity to the choice of hyperparameters. This sensitivity can partly be attributed to the non-stationarity of the RL problem, potentially requiring different hyperparameter settings at different stages of the learning process. Additionally, in the RL setting, hyperparameter optimization (HPO) requires a large number of environment interactions, hindering the transfer of the successes in RL to real-world applications. In this work, we tackle the issues of sample-efficient and dynamic HPO in RL. We propose a population-based automated RL (AutoRL) framework to meta-optimize arbitrary off-policy RL algorithms. In this framework, we optimize the hyperparameters, including architecture hyperparameters while simultaneously training the agent. By sharing the collected experience across the population, we substantially increase the sample efficiency of the meta-optimization. We demonstrate the capabilities of our sample-efficient AutoRL approach in a case study with the popular TD3 algorithm in the MuJoCo benchmark suite, where we reduce the number of environment interactions needed for meta-optimization by up to an order of magnitude compared to population-based training.


page 1

page 2

page 3

page 4


Hyperparameter Tuning for Deep Reinforcement Learning Applications

Reinforcement learning (RL) applications, where an agent can simply lear...

Towards Augmented Microscopy with Reinforcement Learning-Enhanced Workflows

Here, we report a case study implementation of reinforcement learning (R...

Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL

Despite a series of recent successes in reinforcement learning (RL), man...

REIN-2: Giving Birth to Prepared Reinforcement Learning Agents Using Reinforcement Learning Agents

Deep Reinforcement Learning (Deep RL) has been in the spotlight for the ...

Data Efficient Training for Reinforcement Learning with Adaptive Behavior Policy Sharing

Deep Reinforcement Learning (RL) is proven powerful for decision making ...

Bayesian Generational Population-Based Training

Reinforcement learning (RL) offers the potential for training generally ...

Continual Meta-Reinforcement Learning for UAV-Aided Vehicular Wireless Networks

Unmanned aerial base stations (UABSs) can be deployed in vehicular wirel...