Training Larger Networks for Deep Reinforcement Learning

02/16/2021
by   Kei Ota, et al.
127

The success of deep learning in the computer vision and natural language processing communities can be attributed to training of very deep neural networks with millions or billions of parameters which can then be trained with massive amounts of data. However, similar trend has largely eluded training of deep reinforcement learning (RL) algorithms where larger networks do not lead to performance improvement. Previous work has shown that this is mostly due to instability during training of deep RL agents when using larger networks. In this paper, we make an attempt to understand and address training of larger networks for deep RL. We first show that naively increasing network capacity does not improve performance. Then, we propose a novel method that consists of 1) wider networks with DenseNet connection, 2) decoupling representation learning from training of RL, 3) a distributed training method to mitigate overfitting problems. Using this three-fold technique, we show that we can train very large networks that result in significant performance gains. We present several ablation studies to demonstrate the efficacy of the proposed method and some intuitive understanding of the reasons for performance gain. We show that our proposed method outperforms other baseline algorithms on several challenging locomotion tasks.

READ FULL TEXT

page 3

page 5

page 6

page 16

research
03/03/2020

Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?

Deep reinforcement learning (RL) algorithms have recently achieved remar...
research
06/02/2021

Towards Deeper Deep Reinforcement Learning

In computer vision and natural language processing, innovations in model...
research
10/24/2019

Pre-training in Deep Reinforcement Learning for Automatic Speech Recognition

Deep reinforcement learning (deep RL) is a combination of deep learning ...
research
05/21/2020

Deep Reinforcement Learning with Pre-training for Time-efficient Training of Automatic Speech Recognition

Deep reinforcement learning (deep RL) is a combination of deep learning ...
research
10/07/2021

Generalization in Deep RL for TSP Problems via Equivariance and Local Search

Deep reinforcement learning (RL) has proved to be a competitive heuristi...
research
10/16/2019

On Learning Paradigms for the Travelling Salesman Problem

We explore the impact of learning paradigms on training deep neural netw...
research
05/22/2020

Adaptive Reinforcement Learning through Evolving Self-Modifying Neural Networks

The adaptive learning capabilities seen in biological neural networks ar...

Please sign up or login with your details

Forgot password? Click here to reset