Stabilizing Transformer-Based Action Sequence Generation For Q-Learning

10/23/2020
by   Gideon Stein, et al.
13

Since the publication of the original Transformer architecture (Vaswani et al. 2017), Transformers revolutionized the field of Natural Language Processing. This, mainly due to their ability to understand timely dependencies better than competing RNN-based architectures. Surprisingly, this architecture change does not affect the field of Reinforcement Learning (RL), even though RNNs are quite popular in RL, and time dependencies are very common in RL. Recently, (Parisotto et al. 2019) conducted the first promising research of Transformers in RL. To support the findings of this work, this paper seeks to provide an additional example of a Transformer-based RL method. Specifically, the goal is a simple Transformer-based Deep Q-Learning method that is stable over several environments. Due to the unstable nature of Transformers and RL, an extensive method search was conducted to arrive at a final method that leverages developments around Transformers as well as Q-learning. The proposed method can match the performance of classic Q-learning on control environments while showing potential on some selected Atari benchmarks. Furthermore, it was critically evaluated to give additional insights into the relation between Transformers and RL.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/14/2020

Efficient Transformers: A Survey

Transformer model architectures have garnered immense interest lately du...
research
12/29/2022

On Transforming Reinforcement Learning by Transformer: The Development Trajectory

Transformer, originally devised for natural language processing, has als...
research
05/16/2023

Cooperation Is All You Need

Going beyond 'dendritic democracy', we introduce a 'democracy of local p...
research
04/08/2020

Adaptive Transformers in RL

Recent developments in Transformers have opened new interesting areas of...
research
12/30/2022

Transformer in Transformer as Backbone for Deep Reinforcement Learning

Designing better deep networks and better reinforcement learning (RL) al...
research
10/13/2019

Stabilizing Transformers for Reinforcement Learning

Owing to their ability to both effectively integrate information over lo...
research
10/08/2021

Pathologies in priors and inference for Bayesian transformers

In recent years, the transformer has established itself as a workhorse i...

Please sign up or login with your details

Forgot password? Click here to reset