On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learning

05/04/2021
by   Marc Aurel Vischer, et al.
12

The lottery ticket hypothesis questions the role of overparameterization in supervised deep learning. But how does the distributional shift inherent to the reinforcement learning problem affect the performance of winning lottery tickets? In this work, we show that feed-forward networks trained via supervised policy distillation and reinforcement learning can be pruned to the same level of sparsity. Furthermore, we establish the existence of winning tickets for both on- and off-policy methods in a visual navigation and classic control task. Using a set of carefully designed baseline conditions, we find that the majority of the lottery ticket effect in reinforcement learning can be attributed to the identified mask. The resulting masked observation space eliminates redundant information and yields minimal task-relevant representations. The mask identified by iterative magnitude pruning provides an interpretable inductive bias. Its costly generation can be amortized by training dense agents with low-dimensional input and thereby at lower computational cost.

READ FULL TEXT

page 2

page 6

page 12

page 13

research
07/04/2021

Low-Dimensional State and Action Representation Learning with MDP Homomorphism Metrics

Deep Reinforcement Learning has shown its ability in solving complicated...
research
12/06/2018

Deep Reinforcement Learning and the Deadly Triad

We know from reinforcement learning theory that temporal difference lear...
research
06/06/2022

Real2Sim or Sim2Real: Robotics Visual Insertion using Deep Reinforcement Learning and Real2Sim Policy Adaptation

Reinforcement learning has shown a wide usage in robotics tasks, such as...
research
03/11/2022

Graph Neural Networks for Relational Inductive Bias in Vision-based Deep Reinforcement Learning of Robot Control

State-of-the-art reinforcement learning algorithms predominantly learn a...
research
01/15/2021

Affordance-based Reinforcement Learning for Urban Driving

Traditional autonomous vehicle pipelines that follow a modular approach ...
research
03/31/2022

Mask Atari for Deep Reinforcement Learning as POMDP Benchmarks

We present Mask Atari, a new benchmark to help solve partially observabl...
research
10/06/2022

Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?

Modern deep learning involves training costly, highly overparameterized ...

Please sign up or login with your details

Forgot password? Click here to reset