Student-Initiated Action Advising via Advice Novelty

10/01/2020
by   Ercüment İlhan, et al.
3

Action advising is a knowledge exchange mechanism between peers, namely student and teacher, that can help tackle exploration and sample inefficiency problems in deep reinforcement learning. Due to the practical limitations in peer-to-peer communication and the negative implications of over-advising, the peer responsible for initiating these interactions needs to do so only when it's most adequate to exchange advice. Most recently, student-initiated techniques that utilise state novelty and uncertainty estimations have obtained promising results. However, these estimations have several weaknesses, such as having no information regarding the characteristics of convergence and being subject to delays that occur in the presence of experience replay dynamics. We propose a student-initiated action advising algorithm that alleviates these shortcomings. Specifically, we employ Random Network Distillation (RND) to measure the novelty of an advice, for the student to determine whether to proceed with the request; furthermore, we perform RND updates only for the advised states to ensure that the student's convergence will not prevent it from utilising the teacher's knowledge at any stage of learning. Experiments in GridWorld and simplified versions of five Atari games show that our approach can perform on par with the state-of-the-art and demonstrate significant advantages in the scenarios where the existing methods are prone to fail.

READ FULL TEXT

page 1

page 4

page 7

research
04/17/2021

Action Advising with Advice Imitation in Deep Reinforcement Learning

Action advising is a peer-to-peer knowledge exchange technique built on ...
research
04/19/2019

Teaching on a Budget in Multi-Agent Deep Reinforcement Learning

Deep Reinforcement Learning algorithms can solve complex sequential deci...
research
04/17/2021

Learning on a Budget via Teacher Imitation

Deep Reinforcement Learning (RL) techniques can benefit greatly from lev...
research
11/15/2022

Explainable Action Advising for Multi-Agent Reinforcement Learning

Action advising is a knowledge transfer technique for reinforcement lear...
research
04/14/2022

Methodical Advice Collection and Reuse in Deep Reinforcement Learning

Reinforcement learning (RL) has shown great success in solving many chal...
research
12/09/2020

Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation

In reinforcement learning, domain randomisation is an increasingly popul...

Please sign up or login with your details

Forgot password? Click here to reset