High-Dimensional Control Using Generalized Auxiliary Tasks

09/26/2019
by   Yannis Flet-Berliac, et al.
17

A long-standing challenge in reinforcement learning is the design of function approximations and efficient learning algorithms that provide agents with fast training, robust learning , and high performance in complex environments. To this end, the use of prior knowledge, while promising, is often costly and, in essence, challenging to scale up. In contrast, we consider problem knowledge signals, that are any relevant indicator useful to solve a task, e.g., metrics of uncertainty or proactive prediction of future states. Our framework consists of predicting such complementary quantities associated with self-performance assessment and accurate expectations. Therefore, policy and value functions are no longer only optimized for a reward but are learned using environment-agnostic quantities. We propose a generally applicable framework for structuring reinforcement learning by injecting problem knowledge in policy gradient updates. In this paper: (a) We introduce MERL, our multi-head reinforcement learning framework for generalized auxiliary tasks. (b) We conduct experiments across a variety of standard benchmark environments. Our results show that MERL improves performance for on-and off-policy methods. (c) We show that MERL also improves transfer learning on a set of challenging tasks. (d) We investigate how our approach addresses the problem of reward sparsity and pushes the function approximations into a better-constrained parameter configuration.

READ FULL TEXT

page 5

page 6

page 9

page 10

page 14

research
02/28/2023

Auxiliary Task-based Deep Reinforcement Learning for Quantum Control

Due to its property of not requiring prior knowledge of the environment,...
research
05/25/2021

Transfer Learning and Curriculum Learning in Sokoban

Transfer learning can speed up training in machine learning and is regul...
research
12/10/2021

Encoding priors in the brain: a reinforcement learning model for mouse decision making

In two-alternative forced choice tasks, prior knowledge can improve perf...
research
05/31/2016

Information Theoretically Aided Reinforcement Learning for Embodied Agents

Reinforcement learning for embodied agents is a challenging problem. The...
research
06/16/2016

Successor Features for Transfer in Reinforcement Learning

Transfer in reinforcement learning refers to the notion that generalizat...
research
10/10/2022

Long N-step Surrogate Stage Reward to Reduce Variances of Deep Reinforcement Learning in Complex Problems

High variances in reinforcement learning have shown impeding successful ...
research
04/27/2023

Discovering Object-Centric Generalized Value Functions From Pixels

Deep Reinforcement Learning has shown significant progress in extracting...

Please sign up or login with your details

Forgot password? Click here to reset