Meta-Reinforcement Learning by Tracking Task Non-stationarity

05/18/2021
by   Riccardo Poiani, et al.
0

Many real-world domains are subject to a structured non-stationarity which affects the agent's goals and the environmental dynamics. Meta-reinforcement learning (RL) has been shown successful for training agents that quickly adapt to related tasks. However, most of the existing meta-RL algorithms for non-stationary domains either make strong assumptions on the task generation process or require sampling from it at training time. In this paper, we propose a novel algorithm (TRIO) that optimizes for the future by explicitly tracking the task evolution through time. At training time, TRIO learns a variational module to quickly identify latent parameters from experience samples. This module is learned jointly with an optimal exploration policy that takes task uncertainty into account. At test time, TRIO tracks the evolution of the latent parameters online, hence reducing the uncertainty over future tasks and obtaining fast adaptation through the meta-learned policy. Unlike most existing methods, TRIO does not assume Markovian task-evolution processes, it does not require information about the non-stationarity at training time, and it captures complex changes undergoing in the environment. We evaluate our algorithm on different simulated problems and show it outperforms competitive baselines.

READ FULL TEXT

page 6

page 14

research
04/29/2023

Meta-Reinforcement Learning Based on Self-Supervised Task Representation Learning

Meta-reinforcement learning enables artificial agents to learn from rela...
research
10/17/2019

Single Episode Policy Transfer in Reinforcement Learning

Transfer and adaptation to new unknown environmental dynamics is a key c...
research
05/16/2019

Meta Reinforcement Learning with Task Embedding and Shared Policy

Despite significant progress, deep reinforcement learning (RL) suffers f...
research
02/20/2023

Meta-World Conditional Neural Processes

We propose Meta-World Conditional Neural Processes (MW-CNP), a condition...
research
05/20/2023

On First-Order Meta-Reinforcement Learning with Moreau Envelopes

Meta-Reinforcement Learning (MRL) is a promising framework for training ...
research
05/06/2020

Learning Adaptive Exploration Strategies in Dynamic Environments Through Informed Policy Regularization

We study the problem of learning exploration-exploitation strategies tha...
research
10/29/2022

BIMRL: Brain Inspired Meta Reinforcement Learning

Sample efficiency has been a key issue in reinforcement learning (RL). A...

Please sign up or login with your details

Forgot password? Click here to reset