Characterizing Policy Divergence for Personalized Meta-Reinforcement Learning

10/09/2020
by   Michael Zhang, et al.
0

Despite ample motivation from costly exploration and limited trajectory data, rapidly adapting to new environments with few-shot reinforcement learning (RL) can remain a challenging task, especially with respect to personalized settings. Here, we consider the problem of recommending optimal policies to a set of multiple entities each with potentially different characteristics, such that individual entities may parameterize distinct environments with unique transition dynamics. Inspired by existing literature in meta-learning, we extend previous work by focusing on the notion that certain environments are more similar to each other than others in personalized settings, and propose a model-free meta-learning algorithm that prioritizes past experiences by relevance during gradient-based adaptation. Our algorithm involves characterizing past policy divergence through methods in inverse reinforcement learning, and we illustrate how such metrics are able to effectively distinguish past policy parameters by the environment they were deployed in, leading to more effective fast adaptation during test time. To study personalization more effectively we introduce a navigation testbed to specifically incorporate environment diversity across training episodes, and demonstrate that our approach outperforms meta-learning alternatives with respect to few-shot reinforcement learning in personalized settings.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/04/2020

Meta-Model-Based Meta-Policy Optimization

Model-based reinforcement learning (MBRL) has been applied to meta-learn...
research
09/14/2021

Few-shot Quality-Diversity Optimisation

In the past few years, a considerable amount of research has been dedica...
research
03/04/2019

NoRML: No-Reward Meta Learning

Efficiently adapting to new environments and changes in dynamics is crit...
research
02/19/2020

Curriculum in Gradient-Based Meta-Reinforcement Learning

Gradient-based meta-learners such as Model-Agnostic Meta-Learning (MAML)...
research
08/13/2020

Offline Meta-Reinforcement Learning with Advantage Weighting

Massive datasets have proven critical to successfully applying deep lear...
research
11/16/2019

Six Degree-of-Freedom Hovering using LIDAR Altimetry via Reinforcement Meta-Learning

We optimize a six degrees of freedom hovering policy using reinforcement...
research
10/06/2022

Few-shot Generation of Personalized Neural Surrogates for Cardiac Simulation via Bayesian Meta-Learning

Clinical adoption of personalized virtual heart simulations faces challe...

Please sign up or login with your details

Forgot password? Click here to reset