Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualization

by   Tiantian Zhang, et al.

A key challenge of continual reinforcement learning (CRL) in dynamic environments is to promptly adapt the RL agent's behavior as the environment changes over its lifetime, while minimizing the catastrophic forgetting of the learned information. To address this challenge, in this article, we propose DaCoRL, i.e., dynamics-adaptive continual RL. DaCoRL learns a context-conditioned policy using progressive contextualization, which incrementally clusters a stream of stationary tasks in the dynamic environment into a series of contexts and opts for an expandable multihead neural network to approximate the policy. Specifically, we define a set of tasks with similar dynamics as an environmental context and formalize context inference as a procedure of online Bayesian infinite Gaussian mixture clustering on environment features, resorting to online Bayesian inference to infer the posterior distribution over contexts. Under the assumption of a Chinese restaurant process prior, this technique can accurately classify the current task as a previously seen context or instantiate a new context as needed without relying on any external indicator to signal environmental changes in advance. Furthermore, we employ an expandable multihead neural network whose output layer is synchronously expanded with the newly instantiated context, and a knowledge distillation regularization term for retaining the performance on learned tasks. As a general framework that can be coupled with various deep RL algorithms, DaCoRL features consistent superiority over existing methods in terms of the stability, overall performance and generalization ability, as verified by extensive experiments on several robot navigation and MuJoCo locomotion tasks.


page 1

page 9

page 15


Lifelong Incremental Reinforcement Learning with Online Bayesian Inference

A central capability of a long-lived reinforcement learning (RL) agent i...

Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation

The powerful learning ability of deep neural networks enables reinforcem...

A Dirichlet Process Mixture of Robust Task Models for Scalable Lifelong Reinforcement Learning

While reinforcement learning (RL) algorithms are achieving state-of-the-...

Reinforcement Learning in Presence of Discrete Markovian Context Evolution

We consider a context-dependent Reinforcement Learning (RL) setting, whi...

An Adaptive Deep RL Method for Non-Stationary Environments with Piecewise Stable Context

One of the key challenges in deploying RL to real-world applications is ...

The Hierarchical Adaptive Forgetting Variational Filter

A common problem in Machine Learning and statistics consists in detectin...

Prototypical context-aware dynamics generalization for high-dimensional model-based reinforcement learning

The latent world model provides a promising way to learn policies in a c...

Please sign up or login with your details

Forgot password? Click here to reset