MRAC-RL: A Framework for On-Line Policy Adaptation Under Parametric Model Uncertainty

11/20/2020
by   Anubhav Guha, et al.
0

Reinforcement learning (RL) algorithms have been successfully used to develop control policies for dynamical systems. For many such systems, these policies are trained in a simulated environment. Due to discrepancies between the simulated model and the true system dynamics, RL trained policies often fail to generalize and adapt appropriately when deployed in the real-world environment. Current research in bridging this sim-to-real gap has largely focused on improvements in simulation design and on the development of improved and specialized RL algorithms for robust control policy generation. In this paper we apply principles from adaptive control and system identification to develop the model-reference adaptive control reinforcement learning (MRAC-RL) framework. We propose a set of novel MRAC algorithms applicable to a broad range of linear and nonlinear systems, and derive the associated control laws. The MRAC-RL framework utilizes an inner-loop adaptive controller that allows a simulation-trained outer-loop policy to adapt and operate effectively in a test environment, even when parametric model uncertainty exists. We demonstrate that the MRAC-RL approach improves upon state-of-the-art RL algorithms in developing control policies that can be applied to systems with modeling errors.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/18/2020

A Framework for Studying Reinforcement Learning and Sim-to-Real in Robot Soccer

This article introduces an open framework, called VSSS-RL, for studying ...
research
02/14/2022

QuadSim: A Quadcopter Rotational Dynamics Simulation Framework For Reinforcement Learning Algorithms

This study focuses on designing and developing a mathematically based qu...
research
01/30/2023

Passivizing learned policies and learning passive policies with virtual energy tanks in robotics

Within a robotic context, we merge the techniques of passivity-based con...
research
12/03/2021

Improving the Robustness of Reinforcement Learning Policies with ℒ_1 Adaptive Control

A reinforcement learning (RL) control policy trained in a nominal enviro...
research
07/23/2023

Uncertainty-aware Grounded Action Transformation towards Sim-to-Real Transfer for Traffic Signal Control

Traffic signal control (TSC) is a complex and important task that affect...
research
01/14/2022

Reinforcement Learning in Time-Varying Systems: an Empirical Study

Recent research has turned to Reinforcement Learning (RL) to solve chall...
research
07/01/2021

Model Mediated Teleoperation with a Hand-Arm Exoskeleton in Long Time Delays Using Reinforcement Learning

Telerobotic systems must adapt to new environmental conditions and deal ...

Please sign up or login with your details

Forgot password? Click here to reset