Exact Asymptotics for Linear Quadratic Adaptive Control

11/02/2020
by   Feicheng Wang, et al.
0

Recent progress in reinforcement learning has led to remarkable performance in a range of applications, but its deployment in high-stakes settings remains quite rare. One reason is a limited understanding of the behavior of reinforcement algorithms, both in terms of their regret and their ability to learn the underlying system dynamics—existing work is focused almost exclusively on characterizing rates, with little attention paid to the constants multiplying those rates that can be critically important in practice. To start to address this challenge, we study perhaps the simplest non-bandit reinforcement learning problem: linear quadratic adaptive control (LQAC). By carefully combining recent finite-sample performance bounds for the LQAC problem with a particular (less-recent) martingale central limit theorem, we are able to derive asymptotically-exact expressions for the regret, estimation error, and prediction error of a rate-optimal stepwise-updating LQAC algorithm. In simulations on both stable and unstable systems, we find that our asymptotic theory also describes the algorithm's finite-sample behavior remarkably well.

READ FULL TEXT

page 1

page 2

page 3

page 4

03/27/2017

Thompson Sampling for Linear-Quadratic Control Problems

We consider the exploration-exploitation tradeoff in linear quadratic (L...
11/20/2017

Finite Time Analysis of Optimal Adaptive Policies for Linear-Quadratic Systems

We consider the classical problem of control of linear systems with quad...
03/18/2022

Finite-sample analysis of identification of switched linear systems with arbitrary or restricted switching

This work aims to derive a data-independent finite-sample error bound fo...
06/17/2022

Thompson Sampling Achieves Õ(√(T)) Regret in Linear Quadratic Control

Thompson Sampling (TS) is an efficient method for decision-making under ...
09/29/2012

Optimistic Agents are Asymptotically Optimal

We use optimism to introduce generic asymptotically optimal reinforcemen...
12/06/2018

Finite-Sample Analyses for Fully Decentralized Multi-Agent Reinforcement Learning

Despite the increasing interest in multi-agent reinforcement learning (M...
07/22/2018

Finite Time Adaptive Stabilization of LQ Systems

Stabilization of linear systems with unknown dynamics is a canonical pro...