Log In Sign Up

Minimal Expected Regret in Linear Quadratic Control

by   Yassir Jedra, et al.

We consider the problem of online learning in Linear Quadratic Control systems whose state transition and state-action transition matrices A and B may be initially unknown. We devise an online learning algorithm and provide guarantees on its expected regret. This regret at time T is upper bounded (i) by O((d_u+d_x)√(d_xT)) when A and B are unknown, (ii) by O(d_x^2log(T)) if only A is unknown, and (iii) by O(d_x(d_u+d_x)log(T)) if only B is unknown and under some mild non-degeneracy condition (d_x and d_u denote the dimensions of the state and of the control input, respectively). These regret scalings are minimal in T, d_x and d_u as they match existing lower bounds in scenario (i) when d_x≤ d_u [SF20], and in scenario (ii) [lai1986]. We conjecture that our upper bounds are also optimal in scenario (iii) (there is no known lower bound in this setting). Existing online algorithms proceed in epochs of (typically exponentially) growing durations. The control policy is fixed within each epoch, which considerably simplifies the analysis of the estimation error on A and B and hence of the regret. Our algorithm departs from this design choice: it is a simple variant of certainty-equivalence regulators, where the estimates of A and B and the resulting control policy can be updated as frequently as we wish, possibly at every step. Quantifying the impact of such a constantly-varying control policy on the performance of these estimates and on the regret constitutes one of the technical challenges tackled in this paper.


page 1

page 2

page 3

page 4


Logarithmic Regret for Learning Linear Quadratic Regulators Efficiently

We consider the problem of learning in Linear Quadratic Control systems ...

Naive Exploration is Optimal for Online LQR

We consider the problem of online adaptive control of the linear quadrat...

Best-Case Lower Bounds in Online Learning

Much of the work in online learning focuses on the study of sublinear up...

Learning to Control Linear Systems can be Hard

In this paper, we study the statistical difficulty of learning to contro...

Online Linear Quadratic Control

We study the problem of controlling linear time-invariant systems with k...

Regret Bounds for Learning Decentralized Linear Quadratic Regulator with Partially Nested Information Structure

We study the problem of learning decentralized linear quadratic regulato...

Regret Lower Bounds for Learning Linear Quadratic Gaussian Systems

This paper presents local minimax regret lower bounds for adaptively con...