Naive Exploration is Optimal for Online LQR

by   Max Simchowitz, et al.

We consider the problem of online adaptive control of the linear quadratic regulator, where the true system parameters are unknown. We prove new upper and lower bounds demonstrating that the optimal regret scales as Θ(√(d_u^2 d_x T)), where T is the number of time steps, d_u is the dimension of the input space, and d_x is the dimension of the system state. Notably, our lower bounds rule out the possibility of a poly(logT)-regret algorithm, which has been conjectured due to the apparent strong convexity of the problem. Our upper bounds are attained by a simple variant of certainty equivalence control, where the learner selects control inputs according to the optimal controller for their estimate of the system while injecting exploratory random noise. While this approach was shown to achieve √(T)-regret by Mania et al. 2019, we show that if the learner continually refines their estimates of the system matrices, the method attains optimal dimension dependence as well. Central to our upper and lower bounds is a new approach for controlling perturbations of Ricatti equations, which we call the self-bounding ODE method. The approach enables regret upper bounds which hold for any stabilizable instance, require no foreknowledge of the system except for a single stabilizing controller, and scale with natural control-theoretic quantities.


page 1

page 2

page 3

page 4


Minimal Expected Regret in Linear Quadratic Control

We consider the problem of online learning in Linear Quadratic Control s...

Learning to Control Linear Systems can be Hard

In this paper, we study the statistical difficulty of learning to contro...

Towards a Dimension-Free Understanding of Adaptive Linear Control

We study the problem of adaptive control of the linear quadratic regulat...

Learning in the Presence of Corruption

In supervised learning one wishes to identify a pattern present in a joi...

Making Non-Stochastic Control (Almost) as Easy as Stochastic

Recent literature has made much progress in understanding online LQR: a ...

Logistic Regression Regret: What's the Catch?

We address the problem of the achievable regret rates with online logist...

On Uninformative Optimal Policies in Adaptive LQR with Unknown B-Matrix

This paper presents local asymptotic minimax regret lower bounds for ada...

Please sign up or login with your details

Forgot password? Click here to reset