Maximum entropy optimal density control of discrete-time linear systems and Schrödinger bridges

by   Kaito Ito, et al.

We consider an entropy-regularized version of optimal density control of deterministic discrete-time linear systems. Entropy regularization, or a maximum entropy (MaxEnt) method for optimal control has attracted much attention especially in reinforcement learning due to its many advantages such as a natural exploration strategy. Despite the merits, high-entropy control policies introduce probabilistic uncertainty into systems, which severely limits the applicability of MaxEnt optimal control to safety-critical systems. To remedy this situation, we impose a Gaussian density constraint at a specified time on the MaxEnt optimal control to directly control state uncertainty. Specifically, we derive the explicit form of the MaxEnt optimal density control. In addition, we also consider the case where a density constraint is replaced by a fixed point constraint. Then, we characterize the associated state process as a pinned process, which is a generalization of the Brownian bridge to linear systems. Finally, we reveal that the MaxEnt optimal density control induces the so-called Schrödinger bridge associated to a discrete-time linear system.


Stochastic Optimal Control as Approximate Input Inference

Optimal control of stochastic nonlinear dynamical systems is a major cha...

An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks

Deep learning is formulated as a discrete-time optimal control problem. ...

Continuous Random Variable Estimation is not Optimal for the Witsenhausen Counterexample

Optimal design of distributed decision policies can be a difficult task,...

COSMIC: fast closed-form identification from large-scale data for LTV systems

We introduce a closed-form method for identification of discrete-time li...

Algorithms for Optimal Control with Fixed-Rate Feedback

We consider a discrete-time linear quadratic Gaussian networked control ...

Optimal Control as Variational Inference

In this article we address the stochastic and risk sensitive optimal con...

A Projection Approach to Equality Constrained Iterative Linear Quadratic Optimal Control

This paper presents a state and state-input constrained variant of the d...