Hierarchical Reinforcement Learning for Optimal Control of Linear Multi-Agent Systems: the Homogeneous Case

by   Gangshan Jing, et al.

Individual agents in a multi-agent system (MAS) may have decoupled open-loop dynamics, but a cooperative control objective usually results in coupled closed-loop dynamics thereby making the control design computationally expensive. The computation time becomes even higher when a learning strategy such as reinforcement learning (RL) needs to be applied to deal with the situation when the agents dynamics are not known. To resolve this problem, this paper proposes a hierarchical RL scheme for a linear quadratic regulator (LQR) design in a continuous-time linear MAS. The idea is to exploit the structural properties of two graphs embedded in the Q and R weighting matrices in the LQR objective to define an orthogonal transformation that can convert the original LQR design to multiple decoupled smaller-sized LQR designs. We show that if the MAS is homogeneous then this decomposition retains closed-loop optimality. Conditions for decomposability, an algorithm for constructing the transformation matrix, a hierarchical RL algorithm, and robustness analysis when the design is applied to non-homogeneous MAS are presented. Simulations show that the proposed approach can guarantee significant speed-up in learning without any loss in the cumulative value of the LQR cost.



There are no comments yet.


page 1


Reduced-Dimensional Reinforcement Learning Control using Singular Perturbation Approximations

We present a set of model-free, reduced-dimensional reinforcement learni...

Model-Free Optimal Control of Linear Multi-Agent Systems via Decomposition and Hierarchical Approximation

Designing the optimal linear quadratic regulator (LQR) for a large-scale...

KnowSR: Knowledge Sharing among Homogeneous Agents in Multi-agent Reinforcement Learning

Recently, deep reinforcement learning (RL) algorithms have made great pr...

Optimal Distributed Control of Multi-agent Systems in Contested Environments via Reinforcement Learning

This paper presents a model-free reinforcement learning (RL) based distr...

Hybrid Information-driven Multi-agent Reinforcement Learning

Information theoretic sensor management approaches are an ideal solution...

Reinforcement Learning of Structured Control for Linear Systems with Unknown State Matrix

This paper delves into designing stabilizing feedback control gains for ...

A statistical learning strategy for closed-loop control of fluid flows

This work discusses a closed-loop control strategy for complex systems u...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.