Model-Free Optimal Control of Linear Multi-Agent Systems via Decomposition and Hierarchical Approximation

by   Gangshan Jing, et al.

Designing the optimal linear quadratic regulator (LQR) for a large-scale multi-agent system (MAS) is time-consuming since it involves solving a large-size matrix Riccati equation. The situation is further exasperated when the design needs to be done in a model-free way using schemes such as reinforcement learning (RL). To reduce this computational complexity, we decompose the large-scale LQR design problem into multiple sets of smaller-size LQR design problems. We consider the objective function to be specified over an undirected graph, and cast the decomposition as a graph clustering problem. The graph is decomposed into two parts, one consisting of multiple decoupled subgroups of connected components, and the other containing edges that connect the different subgroups. Accordingly, the resulting controller has a hierarchical structure, consisting of two components. The first component optimizes the performance of each decoupled subgroup by solving the smaller-size LQR design problem in a model-free way using an RL algorithm. The second component accounts for the objective coupling different subgroups, which is achieved by solving a least squares problem in one shot. Although suboptimal, the hierarchical controller adheres to a particular structure as specified by the inter-agent coupling in the objective function and by the decomposition strategy. Mathematical formulations are established to find a decomposition that minimizes required communication links or reduces the optimality gap. Numerical simulations are provided to highlight the pros and cons of the proposed designs.


Hierarchical Reinforcement Learning for Optimal Control of Linear Multi-Agent Systems: the Homogeneous Case

Individual agents in a multi-agent system (MAS) may have decoupled open-...

Reduced-Dimensional Reinforcement Learning Control using Singular Perturbation Approximations

We present a set of model-free, reduced-dimensional reinforcement learni...

Model-free two-step design for improving transient learning performance in nonlinear optimal regulator problems

Reinforcement learning (RL) provides a model-free approach to designing ...

A Reinforcement Learning Based Approach for Joint Multi-Agent Decision Making

Reinforcement Learning (RL) is being increasingly applied to optimize co...

On Multi-Agent Learning in Team Sports Games

In recent years, reinforcement learning has been successful in solving v...

High efficiency rl agent

Now a day, model free algorithm achieve state of art performance on many...

Methodology of Designing Multi-agent Robot Control Systems Utilising Hierarchical Petri Nets

A robot system is designed as a set of embodied agents. An embodied agen...

Please sign up or login with your details

Forgot password? Click here to reset