-
Local Stochastic Approximation: A Unified View of Federated Learning and Distributed Multi-Task Reinforcement Learning Algorithms
Motivated by broad applications in reinforcement learning and federated ...
read it
-
Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability
Many real-world tasks involve multiple agents with partial observability...
read it
-
Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
We consider the problem of fully decentralized multi-agent reinforcement...
read it
-
Finite-Time Performance of Distributed Temporal Difference Learning with Linear Function Approximation
We study the policy evaluation problem in multi-agent reinforcement lear...
read it
-
Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization
Despite the success of single-agent reinforcement learning, multi-agent ...
read it
-
A Multi-Agents Architecture to Learn Vision Operators and their Parameters
In a vision system, every task needs that the operators to apply should ...
read it
-
Communication-Based Decomposition Mechanisms for Decentralized MDPs
Multi-agent planning in stochastic environments can be framed formally a...
read it
Finite-Time Analysis of Decentralized Stochastic Approximation with Applications in Multi-Agent and Multi-Task Learning
Stochastic approximation, a data-driven approach for finding the fixed point of an unknown operator, provides a unified framework for treating many problems in stochastic optimization and reinforcement learning. Motivated by a growing interest in multi-agent and multi-task learning, we consider in this paper a decentralized variant of stochastic approximation. A network of agents, each with their own unknown operator and data observations, cooperatively find the fixed point of the aggregate operator. The agents work by running a local stochastic approximation algorithm using noisy samples from their operators while averaging their iterates with their neighbors' on a decentralized communication graph. Our main contribution provides a finite-time analysis of this decentralized stochastic approximation algorithm and characterizes the impacts of the underlying communication topology between agents. Our model for the data observed at each agent is that it is sampled from a Markov processes; this lack of independence makes the iterates biased and (potentially) unbounded. Under mild assumptions on the Markov processes, we show that the convergence rate of the proposed methods is essentially the same as if the samples were independent, differing only by a log factor that represents the mixing time of the Markov process. We also present applications of the proposed method on a number of interesting learning problems in multi-agent systems, including a decentralized variant of Q-learning for solving multi-task reinforcement learning.
READ FULL TEXT
Comments
There are no comments yet.