Exploiting Anonymity in Approximate Linear Programming: Scaling to Large Multiagent MDPs (Extended Version)

11/29/2015
by   Philipp Robbel, et al.
0

Many exact and approximate solution methods for Markov Decision Processes (MDPs) attempt to exploit structure in the problem and are based on factorization of the value function. Especially multiagent settings, however, are known to suffer from an exponential increase in value component sizes as interactions become denser, meaning that approximation architectures are restricted in the problem sizes and types they can handle. We present an approach to mitigate this limitation for certain types of multiagent systems, exploiting a property that can be thought of as "anonymous influence" in the factored MDP. Anonymous influence summarizes joint variable effects efficiently whenever the explicit representation of variable identity in the problem can be avoided. We show how representational benefits from anonymity translate into computational efficiencies, both for general variable elimination in a factor graph but in particular also for the approximate linear programming solution to factored MDPs. The latter allows to scale linear programming to factored MDPs that were previously unsolvable. Our results are shown for the control of a stochastic disease process over a densely connected graph with 50 nodes and 25 agents.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/13/2012

Partitioned Linear Programming Approximations for MDPs

Approximate linear programming (ALP) is an efficient approach to solving...
research
06/09/2011

Efficient Solution Algorithms for Factored MDPs

This paper addresses the problem of planning under uncertainty in large ...
research
06/12/2022

Geometric Policy Iteration for Markov Decision Processes

Recently discovered polyhedral structures of the value function for fini...
research
07/04/2012

Approximate Linear Programming for First-order MDPs

We introduce a new approximate solution technique for first-order Markov...
research
06/26/2013

Scaling Up Robust MDPs by Reinforcement Learning

We consider large-scale Markov decision processes (MDPs) with parameter ...
research
04/16/2014

An Analysis of State-Relevance Weights and Sampling Distributions on L1-Regularized Approximate Linear Programming Approximation Accuracy

Recent interest in the use of L_1 regularization in the use of value fun...
research
12/30/2022

Linear programming word problems formulation using EnsembleCRF NER labeler and T5 text generator with data augmentations

We propose an ensemble approach to predict the labels in linear programm...

Please sign up or login with your details

Forgot password? Click here to reset