HSVI fo zs-POSGs using Concavity, Convexity and Lipschitz Properties

10/25/2021
by   Aurélien Delage, et al.
0

Dynamic programming and heuristic search are at the core of state-of-the-art solvers for sequential decision-making problems. In partially observable or collaborative settings (, POMDPs and Dec-POMDPs), this requires introducing an appropriate statistic that induces a fully observable problem as well as bounding (convex) approximators of the optimal value function. This approach has succeeded in some subclasses of 2-player zero-sum partially observable stochastic games (zs-POSGs) as well, but failed in the general case despite known concavity and convexity properties, which only led to heuristic algorithms with poor convergence guarantees. We overcome this issue, leveraging on these properties to derive bounding approximators and efficient update and selection operators, before deriving a prototypical solver inspired by HSVI that provably converges to an ϵ-optimal solution in finite time, and which we empirically evaluate. This opens the door to a novel family of promising approaches complementing those relying on linear programming or iterative methods.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 15

06/29/2020

On Bellman's Optimality Principle for zs-POSGs

Many non-trivial sequential decision-making problems are efficiently sol...
06/22/2016

Structure in the Value Function of Two-Player Zero-Sum Games of Incomplete Information

Zero-sum stochastic games provide a rich model for competitive decision ...
02/06/2013

Incremental Pruning: A Simple, Fast, Exact Method for Partially Observable Markov Decision Processes

Most exact algorithms for general partially observable Markov decision p...
02/26/2019

Information Gathering in Decentralized POMDPs by Policy Graph Improvement

Decentralized policies for information gathering are required when multi...
12/31/2019

Numerical approximation of the value of a stochastic differential game with asymmetric information

We consider a convexity constrained Hamilton-Jacobi-Bellman-type obstacl...
01/16/2014

An Investigation into Mathematical Programming for Finite Horizon Decentralized POMDPs

Decentralized planning in uncertain environments is a complex task gener...
04/26/2021

Universal Off-Policy Evaluation

When faced with sequential decision-making problems, it is often useful ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.