DeepAI AI Chat
Log In Sign Up

Model-Free Learning for Two-Player Zero-Sum Partially Observable Markov Games with Perfect Recall

by   Tadashi Kozuno, et al.

We study the problem of learning a Nash equilibrium (NE) in an imperfect information game (IIG) through self-play. Precisely, we focus on two-player, zero-sum, episodic, tabular IIG under the perfect-recall assumption where the only feedback is realizations of the game (bandit feedback). In particular, the dynamic of the IIG is not known – we can only access it by sampling or interacting with a game simulator. For this learning setting, we provide the Implicit Exploration Online Mirror Descent (IXOMD) algorithm. It is a model-free algorithm with a high-probability bound on the convergence rate to the NE of order 1/√(T) where T is the number of played games. Moreover, IXOMD is computationally efficient as it needs to perform the updates only along the sampled trajectory.


page 1

page 2

page 3

page 4


DREAM: Deep Regret minimization with Advantage baselines and Model-free learning

We introduce DREAM, a deep reinforcement learning algorithm that finds o...

Uncoupled and Convergent Learning in Two-Player Zero-Sum Markov Games

We revisit the problem of learning in two-player zero-sum Markov games, ...

Adapting to game trees in zero-sum imperfect information games

Imperfect information games (IIG) are games in which each player only pa...

Constructing Imperfect Recall Abstractions to Solve Large Extensive-Form Games

Extensive-form games are an important model of finite sequential interac...

Almost Optimal Algorithms for Two-player Markov Games with Linear Function Approximation

We study reinforcement learning for two-player zero-sum Markov games wit...

Offline congestion games: How feedback type affects data coverage requirement

This paper investigates when one can efficiently recover an approximate ...

Fictitious Play: Convergence, Smoothness, and Optimism

We consider the dynamics of two-player zero-sum games, with the goal of ...