
Value Function Approximation in ZeroSum Markov Games
This paper investigates value function approximation in the context of z...
read it

On Reinforcement Learning for Turnbased Zerosum Markov Games
We consider the problem of finding Nash equilibrium for twoplayer turn...
read it

NearOptimal Reinforcement Learning with SelfPlay
This paper considers the problem of designing optimal algorithms for rei...
read it

Learning Minimax Estimators via Online Learning
We consider the problem of designing minimax estimators for estimating t...
read it

A fixedpoint policyiterationtype algorithm for symmetric nonzerosum stochastic impulse games
Nonzerosum stochastic differential games with impulse controls offer a ...
read it

Game Efficiency through Linear Programming Duality
The efficiency of a game is typically quantified by the price of anarchy...
read it

Performance Analysis of Trial and Error Algorithms
Modelfree decentralized optimizations and learning are receiving increa...
read it
Learning ZeroSum SimultaneousMove Markov Games Using Function Approximation and Correlated Equilibrium
We develop provably efficient reinforcement learning algorithms for twoplayer zerosum Markov games in which the two players simultaneously take actions. To incorporate function approximation, we consider a family of Markov games where the reward function and transition kernel possess a linear structure. Both the offline and online settings of the problems are considered. In the offline setting, we control both players and the goal is to find the Nash Equilibrium efficiently by minimizing the worstcase duality gap. In the online setting, we control a single player to play against an arbitrary opponent and the goal is to minimize the regret. For both settings, we propose an optimistic variant of the leastsquares minimax value iteration algorithm. We show that our algorithm is computationally efficient and provably achieves an Õ(√(d^3 H^3 T)) upper bound on the duality gap and regret, without requiring additional assumptions on the sampling model. We highlight that our setting requires overcoming several new challenges that are absent in Markov decision processes or turnbased Markov games. In particular, to achieve optimism in simultaneousmove Marko games, we construct both upper and lower confidence bounds of the value function, and then compute the optimistic policy by solving a generalsum matrix game with these bounds as the payoff matrices. As finding the Nash Equilibrium of such a generalsum game is computationally hard, our algorithm instead solves for a Coarse Correlated Equilibrium (CCE), which can be obtained efficiently via linear programming. To our best knowledge, such a CCEbased scheme for implementing optimism has not appeared in the literature and might be of interest in its own right.
READ FULL TEXT
Comments
There are no comments yet.