Learning Two-Player Mixture Markov Games: Kernel Function Approximation and Correlated Equilibrium

08/10/2022
by   Chris Junchi Li, et al.
11

We consider learning Nash equilibria in two-player zero-sum Markov Games with nonlinear function approximation, where the action-value function is approximated by a function in a Reproducing Kernel Hilbert Space (RKHS). The key challenge is how to do exploration in the high-dimensional function space. We propose a novel online learning algorithm to find a Nash equilibrium by minimizing the duality gap. At the core of our algorithms are upper and lower confidence bounds that are derived based on the principle of optimism in the face of uncertainty. We prove that our algorithm is able to attain an O(√(T)) regret with polynomial computational complexity, under very mild assumptions on the reward function and the underlying dynamic of the Markov Games. We also propose several extensions of our algorithm, including an algorithm with Bernstein-type bonus that can achieve a tighter regret bound, and another algorithm for model misspecification that can be applied to neural function approximation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/17/2020

Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibrium

We develop provably efficient reinforcement learning algorithms for two-...
research
02/15/2021

Almost Optimal Algorithms for Two-player Markov Games with Linear Function Approximation

We study reinforcement learning for two-player zero-sum Markov games wit...
research
02/25/2020

On Reinforcement Learning for Turn-based Zero-sum Markov Games

We consider the problem of finding Nash equilibrium for two-player turn-...
research
11/28/2014

Solving Games with Functional Regret Estimation

We propose a novel online learning method for minimizing regret in large...
research
03/06/2021

Zero-Sum Semi-Markov Games with State-Action-Dependent Discount Factors

Semi-Markov model is one of the most general models for stochastic dynam...
research
07/30/2021

Towards General Function Approximation in Zero-Sum Markov Games

This paper considers two-player zero-sum finite-horizon Markov games wit...
research
10/21/2019

Semi-Decentralized Coordinated Online Learning for Continuous Games with Coupled Constraints via Augmented Lagrangian

We consider a class of concave continuous games in which the correspondi...

Please sign up or login with your details

Forgot password? Click here to reset