Finding mixed-strategy equilibria of continuous-action games without gradients using randomized policy networks

11/29/2022
by   Carlos Martin, et al.
0

We study the problem of computing an approximate Nash equilibrium of continuous-action game without access to gradients. Such game access is common in reinforcement learning settings, where the environment is typically treated as a black box. To tackle this problem, we apply zeroth-order optimization techniques that combine smoothed gradient estimators with equilibrium-finding dynamics. We model players' strategies using artificial neural networks. In particular, we use randomized policy networks to model mixed strategies. These take noise in addition to an observation as input and can flexibly represent arbitrary observation-dependent, continuous-action distributions. Being able to model such mixed strategies is crucial for tackling continuous-action games that lack pure-strategy equilibria. We evaluate the performance of our method using an approximation of the Nash convergence metric from game theory, which measures how much players can benefit from unilaterally changing their strategy. We apply our method to continuous Colonel Blotto games, single-item and multi-item auctions, and a visibility game. The experiments show that our method can quickly find high-quality approximate equilibria. Furthermore, they show that the dimensionality of the input noise is crucial for performance. To our knowledge, this paper is the first to solve general continuous-action games with unrestricted mixed strategies and without any gradient information.

READ FULL TEXT

page 8

page 14

research
11/07/2020

Stability of Gradient Learning Dynamics in Continuous Games: Scalar Action Spaces

Learning processes in games explain how players grapple with one another...
research
06/12/2020

Algorithm for Computing Approximate Nash equilibrium in Continuous Games with Application to Continuous Blotto

Successful algorithms have been developed for computing Nash equilibrium...
research
01/28/2021

Equilibrium Learning in Combinatorial Auctions: Computing Approximate Bayesian Nash Equilibria via Pseudogradient Dynamics

Applications of combinatorial auctions (CA) as market mechanisms are pre...
research
11/02/2022

An Exponentially Converging Particle Method for the Mixed Nash Equilibrium of Continuous Games

We consider the problem of computing mixed Nash equilibria of two-player...
research
06/10/2020

Robust Spammer Detection by Nash Reinforcement Learning

Online reviews provide product evaluations for customers to make decisio...
research
06/10/2020

Robust Detection of Adaptive Spammers by Nash Reinforcement Learning

Online reviews provide product evaluations for customers to make decisio...
research
04/20/2020

Black-Box Strategies and Equilibrium for Games with Cumulative Prospect Theoretic Players

The betweenness property of preference relations states that a probabili...

Please sign up or login with your details

Forgot password? Click here to reset