Geometrical Regret Matching of Mixed Strategies

I Introduction

In 2000 Hart and Mas-Colell proposed an iterative algorithm called regret matching to approximate an equilibrium Hart2000; Hart2001. The players keep tracking the regrets of their own past plays and make the future plays with probabilities proportional to positive regret measures. This algorithm is particularly natural in that the players don’t have to know their opponents’ payoff functions, as opposed to the non-adaptive variety, e.g. the celebrated Lemke-Howson algorithm Lemke1964; Lemke1965 which takes two players’ payoff matrices as input and pinpoints equilibrium points as output. In other words, regret matching allows the players to reach equilibrium without strict mediation from beyond themselves. The concept of regret matching has since been adopted and developed by other algorithms cfr2008.

Iterative regret matching can be seen as continuous updating of mixed strategy with regret information: the mixed strategy to update is statistical structure of the whole past plays, and the mixed strategy updated will determine the probabilities of plays in the immediate future. Regret matching is essentially a function between them. In general the existing regret-matching functions update the mixed strategy proportional to positive regret measures, meaning that each matching is a “strategy jump” and the past mixed strategy has little relevance except for it being used for regret evaluation. In this paper, we try to propose a “smoother” regret matching with more respect to the past plays. Partly inspired by the mapping $T$ in Nash’s existential proof of equilibrium point Nash1951, our function for regret matching takes the form of

s_{i}^{'} = \frac{s_{i} + r_{i} R_{i, S} (s_{i})}{1 + r_{i} ∣ ∣ R_{i, S} (s_{i}) ∣ ∣}, % where r_{i} > 0.

As shown in FIG 1

our regret-matching function can be interpreted geometrically if we treat mixed strategies and regrets on pure strategies both as vectors: each regret matching will “push” mixed strategy vector towards regret vector by a small angle, and the smaller

r_{i}

is the “smoother” matching will be. Other than the smoothness for strategy updating, more importantly our regret matching agrees well with the behavioral tendency of immediate suppressing “unprofitable” pure strategies, which makes the matching a natural one. Our regret matching turns out to have clear analytical results for regret minimization. The test results in FIG 2, 3 and 4 show that, the matching delivers strong numerical convergence to Nash equilibrium both for two-person and many-person non-cooperative games. And as shown in FIG 5, the convergence is independent of the initial mixed strategies to start game with.

Figure 1: An example for the geometrical relation of mixed strategy $s_{i}$ , regret vector $R_{i} (s_{i})$ and regret-matched $s_{i}^{'}$ . The dotted line represents the simplex in $R^{2}$ . In this example regret vector $R_{i} (s_{i})$ must be to the direction of one vertex since the other one is “less profitable than average” such that the corresponding regret component in vector $R_{i} (s_{i})$ is zero.

Figure 2: The pair of “strategy paths” on the simplex of $R^{3}$ for a two-person game. The game has the code name 3X3-1eq2sp to refer to, indicating that the game has one single equilibrium point and either player can use three pure strategies.

Figure 3: The pair of “strategy paths” for a two-person game where two players use $60$ and $40$ pure strategies respectively. The $60$ or $40$ dimensions of “strategy path” data are reduced with PCA to three dimensions for the visualization in $R^{3}$ .

Figure 4: The “strategy paths” for a six-person game, where each player can use three pure strategies.

Figure 5: The equilibrium point of game 3X3-1eq2sp is an “attractor” for all possible “strategy paths”.

The remainder of paper goes as follows. In Section II, we vectorize the key concepts of non-cooperative game as well as regret measures in preparation of Section III, where our regret matching is elaborated and applied directly upon mixed strategies so as to approximate a Nash equilibrium point. In Section IV, the approximation algorithm is verified with the examples both of two-person game and general $n$ -person game, and their “strategy paths" are visualized as shown in FIG 2–5. In Section V, we address the accuracy issues of equilibrium approximation. In Section VI, we consider some generalizations for our theory.

Ii Non-cooperative game and regret

In this section, we will define the condition of Nash equilibrium as well as the regret measures in the form of vector and its $L 1$ -norm.

First of all, let us define the familiar concepts of non-cooperative game and set up their notations.

There are $n$ players, and each player $i$ has $g_{i}$ pure strategies. The $i$ ^th player’s $j$ ^th pure strategy is denoted by $π_{i j}$ .
The mixed strategy $s_{i}$ of each player $i$ is a point of a probability simplex $I_{i}$ , which is a compact and convex subset of the real vector space $R^{g_{i}}$ of $g_{i}$ dimensions. That is, $s_{i} \in I_{i} \subset R^{g_{i}}$ .
Each player $i$ has a payoff function ${¯ p}_{i} : S \to R$ . Any $S \in S$ is an $n$ -tuple of mixed strategies, e.g. $(s_{1}, \dots, s_{i}, \dots, s_{n})$ in which each item $s_{i}$ is associated with the $i$ ^th player such that $s_{i} \in I_{i}$ . Therefore, $S$ is the product of all $n$ players’ simplices, i.e. $I_{1} \times I_{2} \times \dots \times I_{n}$ .
Any player $i$ can, from the given $S \in S$ , form a new $n$ -tuple of mixed strategies, denoted by $s_{i}; S \in S$ , by unilaterally substituting the $i$ ^th item of $S$ with its new mixed strategy $s_{i}$ , or as a special case, $π_{i j}; S \in S$ by unilaterally substituting with its $j$ ^th pure strategy. Obviously, $s_{i}; S = S$ if $S$ already uses $s_{i}$ .

Intendedly, these vector-centric definitions and notations allow us to, for any player $i = 1, 2, \dots, n$ , translate the linearity of its payoff ${¯ p}_{i} (S)$ in $s_{i}$ into ${¯ p}_{i} (S) = ⟨ s_{i}, v_{i} ⟩$ . Here $⟨ \cdot, \cdot ⟩$ is the inner product of vectors, $s_{i}$ is the mixed strategy player $i$ uses in $S$ and $v_{i}$ is the player’s payoff vector for the $g_{i}$ vertices of its simplex $I_{i}$ . Given any $n$ -tuple $S \in S$ , for each player $i$ we have

v_{i, S} = ({¯ p}_{i} (π_{i 1}; S), \dots, {¯ p}_{i} (π_{i j}; S), \dots, {¯ p}_{i} (π_{i g_{i}}; S)) .

(1)

Given any $S$ , each player can substitute its item in $S$ with $s_{i} \in I_{i}$ to form a new one $s_{i}; S \in S$ , which means that we can designate for each player $i$ a variant payoff function $p_{i, S}$ of substitute strategy $s_{i}$ :

p_{i, S} (s_{i}) = {¯ p}_{i} (s_{i}; S) = ⟨ s_{i}, v_{i, S} ⟩ .

(2)

Note that $p_{i, S} (s_{i})$ is the expected value of payoff or simply the average payoff, since $s_{i}$

can be conveniently treated as probability distribution. From Eq. (

1) and (2) we can derive for each player

i

a function

R_{i, S} (s_{i}) = (φ_{i 1, S} (s_{i}), \dots, φ_{i j, S} (s_{i}), \dots, φ_{i g_{i}, S} (s_{i})) .

(3)

Here each component of vector $R_{i, S} (s_{i})$ is defined by the very function proposed by Nash in his existential proof:

φ_{i j, S} (s_{i}) = max {0, {¯ p}_{i} (π_{i j}; S) - p_{i, S} (s_{i})} .

(4)

Each component $φ_{i j, S} (s_{i})$ represents the payoff gain for player $i$ when its strategy moves from mixed $s_{i}$ to its $j$ ^th pure strategy. We say a pure strategy is “more profitable” than $s_{i}$ if its $φ_{i j, S} (s_{i}) > 0$ or otherwise “less (or equally) profitable”. Vector $R_{i, S} (s_{i})$ has at least one zero component, since there must be one “least profitable” pure strategy as implied by the linearity of $p_{i, S} (s_{i})$ in $s_{i}$ .

Given $S$ , vector $R_{i, S} (s_{i})$ measures, in terms of payoff, how far a substitute $s_{i}$ is from the optimal one. Specifically, the $L 1$ -norm of $R_{i, S} (s_{i})$ , i.e. $∣ ∣ R_{i, S} (s_{i}) ∣ ∣ = \sum_{j}^{g_{i}} φ_{i j, S} (s_{i})$ , decreases as $p_{i, S} (s_{i})$ increases and would turn zero when $p_{i, S} (s_{i})$ is maximized with the optimal substitute. Recall that an equilibrium point is an $n$ -tuple $S$ in which each player’s mixed strategy is optimal against those of its opponentsNash1951. That is, an $n$ -tuple $S^{+} \in S$ is an equilibrium point, if and only if $R_{i, S^{+}} (s_{i}^{+}) = 0$ or equivalently $∣ ∣ R_{i, S^{+}} (s_{i}^{+}) ∣ ∣ = 0$ for each item $s_{i}^{+}$ in $S^{+}$ .

Vector $R_{i, S} (s_{i})$ will be the focal point of our study. We call $R_{i, S} (s_{i})$ the regret vector of player $i$ in that the vector indicates how much payoff it would have gained if its strategy moved to the vertices of its simplex $I_{i}$ , and correspondingly $∣ ∣ R_{i, S} (s_{i}) ∣ ∣$ the regret sum of player $i$ . We shall occasionally abbreviate equilibrium point by EqPt, and write $p_{i} (s_{i})$ , $φ_{i j} (s_{i})$ and $R_{i} (s_{i})$ as implicitly indexed by given $S$ .

Iii Regret Matching and Fixed point iteration

In this section, we will conduct regret matching with regret vector iteratively on the initial mixed strategies to approximate a Nash equilibrium. Ideally, this iterative regret matching could develop into a fixed point iteration fpi.

We define each player’s regret matching to be a function $R_{i, S} : I_{i} \to I_{i}$ :

(5)

Here $r_{i} > 0$ . By $s_{i}^{'} = ψ_{i, S} (s_{i})$ we translate that, on the game with given $n$ -tuple $S$ , if the player $i$ unilaterally changes its mixed strategy to any other $s_{i}$ , it will regret that $s_{i}^{'}$ should have been used instead. We write $ψ_{i} (s_{i})$ occasionally. Our design of $ψ_{i}$ for regret matching has three important aspects to consider.

First, function $ψ_{i}$ has a geometrical interpretation.

We can see in FIG 1 that vector $s_{i}^{'}$ is “closer” to vector $R_{i} (s_{i})$ than vector $s_{i}$ is. Generally, the “closeness” of two vectors, say $u$ and $w$ , can be measured by their angle $∠ (u, w)$ , which further can be measured by $cos ∠ (u, w) = \frac{⟨ u, w ⟩}{∥ u ∥ ∥ w ∥}$ . By Theorem 1, we have $cos ∠ (s_{i} + r_{i} R_{i} (s_{i}), R_{i} (s_{i})) \geq cos ∠ (s_{i}, R_{i} (s_{i}))$ and thus

cos ∠ (s_{i}^{'}, R_{i} (s_{i})) \geq cos ∠ (s_{i}, R_{i} (s_{i})) .

(6)

Theorem 1.

For any non-zero real vectors $u$ , $w$ and $u + r w$ where real $r > 0$ , $cos ∠ (u + r w, w) \geq cos ∠ (u, w)$ .

Proof.

By definition, we have $cos ∠ (u + r w, w) = \frac{⟨ u + r w, w ⟩}{∥ u + r w ∥ ∥ w ∥}$ and $cos ∠ (u, w) = \frac{⟨ u, w ⟩}{∥ u ∥ ∥ w ∥}$ to compare. There are three cases to consider with respect to $⟨ u + r w, w ⟩$ , as follows:
(i) If $⟨ u + r w, w ⟩ > 0$ , by the triangle inequality we have



		$\geq \frac{⟨ u, w ⟩}{∥ u ∥ ∥ w ∥} = cos ∠ (u, w) .$		(7)

(ii) If $⟨ u + r w, w ⟩ = 0$ , $⟨ u + r w, w ⟩ = ⟨ u, w ⟩ + r ∥ w ∥^{2} = 0$ such that $cos ∠ (u, w) < 0 = cos ∠ (u + r w, w)$ .
(iii) If $⟨ u + r w, w ⟩ < 0$ , $⟨ u, w ⟩ < - r ∥ w ∥^{2} < 0$ . Let $u^{'} = u + r w$ and $w^{'} = - w$ . Because $⟨ u^{'} + r w^{'}, w^{'} ⟩ = - ⟨ u, w ⟩ > 0$ , as with case (i) we have $cos ∠ (u^{'} + r w^{'}, w^{'}) > cos ∠ (u^{'}, w^{'})$ and thus $cos ∠ (u, - w) > cos ∠ (u + r w, - w)$ , which easily gives $cos ∠ (u, w) < cos ∠ (u + r w, w)$ . ∎

Second, function $ψ_{i}$ has a behavioral interpretation.

By comparing vectors $s_{i}$ and $s_{i}^{'}$ component-wise, there must be less proportion of pure strategy $π_{i j}$ used in $s_{i}$ than in $s_{i}^{'}$ , if its corresponding component $φ_{i j} (s_{i})$ in vector $R_{i} (s_{i})$ is zero. In other words, $ψ_{i}$ preferably suppresses the use of “less profitable than average” pure strategies and meantime enhances the use of some of the “more profitable than average” ones. Note that this behavioral aspect was mentioned in Nash’s existential proof.

Third, $s_{i}^{'}$ gives more payoff and less regret than $s_{i}$ .

Theorem 2 and Theorem 3 show that $s_{i}^{'}$ will always be a better choice in terms of payoff and regret sum, no matter what $s_{i}$ the player unilaterally changes its mixed strategy to, unless $s_{i}$ itself is optimal.

Theorem 2.

For any $S \in S$ and $s_{i} \in I_{i}$ , $p_{i, S} (s_{i}^{'}) \geq p_{i, S} (s_{i})$ where $s_{i}^{'} = ψ_{i, S} (s_{i})$ .

Proof.

By Eq. (2) we know $p_{i} (s_{i}^{'}) = ⟨ ψ_{i} (s_{i}), v_{i} ⟩$ where $v_{i}$ is the player $i$ ’s vertex payoff vector as in Eq. (1). Then by Eq. (5) we have,


		$= \frac{1}{1 + r_{i} ∣ ∣ R_{i} (s_{i}) ∣ ∣} (⟨ s_{i}, v_{i} ⟩ + r_{i} ⟨ R_{i} (s_{i}), v_{i} ⟩)$
		$= ⟨ s_{i}, v_{i} ⟩ + \frac{⟨ R_{i} (s_{i}), v_{i} ⟩ - ⟨ s_{i}, v_{i} ⟩ ∣ ∣ R_{i} (s_{i}) ∣ ∣}{r_{i}^{- 1} + ∣ ∣ R_{i} (s_{i}) ∣ ∣} .$		(8)

Also by Eq. (2) we have $⟨ s_{i}, v_{i} ⟩ = p_{i} (s_{i})$ . And

		$⟨ R_{i} (s_{i}), v_{i} ⟩ - ⟨ s_{i}, v_{i} ⟩ ∣ ∣ R_{i} (s_{i}) ∣ ∣$
		$= g_{i} \sum j φ_{i j} (s_{i}) {¯ p}_{i} (π_{i j}; S) - g_{i} \sum j p_{i} (s_{i}) φ_{i j} (s_{i})$
		$= g_{i} \sum j φ_{i j} (s_{i}) ({¯ p}_{i} (π_{i j}; S) - p_{i} (s_{i}))$
				(9)

Here $∥ \cdot ∥$ is $L 2$ -norm of vector. Finally because $∥ R_{i} (s_{i}) ∥^{2} \geq 0$ and $r_{i}^{- 1} + ∣ ∣ R_{i} (s_{i}) ∣ ∣ \geq r_{i}^{- 1} > 0$ , we have

(10)

And $p_{i} (s_{i}^{'}) = p_{i} (s_{i})$ if and only if $∣ ∣ R_{i} (s_{i}) ∣ ∣ = 0$ . ∎

Theorem 3.

For any $S \in S$ and $s_{i} \in I_{i}$ , where $s_{i}^{'} = ψ_{i, S} (s_{i})$ .

Proof.

For each component $φ_{i j} (s)$ in vector $R_{i} (s)$ and its counterpart $φ_{i j} (s_{i}^{'})$ in vector $R_{i} (s_{i}^{'})$ , by Eq. (4) and Theorem 2 we have $φ_{i j} (s) \geq φ_{i j} (s_{i}^{'})$ such that $\sum_{j}^{g_{i}} φ_{i j} (s_{i}^{'}) \leq \sum_{j}^{g_{i}} φ_{i j} (s_{i})$ . ∎

These three aspects makes function $ψ_{i}$ a natural regret matching of mixed strategies: if a player $i$ unilaterally and continuously adjusts its strategy by $ψ_{i}$ , intuitively its strategy will keep “pushing” towards regret vector as target, the use of “unprofitable” pure strategies will keep dropping, and most importantly its payoff will monotonically increase and regret sum will monotonically decrease. Since a player’s payoff is bounded, continuous strategy adjustment by $ψ_{i}$ would preferably lead to the optimal mixed strategy with maximum payoff and zero regret sum.

The rest of this section considers all players adjusting their strategies simultaneously and continuously.

On the game with given $S$ , if all $n$ players simultaneously adjust strategies by their own $ψ_{i}$ , there will form a new $S^{'}$ . Let $S^{'} = Ψ (S)$ and we have

Ψ (S) = (ψ_{1, S} (s_{1}), \dots, ψ_{i, S} (s_{i}), \dots, ψ_{n, S} (s_{n})) .

(11)

Note that $Ψ (S) \in S$ for any $S \in S$ because $ψ_{i} (s_{i}) \in I_{i}$ for any $s_{i} \in I_{i}$ . And all players’ continuous strategy adjustment can be described by an iterated function $Ψ \circ Ψ \circ \dots \circ Ψ (S)$ , denoted by $Ψ^{t} (S)$ with $t \in N$ being the times of iterations. Given $S = S_{0}$ , those iterated functions $Ψ^{t} (S)$ with $t = 0, 1, 2, \dots$ give rise to an infinite sequence of points in $S$ for our study:

$S_{0}$	$= Ψ^{0} (S_{0}),$
$S_{1}$	$= Ψ^{1} (S_{0}) = Ψ (S_{0}),$
$S_{2}$	$= Ψ^{2} (S_{0}) = Ψ (S_{1}),$
	$\dots$
$S_{t}$	$= Ψ^{t} (S_{0}) = Ψ (S_{t - 1}),$	(12)
	$\dots$

Here $Ψ^{0}$ denotes the identity function. This sequence is referred to as $(S_{t})_{N}$ implicitly indexed by $S_{0}$ and $Ψ$ . In fact, the tuple $[Ψ, S_{0}]$ uniquely determines a sequence $(S_{t})_{N}$ ; we say $[Ψ, S_{0}]$ generates $(S_{t})_{N}$ . Sequence $(S_{t})_{N}$ describes an infinite dynamics of players adjusting their strategies in parallel. Specifically, at the $t$ ^th iteration, each player $i$ adjusts its strategy by substituting $s_{i}$ in $S_{t - 1}$ with $ψ_{i} (s_{i})$ so as to collectively form a new $n$ -tuple $S_{t} = Ψ (S_{t - 1})$ ; at the $(t + 1)$ ^th iteration $S_{t + 1}$ is formed by adjusting $S_{t}$ in the same manner; and so forth.

Now we might hope that each and every player still has its payoff monotonically increase and its regret sum monotonically decrease in the process of simultaneous strategy adjustment. In that case, the infinite sequence $(S_{t})_{N}$ must tend towards an EqPt. Moreover, it would be ideal if the overall regret sum of all players, i.e. $\sum_{i}^{n} ∣ ∣ R_{i} (s_{i}) ∣ ∣$ , converged to zero along $(S_{t})_{N}$ such that ${lim}_{\infty} \sum_{i}^{n} ∣ ∣ R_{i} (s_{i}) {∣ ∣}^{(t)} = 0$ and thus ${lim}_{\infty} ∣ ∣ R_{i} (s_{i}) {∣ ∣}^{(t)} = 0$ for each player $i$ . In that case, by definition ${lim}_{\infty} S_{t}$ must be an EqPt. Denote it by $S^{+}$ and we must have $Ψ (S^{+}) = S^{+}$ . Therefore, EqPt $S^{+} = {lim}_{\infty} S_{t}$ is also a fixed point fixedpoint of function $Ψ$ ; the sequence $(S_{t})_{N}$ generated by $[Ψ, S_{0}]$ is essentially an outcome of the process of fixed point iteration, which can be formulated to ${lim}_{\infty} Ψ \circ Ψ \circ \dots \circ Ψ (S_{0}) = {lim}_{\infty} Ψ^{t} (S_{0}) = S^{+}$ . For our purpose we assume that, with proper finite iterations, $S^{*} = S_{t} = Ψ^{t} (S_{0})$ can be a reasonable approximate of the true EqPt $S^{+}$ such that $S^{*} \approx S^{+}$ , and the accuracy of such EqPt approximation can be measured by the overall regret sum of all players.

Only this hope comes with huge caveat. As it turns out in Section V, the presumption on the monotonicity of payoff and regret sum for all players along $(S_{t})_{N}$ oftentimes doesn’t hold true. Instead, due to simultaneous strategy adjustment each player’s regret sum might fluctuate along $(S_{t})_{N}$ with a tendency to diminish towards zero. Nevertheless, we will see that this tendency can still serve the purpose of approximating an EqPt. Notice that in FIG 1, the parameter $r_{i}$ in Eq. (5) determines the rate of angle by which vector $s_{i}$ and regret vector $R_{i} (s_{i})$ “close up”. Here we call $r_{i}$ adjustment rate. To counter the influence from simultaneous strategy adjustment, as we will see in next section “infinitesimal” adjustment rates are used for all players, in the hope that each player’s strategy adjustment could be treated as negligible by its opponents and thus their simultaneous strategy adjustment could be treated as approximately unilateral.

We should visualize the sequence $(S_{t})_{N}$ of Eq. (III) as players’ perpetual searching for the equilibrium point of non-cooperative game. In this searching, players don’t have to be aware of the equilibrium point per se. Instead, each player is constantly chasing after its regret vector $R_{i} (s_{i})$ in the hope of reducing its regret sum $∣ ∣ R_{i} (s_{i}) ∣ ∣$ to zero. Increasing payoff and meantime reducing regret sum is the players’ sole incentive and purpose, and regret vector is the players’ sole information and target for their strategies adjustment. Regret vector is dynamic along iterations, of course. At any iteration, given the current mixed strategies of its opponents, each player can calculate its regret vector by evaluating the payoff on its pure strategies without any across-iteration retrospective considerations. In this searching, there is no place for the mediator beyond players; no inter-player information exchange other than the current use of mixed strategies.

Iv Game examples and visualizations

In this section we will verify our approach of EqPt approximation proposed in last section. Examples of both two-person game and general $n$ -person game will be tested, although the former ones are our main consideration due to their simplicity and sufficiency in revealing the important aspects of EqPt approximation.

For the two-person game we have $n = 2$ players, say the row player with $k$ pure strategies and the column player with $m$ pure strategies. Accordingly, the $n$ -tuple $S$ is specialized to two-tuple $(s_{r}^{1 \times k}, s_{c}^{1 \times m})$ of vectors, and thus the tuple $[Ψ, S_{0}]$ is specialized to $[(A, B), (r_{r}, r_{c}), (s_{r}, s_{c})_{0}]$ since function $Ψ$ is determined by the two players’ payoff matrices $(A^{k \times m}, B^{k \times m})$ and their adjustment rates $(r_{r}, r_{c})$ . Recall that given any $S_{0} \in S$ the tuple $[Ψ, S_{0}]$ generates an infinite sequence $(S_{t})_{N}$ correspondingly. Then we can implement the fixed point iteration in last section to Algorithm 1¹¹1Source code and demos can be found at https://github.com/lansiz/eqpt., which takes $[(A, B), (r_{r}, r_{c}), (s_{r}, s_{c})_{0}]$ as input, and for our study outputs an approximate EqPt $(s_{r}, s_{c})^{*}$ as $S^{*}$ and its regret sum $(R_{r}, R_{c})^{*}$ , a finite sequence $((s_{r}, s_{c})_{t})_{T}$ in correspondence to $(S_{t})_{N}$ , and a finite sequence $((R_{r}, R_{c})_{t})_{T}$ of regret sum. In Algorithm 1, the $max$ operation compares vectors component-wise as opposed to Eq. (4). Obviously, given $T$ the computational complexity of Algorithm 1 is under $O (g^{2})$ with $g = max {k, m}$ .

1:payoff bimatrix

(A^{k \times m}, B^{k \times m})

, adjustment rates

(r_{r}, r_{c})

, initial two-tuple of mixed strategies

(s_{r}^{1 \times k}, s_{c}^{1 \times m})

and iterations

T

2: approximate EqPt

(s_{r}, s_{c})^{*}

and its regret sum

(R_{r}, R_{c})^{*}

; sequence

((s_{r}, s_{c})_{t})_{T}

; sequence

((R_{r}, R_{c})_{t})_{T}

3:for

t = 1

T

4: vertices payoff:

v_{r} = A s_{c}^{⊺}

v_{c} = B^{⊺} s_{r}^{⊺}

5: payoff:

p_{r} = s_{r} v_{r}

p_{c} = s_{c} v_{c}

6: regret vector:

R_{r} = max {0, v_{r} - p_{r}}

R_{c} = max {0, v_{c} - p_{c}}

7: regret sum:

R_{r} = ∣ ∣ R_{r} ∣ ∣

R_{c} = ∣ ∣ R_{c} ∣ ∣

8: if overall regret sum

R_{r} + R_{c}

is a new minimum then

9: update

(s_{r}, s_{c})^{*}

with

(s_{r}, s_{c})

10: update

(R_{r}, R_{c})^{*}

with

(R_{r}, R_{c})

11: endif

12: append

(s_{r}, s_{c})

into its sequence.

13: append

(R_{r}, R_{c})

into its sequence.

14: adjust mixed strategies:

s_{r} = \frac{s_{r} + r_{r} R_{r}}{1 + r_{r} R_{r}}

s_{c} = \frac{s_{c} + r_{c} R_{c}}{1 + r_{c} R_{c}}

15:end for

Algorithm 1 EqPt approximation of two-person game.

Given the output of Algorithm 1, we can plot the true EqPt $(s_{r}, s_{c})^{+}$ , the approximate EqPt $(s_{r}, s_{c})^{*}$ and the sequence $((s_{r}, s_{c})_{t})_{T}$ on $R^{2}$ or $R^{3}$ to provide an intuitive visualization of $((s_{r}, s_{c})_{t})_{N}$ approximating EqPt. Specifically, for the $3 \times 3$ two-person games with $k = 3$ and $m = 3$ , we split $((s_{r}, s_{c})_{t})_{T}$ into two sequences of $(s_{t})_{T}$ , and then transform each into a sequence of $((x, y)_{t})_{T}$ for plotting on $R^{2}$ plane by defining

(x, y)_{t} = s_{t} ⎡ ⎢ ⎢ ⎣ \begin{matrix} 0 & 0 \frac{\sqrt{2}}{2} & \frac{\sqrt{6}}{2} \sqrt{2} & 0 \end{matrix} ⎤ ⎥ ⎥ ⎦ .

(13)

Note that Eq. (13) transforms the simplex in $R^{3}$ into a subset of $R^{2}$ enclosed by an equilateral triangle. The following are our observations on the test results of four typical $3 \times 3$ games:

Game 3X3-1eq1sp has one unique EqPt, which uses one single pure strategy. As shown in FIG 6, $((s_{r}, s_{c})_{t})_{T}$ converges straight towards the EqPt.
Game 3X3-1eq2sp has one unique EqPt, which uses two pure strategies. As shown in FIG 2, $((s_{r}, s_{c})_{t})_{T}$ converge towards the EqPt with oscillation.
Game 3X3-2eq2sp has two EqPts, each of which uses two pure strategies. As shown in FIG 7, $((s_{r}, s_{c})_{t})_{T}$ converges with oscillation towards one of the two EqPts dependent on the initial strategies.
Game 3X3-1eq3sp has one unique EqPt, which uses three pure strategies. As shown in FIG 8, $((s_{r}, s_{c})_{t})_{T}$ doesn’t converge towards the EqPt but develops into a seemingly perfect circle away from the true EqPt. The last position of $((s_{r}, s_{c})_{t})_{T}$ could be any point in the circle.

Figure 6: The “strategy paths” on the simplex of $R^{3}$ for game 3X3-1eq1sp. The two black crosses combine to represent the true EqPt. The input and output of game are annotated.

Figure 7: The “strategy paths” for game 3X3-2eq2sp. The approximate EqPt annotated corresponds to the true EqPt at the base line of equilateral triangle.

Figure 8: The “strategy paths” for game 3X3-1eq3sp.

Figure 9: The “strategy paths” for a $60 \times 40$ game. The blue and red dots represent the last positions of strategies.

And for the general two-person games with $k > 3$ and $m > 3$ , we split $((s_{r}, s_{c})_{t})_{T}$ into two sequences of $(s_{t})_{T}$

, and then with PCA (Principal Component Analysis for dimension reduction

pca) transform each into a sequence of

((x, y, z)_{t})_{T}

for plotting in

R^{3}

space. The following are our observations on the test results of two typical

60 \times 40

games:

FIG 3 shows a $60 \times 40$ game whose $((s_{r}, s_{c})_{t})_{T}$ converges towards an EqPt.
FIG 9 shows a $60 \times 40$ game whose $((s_{r}, s_{c})_{t})_{T}$ develops into a perfect circle.

For the two-person game, its EqPts are determined by the bimatrix $(A, B)$ , independent of the initial $(s_{r}, s_{c})_{0}$ . Then for the $3 \times 3$ game examples mentioned previously, we test them with many random $(s_{r}, s_{c})_{0}$ and observe that there appear to be two kinds of EqPts, “attractor” EqPt and “repellor” EqPt, as follows:

Figure 10: Both EqPts of game 3X3-2eq2sp are “attractors”. Ten pairs of random “strateg paths” are depicted.

Figure 11: The EqPt of game 3X3-1eq3sp is a “repellor”. Ten pairs of random “strateg paths” are depicted, five of which are started from closely near the true EqPt.

For game 3X3-1eq2sp FIG 5 shows that, any $((s_{r}, s_{c})_{t})_{T}$ is “attracted” towards the unique EqPt.
For game 3X3-2eq2sp FIG 10 shows that, any $((s_{r}, s_{c})_{t})_{T}$ is “attracted” towards one of the two EqPts.
For game 3X3-1eq3sp FIG 11 shows that, any $((s_{r}, s_{c})_{t})_{T}$ is ”repelled” from the EqPt unless $(s_{r}, s_{c})_{0}$ is exactly the EqPt.

We also test the examples of $n$ -person game by extending Algorithm 1 from bivariate case to general multivariate case, which can be better explained by the source code. And we observe the convergence or disconvergence of $(S_{t})_{T}$ as with the two-person games. FIG. 4 shows an example for the convergence case. The computational complexity of this extended algorithm is under $O (n^{2} g^{n})$ with $n$ being the number of players and $g = max {g_{1}, g_{2}, \dots, g_{n}}$ , meaning that it is polynomial time when $n$ is given.

V EqPt approximation accuracy

In last section, we observe that for some games sequence $((s_{r}, s_{c})_{t})_{T}$ seemingly converges towards EqPt, e.g., 3X3-1eq1sp, 3X3-1eq2sp and 3X3-2eq2sp. Whereas for games such as 3X3-1eq3sp, sequence $((s_{r}, s_{c})_{t})_{T}$ clearly doesn’t converge at all; instead, it evolves into a perpetual cyclic path away from EqPt. Roughly speaking, the convergence or disconvergence of $((s_{r}, s_{c})_{t})_{T}$ affects the accuracy of EqPt approximation. In this section, we will look into the approximation accuracy from two numerical perspectives. One is to examine sequence $(R_{r}, R_{c})_{t})_{T}$ since regret sum is a measure of approximation accuracy as discussed in Section III, and the other one is to revisit the convergence of $((s_{r}, s_{c})_{t})_{T}$ with metric. And we will show the connection between them.

First, by taking the $3 \times 3$ games in last section for example, we have the following observations on their sequences $((R_{r}, R_{c})_{t})_{T}$ plotted in FIG 12:

As opposed to the presumption in Section III, regret sum oftentimes doesn’t monotonically decrease. Instead, due to the inter-player influence from simultaneous strategy adjustment, regret sum fluctuates with a decreasing tendency to reach new minimum. Game 3X3-1eq1sp barely has regret sum fluctuation, whereas game 3X3-1eq3sp has intensive regret sum fluctuation.
There is different degree of periodicity in regret sum fluctuation for different games. Regret sum fluctuation of game 3X3-1eq3sp exhibits strong periodicity and yet weak decreasing tendency.
The strong periodicity of regret sum sequence in game 3X3-1eq3sp, caused by the “repellor” EqPt, severely undermines the accuracy of EqPt approximation.
It is painfully slow for regret sum sequence to converge towards zero in that regret sum decreases slowly at the latter iterations, even for 3X3-1eq1sp. That is partly because, as implied by Eq. (10), the ratio of payoff increment to regret sum is getting smaller as iteration goes.

Now it can be concluded that the inter-player influence has a significant impact on the EqPt approximation accuracy which otherwise could, given time, go on to perfection. In addition, FIG 13 shows the decreasing tendency of regret sum for a five-person game.

Figure 13: The regret sum sequences of a five-person game with players using two, three, four, five and six pure strategies.

Next, for the $3 \times 3$ games in last section, we will reexamine their output sequences $((s_{r}, s_{c})_{t})_{T}$ by introducing a metric on $S$ . Generally, metric is necessary for the definitions of limit, convergence and contractiveness of sequence $(S_{t})_{N}$ . Following the definitions in Section III, let us have a set $S$ and a function $Ψ : S \to S$ of Eq. (11). From $S$ we can derive a complete metric space $(S, d)$ with the metric function

d (X, Y) = n \sum i ∥ x_{i} - y_{i} ∥ .

(14)

Here $x_{i}$ and $y_{i}$ are the $i$ ^th items of $X \in S$ and $Y \in S$ respectively. Assume that $Ψ : S \to S$ is a contraction mapping on metric space $(S, d)$ . That is, there exists a $q \in (0, 1)$ such that for any $X, Y \in S$

d (Ψ (X), Ψ (Y)) \leq q d (X, Y) .

(15)

Then by Banach fixed point theorem fixedpoint, the function $Ψ$ must admit one unique fixed point $S^{+} \in S$ such that $Ψ (S^{+}) = S^{+}$ and ${lim}_{\infty} Ψ^{t} (S_{0}) = S^{+}$ for any $S_{0} \in S$ . Apparently the contraction condition of Eq. (15) is too strong for the convergence of $(S_{t})_{N}$ , since it precludes the sequence’s possible convergence towards more than one EqPt, which is an important case of our interest as shown in last section. Instead, let us consider the contraction property of a specific sequence $(S_{t})_{N}$ generated by the given tuple $[Ψ, S_{0}]$ . From $(S_{t})_{N}$ we can derive an infinite real sequence $({˙ d}_{t})_{N_{+}}$ by defining

{˙ d}_{t} = d (S_{t - 1}, S_{t}) = d (S_{t - 1}, Ψ (S_{t - 1})) .

(16)

And from sequence $({˙ d}_{t})_{N_{+}}$ we can further derive an infinite real sequence $({˙ q}_{t})_{N_{+}}$ by defining

(17)

If $(S_{t})_{N}$ is a contractive sequence ideally, i.e. $0 < {˙ q}_{t} < 1$ for any $t \in N_{+}$ , it must be an Cauchy sequence and converge to an EqPt. That is, $(S_{t})_{N}$ is a process of fixed point iteration. Meanwhile, the sequence $({˙ d}_{t})_{N_{+}}$ must monotonically decrease and converge to zero. From the $((s_{r}, s_{c})_{t})_{T}$ sequences generated by the $3 \times 3$ games of last section, we derive and depict their $({˙ d}_{t})_{T - 1}$ and $({˙ q}_{t})_{T - 2}$ in FIG 14 to show that $({˙ d}_{t})_{T - 1}$ sequences don’t monotonically decrease or converge to zero, meaning that neither $((s_{r}, s_{c})_{t})_{T}$ sequences nor their $Ψ$ functions are contractive on $(S, d)$ . Nevertheless, by comparing FIG 12 and FIG 14, we observe that there is strong coherence among the sequences $((R_{r}, R_{c})_{t})_{T}$ , $({˙ d}_{t})_{T - 1}$ and $({˙ q}_{t})_{T - 2}$ of the same game, as follows:

As with regret sum, $˙ d$ oftentimes fluctuates with a decreasing tendency to reach new minimum. $˙ d$ of game 3X3-1eq1sp exhibits almost no fluctuation, whereas $˙ d$ of game 3X3-1eq3sp exhibits strong periodicity.
As with regret sum, in game 3X3-1eq3sp the strong periodicity of $({˙ d}_{t})_{T - 1}$ could have hurt the accuracy of EqPt approximation.
$˙ q$ fluctuates around $1$ at the latter iterations, which explains why $({˙ d}_{t})_{T - 1}$ and regret sum sequence converge at such a low speed.

Therefore, $˙ d$ can be seen as an another overall measure of EqPt approximation accuracy in addition to overall regret sum and to improve accuracy is to reduce $˙ d$ , which in return justifies our definition of metric function in Eq. (14). Trivially, there is an alternative metric function and applying it gives us observations similar to those with Eq. (14). Here is the alternative topology:

d (X, Y) = n max i ∥ x_{i} - y_{i} ∥ .

(18)

Now we can see the shortcomings of our theory: it relies too much on the naked-eye observation. Given $S_{0}$ and $Ψ$ , only by observing the generated sequence $(S_{t})_{N}$ and its derived sequences can we learn about their convergence or disconvergence, their periodicity, EqPts being “attractors” or “repellors”, EqPt approximation accuracy, etc. And yet we fail to, in the case of two-person game, provide a function if any – say $χ (A, B, r_{r}, r_{c}, S_{0})$ – to conveniently determine for $[Ψ, S_{0}]$ . Nor can we provide a function $χ (A, B, r_{r}, r_{c})$ to determine whether every $S_{0} \in S$ leads to a convergent sequence.

As a practical use of our theory, given a game and players’ initial strategy $S_{0}$ we need to find out the optimal parameters for function $Ψ$ in order to optimize EqPt approximation accuracy. That is, in the case of two-person game, given payoff bimatrix $(A, B)$ and initial $(s_{r}, s_{c})_{0}$ , we need to find out the optimal input $[(A, B), (r_{r}, r_{c}), (s_{r}, s_{c})_{0}]$ for Algorithm 1 to minimize overall regret sum. For that purpose, obviously we can try different adjustment rates $(r_{r}, r_{c})$ in input as in FIG 15. And notice that bimatrix $(a_{r} A + b_{r}, a_{c} B + b_{c})$ , where $(a_{r}, a_{c}) \in R_{+}^{2}$ and $(b_{r}, b_{c}) \in R^{2}$ , has the same EqPts as bimatrix $(A, B)$ . Thus we can try different scale factors $(a_{r}, a_{c})$ in input as shown in FIG 16. With parameters $(r_{r}, r_{c})$ , $(a_{r}, a_{c})$ and $(b_{r}, b_{c})$ combined, there forms a search space for function $Ψ$ , which could be overwhelming when it comes to many-person games.

Figure 15: The approximate EqPt’s regret sum $(R_{r}, R_{c})^{*}$ of Algorithm 1 for different adjustment rates in the case of game 3X3-1eq3sp. For simplicity we consider $r_{r} = r_{c}$ for the input $(r_{r}, r_{c})$ .

Figure 16: The approximate EqPt’s regret sum $(R_{r}, R_{c})^{*}$ of Algorithm 1 for different bimatrix scale factors in the case of game 3X3-1eq3sp. For simplicity we consider $a_{r} = a_{c}$ and $(b_{r}, b_{c}) = (0, 0)$ for the input $(a_{r} A + b_{r}, a_{c} B + b_{c})$ . Note that the output $(R_{r}, R_{c})^{*}$ is “scaled back” with $(a_{r}^{- 1}, a_{c}^{- 1})$ for proper comparison.

Vi Variants of function $R_{i, S}$

In this section, we will introduce two variants of $R_{i, S}$ function for the purpose of generalization.

By applying on each component of the vector $R_{i, S} (s_{i})$ a general function $α_{i j} : R_{\geq 0} \to R_{\geq 0}$ , we have the first variant:

		${¯ R}_{i, S} (s_{i}) =$
		$(α_{i 1} \circ φ_{i 1, S} (s_{i}), \dots, α_{i j} \circ φ_{i j, S} (s_{i}), \dots, α_{i g_{i}} \circ φ_{i g_{i}, S} (s_{i})) .$		(19)

Accordingly, we can write Eq. (9) to


		$= g_{i} \sum j α_{i j} \circ φ_{i j} (s_{i}) ({¯ p}_{i} (π_{i j}; S) - p_{i} (s_{i})) .$		(20)

Suppose that $α_{i j} (0) = 0$ for any $i$ and

Geometrical Regret Matching of Mixed Strategies

Authors

The Path to Nash Equilibrium

On the Impossibility of Convergence of Mixed Strategies with No Regret Learning

Strategizing against No-regret Learners

Faster Regret Matching

Evolutionary Dynamics and Φ-Regret Minimization in Games

Solving Games with Functional Regret Estimation

Fictitious Play with Maximin Initialization

Code Repositories

eqpt

I Introduction

Ii Non-cooperative game and regret

Iii Regret Matching and Fixed point iteration

Theorem 1.

Proof.

Theorem 2.

Proof.

Theorem 3.

Proof.

Iv Game examples and visualizations

V EqPt approximation accuracy

Vi Variants of function $R_{i, S}$

Geometrical Regret Matching of Mixed Strategies

Authors

Related Research

The Path to Nash Equilibrium

On the Impossibility of Convergence of Mixed Strategies with No Regret Learning

Strategizing against No-regret Learners

Faster Regret Matching

Evolutionary Dynamics and Φ-Regret Minimization in Games

Solving Games with Functional Regret Estimation

Fictitious Play with Maximin Initialization

Code Repositories

eqpt

This week in AI

I Introduction

Ii Non-cooperative game and regret

Iii Regret Matching and Fixed point iteration

Theorem 1.

Proof.

Theorem 2.

Proof.

Theorem 3.

Proof.

Iv Game examples and visualizations

V EqPt approximation accuracy

Vi Variants of function Ri,S

Vi Variants of function $R_{i, S}$