Control Problems with Vanishing Lie Bracket Arising from Complete Odd Circulant Evolutionary Games

10/24/2017 ∙ by Christopher Griffin, et al. ∙ IEEE Johns Hopkins University 0

We study an optimal control problem arising from a generalization of rock-paper-scissors in which the number of strategies may be selected from any positive odd number greater than 1 and in which the payoff to the winner is controlled by a control variable γ. Using the replicator dynamics as the equations of motion, we show that a quasi-linearization of the problem admits a special optimal control form in which explicit dynamics for the controller can be identified. We show that all optimal controls must satisfy a specific second order differential equation parameterized by the number of strategies in the game. We show that as the number of strategies increases, a limiting case admits a closed form for the open-loop optimal control. In performing our analysis we show necessary conditions on an optimal control problem that allow this analytic approach to function.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Cyclic competition occurs frequently in nature JB75 ; SL96 ; KRFB02 ; GHS03 ; KR04 ; KNS05 ; NHK11 . Cyclic competition in coral reef populations are studied in JB75 . Sinervo and Lively first characterized rock-paper-scissors like competition in lizards SL96 , while Gilg, Hanski and Sittler GHS03 study this behavior in rodents. Cyclic behavior in microbial populations is studied in KRFB02 ; KR04 ; KNS05 ; NHK11 . In classical and evolutionary game theory, cyclic dominance (e.g., matching pennies, rock-paper-scissors) games are commonly studied Mor94 ; Wei95 . Biologically speaking, in an idealized cyclic game, the absolute fitness measure (payoff) resulting from species interaction can be represented by a circulant matrix, in which row is a rotation of row for each . Games with circulant payoff matrices have been studied extensively in evolutionary game theory Ze80 ; SSW80 ; HS98 ; HS03 ; DG09 ; HS00 ; SMJS14 ; GK16 and provide some of the most interesting behaviors SMJS14 .

In early work, cyclic interaction is considered without explicit reference to games. Cyclic (chemo-biological) interactions are studied extensively in SSW78 ; HSSW80 ; SSW79 in which both competitive and cooperative behaviors are identified. Analysis of the replicator in which a circulant matrix emerges is studied in SSW80 as a result of cyclic mass interaction kinetics. Zeeman Ze80 made an early study of the dynamics of cyclic games showing that in rock-paper-scissors a degenerate Hopf bifurcation leads to the emergence of a non-linear center with no limit cycle possible in any 3 strategy game under the replicator dynamics. Since this early work, several authors have investigated various cyclic games and games characterized by circulant matrices. Among many other works: Hofbauer and Schlag HS00 consider imitation in cyclic games; Diekmann and Gils specifically study the cyclic replicator dynamics and focus on the properties of low-dimensional cyclic games DG09 ; Ermentrout et al. consider a transition matrix evolutionary dynamic in which a limit cycle emerges in the rock-paper-scissors game EGB16 ; and Griffin and Belmonte GB17 study a triple public goods game and show that is is diffeomorphic to generalized rock-paper-scissors. Each of these works focuses explicitly on classes of circulant games, while recent work by Granić and Kerns GK16 characterizes the Nash equilibria of arbitrary circulant games, but does not focus on the evolutionary game context.

There has also been extensive work on spatial games with circulant payoff matrices. Peltomäki and Alvara PA08 consider both 3 and 4 state rock-paper-scissors. Other papers consider rock-paper-scissors with variations on reaction rate HMT10 or study the basins of attraction SWYL10 . DeForest and Belmonte dB13 study a fitness gradient variation on the spatial replicator and show rock-paper-scissors can exhibit spatial chaos under these dynamics. Finally more recent work by Szczesny et. al SMR14 considers spiral formations in rock-paper-scissors.

In this paper, we extend work in GB17 by studying an optimal control problem defined on a -strategy () generalization of rock-paper-scissors. Odd cardinality interactions are interesting because they model specific biological cases SL96 ; GHS03 ; SSW78 ; HSSW80 ; SSW79 . Additionally, when is very large, these have the potential to model systems in which individuals a variety of individuals with strengths and weaknesses interact.

In particular, the payoff matrix is (i) defined by the sum of two circulant matrices, and (ii) admits a single control parameter. Thus we consider the general class of control problems first studied in a specific case in GB17 . Our payoff matrix is inspired by the generalized rock-paper-scissors matrix defined in Wei95 . Since every pair of heterogeneous strategic interactions (e.g., rock vs. scissors) results in a non-zero payoff, we refer to this class of games as complete odd circulant games.

The major results of this paper are:

  1. We show that the complete odd circulant games admit only a unique interior fixed point under the replicator dynamics. We also characterize the fixed points of the -strategy complete odd circulant game in terms of the fixed points of the complete odd circulant games.

  2. We show that the replicator dynamics can be written as the sum of an uncontrolled component and a controlled component both of which have circulant Jacobian matrices.

  3. As a consequence, we completely characterize the stability of the interior fixed point and use this to define an optimal control problem with objective to drive the system trajectories toward this interior fixed point.

  4. We describe the properties of a general class of control problems with a vanishing Lie Bracket that will be used to analyze the control problem we define. This suggests interesting geometric and algebraic connections between evolutionary games and optimal control theory. As a by-product, we generalize recent control theoretic results in FG17 .

  5. We show that a quasi-linearization of the control problem (as done in GB17 ) has special form admitting a complete characterization of the dynamics of the optimal control. We also derive a sufficient condition on control optimality and thus completely generalize the results in GB17 to arbitrary complete odd circulant games.

  6. As a part of the generalization, we find a limiting second order ordinary differential equation (as

    grows large) that the optimal control must obey and show that it has a natural closed form solution.

The remainder of this paper is organized as follows: In Section 2 we present preliminary results and notation. In Section 3 we introduce the control problem of interest and study a general class of optimal control problems that will assist in the derivation of our main results. Our main results on control of complete odd circulant games are found in Section 4. Conclusions and future directions are presented in Section 5.

2 Notation and Preliminary Results

A circulant matrix is a square matrix with form:

A circulant matrix is entirely characterized by its first row and all other rows are cyclic permutations of this first row. The set of

circulant matrices forms a commutative algebra, a fact that will be used frequently in this paper. Moreover, the eigenvalues of these matrices have special form. If

is an circulant matrix and are the roots of unity, then eigenvalue () is given by the expression:

Further details on this class of matrices is available in D12 .


be the unit -simplex embedded in -dimensional Euclidean space. Here

is an appropriately sized vector of

’s and is a zero vector.

We consider a family of control problems defined on parametrized cyclic games with strategies, where . If and , define:


For the remainder of this paper, define so that:




By way of example, we illustrate the matrices and for the 5-strategy cyclic game.

From a game-theoretic perspective, we can think of as being the traditional payoff matrix of the complete cyclic game with strategies; e.g., the payoff matrix of rock-paper-scissors. On the other hand, can be thought of an actuating matrix that will determine whether the interior fixed point of the complete cyclic game is stable or unstable, as we show in the sequel.

In the remainder of this paper, we will consider the generalized cyclic game with strategies where , and we note that both and are circulant matrices. The payoff matrix for the generalized cyclic game with strategies and parameter is:

If and , then is just the rock-paper-scissors matrix. Without loss of generality, we assume . Otherwise, the natural winning precedence in the cyclic game is reversed.

The matrix is the adjacency matrix of an vertex circulant directed graph , whose edge direction determines the winning precedence between two strategies in the underlying cyclic game. This is illustrated in Figure 1.

(a) Five Strategy Game
(b) Seven Strategy Game
Figure 1: Five and seven strategy game represented as graphs.

In particular, if , then Strategy defeats Strategy and yields payoff in .

In the control problem defined in the sequel, the replicator dynamics are the nonlinear equations of motion with control parameter :


Here is the vector denoting the proportion of the population playing each of the strategies. It is well known HS98 ; Wei95 that if , then is confined to for all time. For the remainder of this paper, we assume .

Assume and let and . The point-to-set mapping maps a point to the set of points in so that:

  1. elements of the vector consist of the elements of . The remaining elements are .

  2. If the rows and columns of (resp. ) corresponding to the zero-entries in are deleted to form the matrix (resp. ), then (resp. .

The second condition is equivalent to stating that the vertices corresponding to the non-zero strategies in induce a sub-graph of that is isomorphic to .

Lemma 1.

Let and , then:

  1. has among its fixed points () and .

  2. Furthermore, if , , and and is a fixed point of , then are fixed points of .


To prove Statement 1, note first that pure strategies are always fixed points of the replicator dynamics Wei95 . To see that is a fixed point, note that:


The fact that is a fixed point follows immediately.

To prove Statement 2, suppose that for some fixed point of . Let and be the square sub-matrices of and obtained by removing the rows and columns corresponding to the zero entries in . By assumption, it follows at once that:

This completes the proof. ∎

Corollary 2.

The fixed point is the unique interior fixed point for .


If is any interior fixed point, then necessarily it must satisfy the equation:

Note that for any , . Therefore, must satisfy:

It is straight forward to compute:

We now proceed in cases. If , then we solve:

This system has rank with unknowns and therefore admits a unique solution. Elementary row reduction shows that is the unique fixed point in this case.

On the other hand, if , then let:

We can determine a relationship between the variables by solving:

Row-reduction on the system shows that when we obtain the relationship:


Consequently, and necessarily is again the unique interior fixed point. This completes the proof. ∎

For the remainder of this paper, assume and let be defined component-wise as:


The replicator dynamics are then:

which are the dynamics that will be used in the control problem of interest. We note that and are the functional imprints of the standard payoff matrix and the actuation matrix within the replicator framework.

Lemma 3.

The Jacobian matrix of evaluated at is:


We prove the result for Row 1 of . The remainder of the argument follows from the circulant structure of . We have:

because . Note:


Differentiating with respect to and evaluating at yields:

since is odd. Differentiating Expression 8 with respect to and evaluating at yields:

The result now follows from the fact that is a circulant matrix. ∎

Lemma 4.

The Jacobian matrix of evaluated at is:

where is an matrix of ’s.


As in the proof of Lemma 3, we show the result for Row 1 of . The remainder of the argument follows from the circulant structure of . We have already noted in Corollary 2 that:

We compute:

Differentiate with respect to for , corresponding to a non-zero index in the first row of . We have:

Evaluating at we obtain:

for with . Differentiate now with respect to to obtain:


Evaluating at we obtain:

when is substituted for in the numerator. Finally, consider and for . Differentiating with respect to we have:

Evaluating at we obtain:

The result now follows from the fact that is a circulant matrix. ∎

Theorem 5.

If , then the fixed point is asymptotically stable. If , then the fixed point is asymptotically unstable.


From Lemmas 3 and 4, we note that the Jacobian matrix of at has the following form:

By its construction, it is a circulant matrix with first row given by:


Letting for be the roots of unity,111For details see D12 , we know that the eigenvalue of is:

It now remains to show that the sign of the real-part of is entirely dependent on . The real part of the eigenvalue is given by:


The first eigenvalue () is real and readily computed:

It is clear at once that the sign of this eigenvalue is entirely controlled by the sign of .

For , note that the periodicity of the cosine function (and the fact that the roots of unity are the vertices of the regular unit -gon) implies that the coefficient of is identical to the coefficient of if . From this fact and the Expression 10, the sum in Equation 11 becomes:

Factoring further we see:

The roots of unity are evenly distributed on the vertices of the unit -gon in and therefore the sum of the real parts must be zero. It follows that:

We now obtain an exact value for the real parts of the eigenvalues:


Thus we have proved that when , then for all and if , then . The asymptotic stability (resp. instability) of the fixed point follows immediately. ∎

We illustrate the attractive interior point of the five and seven strategy cyclic games by projecting them into regular -gons (), shown in Figure 2. A pure strategy corresponds to a vertex, while a mixed strategy is located in the interior. Note, these are not true trajectories as would be seen in a classic ternary plot on the three rock-paper-scissors, but they are similarly representative.

(a) Five Strategies
(b) Seven Strategies
Figure 2: A figure illustrating the attraction of the unique interior fixed point of the cyclic game when assuming and strategies. The trajectories are projected into regular -gons (). The closer a trajectory is to a vertex, the closer it is to that pure strategy.

3 The Control Problem and Some General Results

We now state our control problem of interest:


where . Such a problem arises naturally if we consider species interacting in a cyclic manner and is a costly control mechanism by which an external manager may control species populations. In GB17 , arises naturally as a tax in a public-goods game, which is shown to be diffeomorphic to a three stragegy cyclic game. As in GB17 , we will show that a quasi-linearization of this control problem has special structure. In this case, however, we show this special structure holds for all cyclic games (i.e., for all ). Furthermore, we discuss the limiting behavior of the control as grows large. To do this, we first consider a very general optimal control problem and obtain necessary conditions for simplifying the Euler-Lagrange necessary conditions. We then use these simplifications to generalize the results in GB17 .

3.1 Control Problems with One Control and Vanishing Lie Bracket

In the remainder of this section, the functions are arbitrary smooth functions, rather than the functions specific to the replicator dynamics for cyclic games given in Equations 6 and 7, is a state vector, and is the control function to be determined.

Consider the general optimal control problem with form:


The functions are smooth. Let , be the terminal time, and be convex. Expression 13 has this structure, so we are simply considering a more general case of our problem of interest.

The Euler-Lagrange necessary conditions for control are simple to derive for this problem and have an almost linear behavior. Note the Hamiltonian is:


The Hamiltonian is (strictly) convex in the control , and thus we propose the following:

Lemma 6.

Any solution to satisfies the necessary conditions:

  1. , and

  2. , the strong Legendre-Clebsch condition;

therefore, it minimizes the Hamiltonian at all times. ∎

Deriving the optimal control by solving for to obtain:


The two conditions in Lemma 6, along with the fact that and solve the resulting Euler-Lagrange two-point boundary value problem (see Expression 18), form the complete set of necessary conditions for the optimal control problem. Adding in the additional requirement that the corresponding matrix Riccati equation is bounded on , these form sufficient conditions for a weak local minimal optimal controller BY65 ; J70 . We discuss this sufficient condition in the sequel.

For simplicity, we refer to the optimal control as (rather than ) in the remainder of this paper and assume it is given by Equation 16. The adjoint dynamics are:


where is the Jacobian (with respect to ). Thus we have the Euler-Lagrange two-point boundary value problem:

Proposition 7.

If is an optimal control, then:


This follows from the transversality condition. ∎

From Equation 16, note that:




Simplifying we have:

If the Lie Bracket vanishes, i.e.,:


then this simplifies to:


and all co-state variables are eliminated. We have shown the following:

Theorem 8.

Consider the general optimal control problem given in Expression 14. If and is an optimal control, then the pair is a solution of the two point boundary value problem:


Geometrically, Equation 22 implies that the flows derived by the vector fields and commute locally. From a game-theoretic view, this means that locally evolutionary motion caused by competition in uncontrolled game commutes with evolutionary motion caused by the actuation payoffs on local space/time scales. As we see in the sequel, this is not true for actuated cyclic games, but is true for quasi-linear approximations of the evolutionary dynamics as in GB17 .

It is worth noting that a differential equation for the control function is derived in B78 , without the assumption of the vanishing Lie Bracket. However, without this assumption the system does not simplify in as useful a way and, in fact, in B78 the relevant Lie Bracket is not considered. Note, in formulating Theorem 8, we are assuming that solving the Euler-Lagrange equations will yield an optimal control. We can use the well known fact that a sufficient condition for optimality is the boundedness of the solution to the matrix Ricatti equation BY65 ; J70 to derive a complete necessary and sufficient condition for optimality of the control. Let:

Then the Matrix Ricatti equation is:


Here is the second differential operator with respect to the state and is an ordinary partial derivative, since there is only one control variable. When taken together with Lemma 6, the system of differential equations in Theorem 8 and the co-state dynamics, Equation 17, we have a complete characterization of the necessary and sufficient conditions for the optimal control. This yields the corollary:

Corollary 9 (Corollary to Theorem 8).

Let in the optimal control problem in Expression 14, with Hamiltonian . Any solution to the system of differential equations:


in which and is bounded for all constitutes a weak local optimal solution for Expression 14.

We note that this is the general analog of Proposition 2 in FG17 , which is specialized to a one-dimensional control problem. In general, checking the boundedness of the solution to the Matrix Ricatti equation must be done numerically. In the sequel we develop a simpler test for optimality using Mangasarian’s sufficiency condition; i.e., by checking that the Hamiltonian is jointly convex.

Problem 13 () does not satisfy the necessary condition that . However, a quasi-linearization of the problem does satisfy this condition (as in GB17 ). We now discuss a special case of Theorem 8 as well as extensions that apply to this quasi-linearized form.

3.2 The Quasi-Linear Case

For the remainder of this section, let and be arbitrary matrices of appropriate size, rather than the Jacobian matrices derived in Lemmas 3 and 4.

In Problem 14, let:


where is a (symmetric) positive definite matrix of appropriate size. We will add additional criteria to and as we proceed. We refer to this as a quasi-linear case because the only non-linearity arises from the interaction of the state and control variables. The following Corollary is immediate from Theorem 8:

Corollary 10.

If and is an optimal control, then the pair is a solution of the two point boundary value problem:


The condition that and commute is exactly the statement that the Lie Bracket of the vector fields in the dynamics vanishes. Therefore, Theorem 8 can be applied to any linear quadratic control problem where the state equation satisfies this condition.

We now derive some special results on and the optimal control in this quasi-linear case. Let and assume . Computing the second derivative of yields:

To simplify this, we will add an additional assumption to ; suppose that (i.e.,

is skew-symmetric) and

. Then:


Before proceeding note that:

Thus, we have the following proposition and its corollary:

Proposition 11.

If , and and , then:

  1. (30)
  2. (31)
Corollary 12.

For some constant ,


is the implicit closed-loop control, where must satisfy:


Furthermore the optimal control exists at time just in case:


4 Application to Complete Odd Circulant Games

We now return to the study of cyclic games with strategies and specifically to the control problem in Expression 13. As noted already, we cannot apply Theorem 8 directly to Problem 13 because the appropriate Lie Bracket does not vanish. However, we can construct the quasi-linearized form of the problem. Let . The quasi-linearized problem is:


In Expression 35, and are the Jacobian matrices of and as defined in Lemmas 3 and 4. Problem 35 is an instance of the general control problem studied in Section 3.

The following useful fact follows at once from Lemma 3.

Corollary 13.

The Jacobian matrix is skew-symmetric. ∎

Lemma 14.

Let be the optimal control for Problem 13. Then:

  1. The (open-loop) optimal control obeys the following differential equations:

  2. The following identity holds:




Problem 35 is an instance of Problem 14, but with quasi-linear system dynamics and quadratic objective as given in the quasi-linear conditions in Expression 27. In particular, Problem 35 sets . As a consequence the matrix . From Corollary 13, we know is skew-symmetric. Further, since the circulant matrices form a commutative algebra, we have . The lemma follows at once from Corollary 10, Proposition 11 and Corollary 12. ∎

Expression 38 is the closed-loop control law for the controlled cyclic game. Furthermore, Equation 36 allows us to determine some structural properties of .

Proposition 15.

The matrix is negative definite and therefore for all .


Consider any vector . Then:

Let be the upper-triangular matrix defined as: