Optimal strategies in the Fighting Fantasy gaming system: influencing stochastic dynamics by gambling with limited resource

02/24/2020 ∙ by Iain G. Johnston, et al. ∙ 0

Fighting Fantasy is a popular recreational fantasy gaming system worldwide. Combat in this system progresses through a stochastic game involving a series of rounds, each of which may be won or lost. Each round, a limited resource (`luck') may be spent on a gamble to amplify the benefit from a win or mitigate the deficit from a loss. However, the success of this gamble depends on the amount of remaining resource, and if the gamble is unsuccessful, benefits are reduced and deficits increased. Players thus dynamically choose to expend resource to attempt to influence the stochastic dynamics of the game, with diminishing probability of positive return. The identification of the optimal strategy for victory is a Markov decision problem that has not yet been solved. Here, we combine stochastic analysis and simulation with dynamic programming to characterise the dynamical behaviour of the system in the absence and presence of gambling policy. We derive a simple expression for the victory probability without luck-based strategy. We use a backward induction approach to solve the Bellman equation for the system and identify the optimal strategy for any given state during the game. The optimal control strategies can dramatically enhance success probabilities, but take detailed forms; we use stochastic simulation to approximate these optimal strategies with simple heuristics that can be practically employed. Our findings provide a roadmap to improving success in the games that millions of people play worldwide, and inform a class of resource allocation problems with diminishing returns in stochastic games.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

Abstract

Fighting Fantasy is a popular recreational fantasy gaming system worldwide. Combat in this system progresses through a stochastic game involving a series of rounds, each of which may be won or lost. Each round, a limited resource (‘luck’) may be spent on a gamble to amplify the benefit from a win or mitigate the deficit from a loss. However, the success of this gamble depends on the amount of remaining resource, and if the gamble is unsuccessful, benefits are reduced and deficits increased. Players thus dynamically choose to expend resource to attempt to influence the stochastic dynamics of the game, with diminishing probability of positive return. The identification of the optimal strategy for victory is a Markov decision problem that has not yet been solved. Here, we combine stochastic analysis and simulation with dynamic programming to characterise the dynamical behaviour of the system in the absence and presence of gambling policy. We derive a simple expression for the victory probability without luck-based strategy. We use a backward induction approach to solve the Bellman equation for the system and identify the optimal strategy for any given state during the game. The optimal control strategies can dramatically enhance success probabilities, but take detailed forms; we use stochastic simulation to approximate these optimal strategies with simple heuristics that can be practically employed. Our findings provide a roadmap to improving success in the games that millions of people play worldwide, and inform a class of resource allocation problems with diminishing returns in stochastic games.
Keywords: stochastic game; Markov decision problem; stochastic simulation; dynamic programming; resource allocation; stochastic optimal control; Bellman equation

Introduction

Fantasy adventure gaming is a popular recreational activity around the world. In addition to perhaps the best-known ‘Dungeons and Dragons’ system [Gygax and Arneson, 1974], a wide range of adventure gamebooks exist, where a single player takes part in an interactive fiction story [Costikyan, 2007]. Here, the reader makes choices that influence the progress through the book, the encounters that occur, and the outcomes of combats. In many cases, die rolls are used to provide stochastic influence over the outcomes of events in these games, particularly combat dynamics. These combat dynamics affect the game outcome, and thus the experience of millions of players worldwide, yet have rarely been studied in detail.

Here we focus on the stochastic dynamics of a particular, highly popular franchise of gamebooks, the Fighting Fantasy (FF) series [Green, 2014]. This series, spanning over 50 adventure gamebooks (exemplified by the famous first book in the series, ‘The Warlock of Firetop Mountain’ [Jackson and Livingstone, 2002]), has sold over million copies and given rise to a range of associated computer games, board games, and apps, and is currently experiencing a dramatic resurgence in popularity [BBC, 2014, Österberg, 2008]. Outside of recreation, these adventures are used as pedagogical tools in the huge industry of game design [Zagal and Lewis, 2015] and in teaching English as a foreign language [Philips, 1994].

In FF, a player is assigned statistics (skill and stamina), dictating combat proficiency and endurance respectively. Opponents are also characterised by these combat statistics. Combat proceeds iteratively through a series of ‘attack rounds’. In a given round, according to die rolls, the player may draw, win or lose, respectively. These outcomes respectively have no effect, damage the opponent, and damage the player. The player then has the option of using a limited resource (luck) to apply control to the outcome of the round. This decision can be made dynamically, allowing the player to choose a policy based on the current state of the system. However, each use of luck is a gamble [Maitra and Sudderth, 2012, Dubins and Savage, 1965], where the probability of success depends on the current level of the resource. If this gamble is successful, the player experiences a positive outcome (damage to the opponent is amplified; damage to the player is weakened). If the gamble is unsuccessful, the player experiences a negative outcome (damage to the opponent is weakened, damage to the player is amplified). The optimal strategy for applying this control in a given state has yet to be found.

This is a stochastic game [Shapley, 1953, Adlakha et al., 2015] played by one or two players (the opponent usually has no agency to decide strategies) on a discrete state space with a finite horizon. The game is Markovian: in the absence of special rules, the statistics of the player and opponent uniquely determine a system state, and this state combined with a choice of policy uniquely determine the transition probabilities to the next state. The problem of determining the optimal strategy is then a Markov decision problem (MDP) [Bellman, 1957, Kallenberg, 2003]. In an MDP, a decision-maker must choose a particular strategy for any given state of a system, which evolves according to Markovian dynamics. In FF combat, the decision is always binary: given a state, whether or not to use the dimishing resource of luck to attempt to influence the outcome of a given round.

The study of stochastic games and puzzles is long established in operational research [Smith, 2007, Bellman, 1965] and has led to several valuable and transferrable insights [Smith, 2007, Little et al., 1963]. Markov analysis, dynamic programming, and simulation have been recently used to explore strategies and outcomes in a variety of games, sports, and TV challenges [Lee, 2012, Smith, 2007, Johnston, 2016, Perea and Puerto, 2007, Percy, 2015, Clarke and Norman, 2003]. Specific analyses of popular one-player recreational games with a stochastic element including Solitaire [Rabb, 1988, Kuykendall and Mackenzie, 1999], Flip [Trick, 2001], Farmer Klaus and the Mouse [Campbell, 2002], Tetris [Kostreva and Hartman, 2003], and The Weakest Link [Thomas, 2003]. These approaches typically aim to identify the optimal strategy for a given state, and, in win/lose games, the overall probability of victory over all possible instances of the game [Smith, 2007]. In stochastic dynamic games, counterintuitive optimal strategies can be revealed through mathematical analysis, not least because ‘risking points is not the same as risking the probability of winning’ [Neller and Presser, 2004].

The FF system has some conceptual similarities with the well-studied recreational game Pig, and other so-called ‘jeopardy race games’ [Neller and Presser, 2004, Smith, 2007], where die rolls are used to build a score then a decision is made, based on the current state of the system, whether to gamble further or not. Neller & Presser have used a value iteration approach to identify optimal strategies in Pig and surveyed other similar games [Neller and Presser, 2004]. In FF combat, however, the player has potential agency both over their effect on the opponent and the opponent’s effect on them. Further, resource allocation in FF is a dynamic choice and also a gamble [Maitra and Sudderth, 2012, Dubins and Savage, 1965], the success probability of which diminishes as more resource is allocated. The probability of a negative outcome, as opposed to a positive one, therefore increases as more resource is used, providing an important ‘diminishing returns’ consideration in policy decision [Deckro and Hebert, 2003]. In an applied context this could correspond to engaging in, for example, espionage and counterespionage [Solan and Yariv, 2004], with increasing probability of negative outcomes with more engagement in these covert activites.

The optimal policy for allocating resource to improve a final success probability has been well studied in the context of research and development (R&D) management [Heidenberger and Stummer, 1999, Baye and Hoppe, 2003, Canbolat et al., 2012, Gerchak and Parlar, 1999]. While policies in this field are often described as ‘static’, where an initial ‘up-front’ decision is made and not updated over time, dynamic policy choices allowing updated decisions to be made based on the state of the system (including the progress of competitors) have also been examined [Blanning, 1981, Hopp, 1987, Posner and Zuckerman, 1990]. Rent-seeking ‘contest’ models [Clark and Riis, 1998] also describe properties of the victory probability as a function of an initial outlay from players. The ‘winner takes all’ R&D model of Canbolat et al., where the first player to complete development receives all the available payoff, and players allocate resource towards this goal [Canbolat et al., 2012], bears some similarity to the outcomes of the FF system. The model of Canbolat et al. did not allow dynamic allocation based on the current system state, but did allow a fixed cost to be spread over a time horizon, and computed Nash equilibria in a variety of cases under this model.

A connected branch of the literature considers how to allocate scarce resource to achieve an optimal defensive outcome [Golany et al., 2015, Valenzuela et al., 2015], a pertinent question both for human [Golany et al., 2009] and animal [Clark and Harvell, 1992] societies. Both optimisation and Nash equilibrium approaches are used in these contexts to identify solutions to the resource allocation problem under different structures [Golany et al., 2015, Valenzuela et al., 2015]. The FF system has such a defensive component, but the same resource can also be employed offensively, and as above takes the unusual form of a gamble with a diminishing success probability.

We will follow the philosophy of these optimisation approaches to identify the optimal strategy for allocating resource to maximise victory probability from a given state in FF. We first describe the system and provide solutions for victory probability in the case of no gambling and gambling according to a simple policy. Next, to account for the influence of different gambling policies on the stochastic dynamics of the game, we use a backwards-induction approach to solve the Bellman equation for the system and optimise victory probability from any given state. We then identify simple heuristic rules that approximate these optimised strategies and can be used in practise.

Game dynamics

Within an FF game, the player has nonnegative integer statistics called skill, stamina, and luck. skill and luck are typically ; stamina is typically , although these bounds are not required by our analysis. In a given combat, the opponent will also have skill and stamina statistics. We label the skill, stamina, and luck of the player (the ‘hero’) as and respectively, and the opponent’s skill and stamina as and . Broadly, combat in the FF system involves a series of rounds, where differences in skill between combatants influences how much stamina is lost in each round; when one combatant’s stamina reaches zero or below, the combat is over and that combatant has lost. The player may choose to use luck in any given round to influence the outcome of that round. More specifically, combat proceeds through Algorithm 1.

Algorithm 1. FF combat system.

  1. [nosep]

  2. Roll two dice and add ; this is the player’s attack strength .

  3. Roll two dice and add ; this is the opponent’s attack strength .

  4. If , this attack round is a draw. Go to 6.

  5. If , the player has won this attack round. Make decision whether to use luck.

    1. [noitemsep]

    2. If yes, roll two dice to obtain . If , set . If , set . For either outcome, set . Go to 6.

    3. If no, set . Go to 6.

  6. If , the opponent has won this attack round. Make decision whether to use luck.

    1. [noitemsep]

    2. If yes, roll two dice to obtain . If , set . If , set . For either outcome, set . Go to 6.

    3. If no, set . Go to 6.

  7. If and , the player has won; if and , the opponent has won. Otherwise go to 1.

It will readily be seen that these dynamics doubly bias battles in favour of the player. First, the opponent has no opportunity to use luck to their benefit. Second, when used offensively (to support an attack round that the player has won), the potential benefit (an additional 2 damage; Step 4a in Algorithm 1) outweighs the potential detriment (a reduction of 1 damage; Step 5a in Algorithm 1). When used defensively, this second bias is absent.

We will retain the skill, stamina, luck terminology throughout this analysis. However, the concepts here can readily be generalised. A player’s skill can be interpreted as their propensity to be successful in any given competitive round. Their stamina can be interpreted as the amount of competitive losses they can suffer before failure. luck is the resource that a player can dynamically allocate to gamble on influencing the outcome of competitive rounds, that diminishes with use (so that the success probability of the gamble also diminishes). For example, within the (counter)espionage analogy above, skill could be perceived as a company’s level of information security, stamina its potential to lose sensitive information before failure, and luck the resource that it can invest in (counter)espionage.

Analysis

In basic combat dynamics, skill does not change throughout a battle. The probabilities of winning, drawing, an losing an attack round depend only on skill, and remain constant throughout the battle. We therefore consider : the probability, at round , of being in a state where the player has stamina and luck , and the opponent has stamina . A discrete-time master equation can readily be written down describing the dynamics above:

(1)

Here, reflects the decision whether to use luck after a given attack round outcome (player loss or player win ), given the current state of the battle. This decision is free for the player to choose. and are respectively the probabilities of a successful and an unsuccessful test against luck . We proceed by first considering the case (no use of luck). We then consider a simplified case of constant luck, before using a dynamic programming approach to address the full general dynamics.

We are concerned with the victory probability with which the player is eventually victorious, corresponding to a state where and . We thus consider the ‘getting to a set’ outcome class of this stochastic game [Maitra and Sudderth, 2012], corresponding to a ‘winner takes all’ race [Canbolat et al., 2012]. We start with the probability of winning, drawing, or losing a given round. Let be the player’s skill and be the opponent’s skill. Let , …,

be random variables drawn from a discrete uniform distribution on

, and . Let . Then, the probabilities of win, draw, and loss events correspond respectively to .

is a discrete distribution on . The point density is proportional to the th sextinomial coefficient, defined via the generating function , for (specific values ; OEIS sequence A063260 [OEIS, 2019]). Hence

(2)
(3)
(4)

Dynamics without luck-based control

We first consider the straightforward case where the player employs no strategy, never electing to use luck. We can then ignore and consider steps through the stamina

space, which form a discrete-time Markov chain. Eqn.

1 then becomes

(5)

We can consider a combinatorial approach based on ‘game histories’ describing steps moving through this space [Maitra and Sudderth, 2012]. Here a game history is a string from the alphabet , with the character at a given position corresponding to respectively to a win, draw, loss in round . We aim to enumerate the number of possible game histories that correspond to a given outcome, and assign each a probability.

We write for the character counts of in a given game history. A victorious game must always end in . Consider the string describing a game history omitting this final . First leaving out s, we have s and s that can be arranged in any order. We therefore have possible strings, each of length . For completeness, we can then place any number of s within these strings, obtaining

(6)

Write and , describing the number of rounds each character can lose before dying. Then, for a player victory, and . can take any nonnegative integer value. The appearance of each character in a game string is accompanied by a multiplicative factor of the corresponding probability, so we obtain

(7)

where the probability associated with the final character has now also been included. After some algebra, and writing and , this expression becomes

(8)

Eqn. 8 is compared with the result of stochastic simulation in Fig. 1, and shown for various skill differences and initial conditions. Intuitively, more favourable increase and less favourable decrease for any given state, and discrepancies between starting and also influence eventual . A pronounced structure is observed, as in the absence of luck, is functionally equivalent to for integer . For lower initial staminas, distributions become more sharply peaked with values, as fewer events are required for an eventual outcome.

Figure 1: Victory probability in the absence of luck-based strategy. (i) Comparison of predicted victory probability from Eqn. 8 with stochastic simulation. (ii) behaviour as skill difference changes.

Analytic dynamics with simplified luck-based control

To increase the probability of victory beyond the basic case in Eqn. 8, the player can elect to use luck in any given round. We will first demonstrate that the above history-counting analysis can obtain analytic results when applied to a simplified situation when luck is not depleted by use, so that the only limit on its employment is its initial level [Blanning, 1981]. We then employ a dynamic programming scheme to analyse the more involved general case.

First consider the case where luck is only used offensively (step 4a in Algorithm 1), and is used in every successful attack round. For now, ignore losses and draws. Then every game history consists of s and s, where is a successful offensive use of luck and is an unsuccessful offensive use of luck.

Consider the game histories that lead to the opponent losing exactly stamina points before the final victorious round. There are string lengths that can achieve this, which are , where runs from to . The strings with a given involve failures and successes.

If we make the simplifying assumption that luck is not depleted with use, every outcome of a luck test has the same success probability . Then the problem is simplified to finding the number of ways of arranging s and s for each possible string:

(9)

Now, for every string with a given , with corresponding string length , we can place s and s as before, giving

(10)

The complete history involves a final victorious round. For now we will write the probability of this event as , then the probability associated with this set of histories is

(11)

and, as ,

(12)

where . Hence

(13)

Now consider the different forms that the final victorious round can take. The opponent’s stamina can be reduced to 4 followed by an , 3 followed by , 2 followed by , or 1 followed by or . If we write for the probability of reducing the opponent’s stamina to then finishing with event ,

(14)

hence

(15)

Fig. 2(i) compares Eqn. 15 and stochastic simulation, and shows that use of luck can dramatically increase victory probability in a range of circumstances. Similar expressions can be derived for the defensive case, where luck is solely used when a round is lost, and with some relaxations on the structure of the sums involved the case where luck is not used in every round can also be considered.

Clearly, for constant luck, the optimal strategy is to always use luck when the expected outcome is positive, and never when it is negative. The expected outcomes for offensive and defensive strategies can readily be computed. For offensive use, the expected damage dealt when using luck is

(16)

In the absence of luck, damage is dealt, so the expected outcome using luck is beneficial if . We therefore obtain as the criterion for employing luck. For comparison, the probability of scoring 6 or under on two dice is and the probability of scoring 5 or under on two dice is .

For defensive use, the expected damage received when using luck in an attack round is

(17)

In the absence of luck, damage is taken, and we thus now obtain as our criterion. For comparison, the probability of scoring 7 or under on two dice is .

The counting-based analyses above assume that luck stays constant. The dynamics of the game actually lead to luck diminishing every time is it is used. Instead of the outcome of a luck test being a constant , it now becomes a function of how many tests have occurred previously.

To explore this situation, consider the above case where luck is always used offensively and never defensively. Describe a given string of luck outcomes as an ordered set of indices labelling where in a string successes occur. For example, the string of length with would be . Let . Then

(18)

where is the probability of success on the th luck test.

As above, for a given , , and we let be the set of ordered sets with different elements between 1 and . Then the probability of a given string of s and s arising is

(19)

We can no longer use the simple counting argument in Eqn. 11 to compute the probabilities of each history, because each probability now depends on the specific structure of the history. It will be possible to enumerate these histories exhaustively but the analysis rapidly expands beyond the point of useful interpretation, so we turn to dynamic programming to investigate the system’s behaviour.

Figure 2: Victory probability with constant luck employed offensively. (i) Eqn. 15 compared to stochastic simulation. (ii) Constant luck outcomes for intermediate and low luck, and for high luck mitigating low values.

Stochastic optimal control with dynamic programming

In a game with a given , we characterise every state of the system with a tuple where is the outcome (win or loss) of the current attack round. The question is, given a state, should the player elect to use luck or not?

A common approach to identify the optimal strategy for a Markov decision problem in a discrete state space is to use the Bellman equation [Bellman, 1957, Kirk, 2012], which in our case is simply

(20)

Here, is a strategy dictating what action to take in state , is the probability under strategy of the transition from state to state , and is the probability-to-victory of state . The joint problem is to compute the optimal , and the strategy that maximises it, for all states. To do so, we employ a dynamic programming approach of backward induction [Bellman, 1957], starting from states where is known and computing backwards through potential precursor states.

Figure 3: Dynamic programming scheme. The discrete state space of the game is represented by horizontal () and vertical () axes, layers () and a binary outcome at each position (). (i) First, is assigned to states corresponding to loss (, blue) and is assigned to states corresponding to victory (, red). Then, iteratively, all states where all outcomes have an assigned are considered. (ii) Considering outcomes after a loss in the current state (here, , highlighted red). The probability-to-victory for a ‘no’ strategy corresponds to the solid line; the probability-to-victory for a ‘yes’ strategy consists of contributions from both dashed lines (both outcomes from a luck test). (iii) Same process for a win outcome from the current state. (iv) Now the previous node has characterised probabilities-to-victory for both loss and win outcomes, a next iteration of nodes can be considered. Now, the probability-to-victory from a ‘yes’ strategy has a term corresponding to the potential outcomes from the previously characterised node.

The dynamic programming approach first assigns a probability-to-victory for termination states (Fig. 3(i)). For all ‘defeat’ states with , we set ; for all ‘victory’ states with , set (states where both and are inaccessible). We then iteratively consider all states in the system where is fully determined for a loss outcome from the current round (Fig. 3(ii)), and similarly for a win outcome (Fig. 3(iii)).

For states involving a loss outcome, we compute two probability propagators. The first corresponds to the strategy where the player elects to use luck, and is of magnitude

(21)

the second corresponds to the strategy where the player does not use luck, and is

(22)

When considering the probability-to-victory for a given state, we hence consider both the next combination that a given event will lead to, and also both possible outcomes (win or loss) from this state. For a given state, if , we record the optimal strategy as using luck and record ; otherwise we record the optimal strategy as not to use luck and record . In practise we replace with the condition to avoid numerical artefacts, thus requiring that the use of luck has a relative advantage to above .

We do the same for states involving a win outcome from this round, where the two probability propagators are now

(23)

and

(24)

Each new pair of states for which the optimal is calculated opens up the opportunity to compute for new pairs of states (Fig. 3(iv)). Eventually a and optimal strategy is computed for each outcome, providing a full ‘roadmap’ of the optimal decision to make under any circumstance. This full map is shown in Supplementary Fig. 6, with a subset of states shown in Fig. 4.

Figure 4: Optimal strategies and victory probabilities throughout state space. The optimal strategy at a given state is given by the number at the corresponding point (1 – use luck regardless of the outcome of this round; 2 – use luck if this round is lost; 3 – use luck if this round is won). No number means that the optimal strategy is not to employ luck regardless of outcome. Colour gives victory probability .
Figure 5: Summary of optimal strategies and victory probabilities throughout state space. Arrows depict regions where the optimal policy depends on current available luck, including the general change in that region with decreasing luck. Empty regions are those where avoiding the use of luck is the optimal policy. Other policies: (i) Hero stamina : use luck defensively (but see (iv), (v)); (ii) Opponent stamina : use luck defensively; (iii) Use luck offensively, particularly when opponent stamina ; (iv) Use luck offensively and defensively; (v) Use luck defensively, and offensively if luck is high and opponent stamina or .

We find that a high luck score and judicious use of luck can dramatically enhance victory probability against some opponents. As an extreme example, with a skill detriment of , , use of luck increased victory probability by a factor of , albeit to a mere . On more reasonable scales, with a skill detriment of , , use of luck increased victory probability 1159-fold, from to 0.011 (the highest fold increase with the final probability greater than 0.01). With a skill detriment of , , use of luck increased victory probability 21-fold, from 0.010 to 0.22 (the highest fold increase with the initial probability greater than 0.01). Using encounters that appear in the FF universe [Gascoigne, 1985], for a player with maximum statistics , optimal use of luck makes victory against an adult White Dragon () merely quite unlikely () rather than implausible (), and victory against a Hell Demon () fairly straightforward () rather than unlikely ().

Structure of optimal policy space

There is substantial similarity in optimal policy choice between several regions of state space. For large skill deficiencies (low probability of victory) the distribution of optimal strategies in stamina space is the same for a given for all . For higher , this similarity continues to higher ; for , only 5 points in stamina space have different optimal strategies for and . At more reasonable victory probabilities, a moderate transition is apparent between and , where the number of points in stamina space where the optimal strategy involves using luck decreases noticeably. This reflects the lower expected advantage for (Eqns. 16-17).

The interplay of several general strategies is observed in the optimal structures. First note that gives the number of hits required for defeat if a hit takes 4 stamina points (a successful offensive luck test) and gives the number of hits required for defeat in the absence of strategy. These scales partition stamina space by the number of rounds required for a given outcome and hence dictate several of the ‘banded’ structures observable in strategy structure. For example, at , it is very advantageous for the player to attempt to mitigate the effect of losing another round. Almost all circumstances display a band of defensive optimal strategy at .

At , a successful offensive luck test is very advantageous (immediate victory). An unsuccessful offensive test, leading to , is not disadvantageous to the same extent: we still need exactly one successful attack round without luck, as we would if we had not used luck and achieved instead. A strip of strategy 3 (or strategy 1) at is thus the next most robustly observed feature, disappearing only when victory probability is already overwhelmingly high. Many other structural features result from a tradeoff between conserving luck and increasing the probability of encountering this advantageous region. An illustration of the broad layout of optimal strategies is shown in Fig. 5; a more fine-grained analysis is provided in the Appendix.

Simulated dynamics with heuristic luck strategies

While the dynamic programming approach above gives the optimal strategy for any circumstance, the detailed information involved does not lend itself to easy memorisation. As in Smith’s discussion of solitaire, ‘the curse of dimensionality applies for the state description, and most wise players use heuristics’

[Smith, 2007]. We therefore consider, in addition to the semi-quantitative summary in Fig. 5, coarse-grained quantitative ‘strategies’ that, rather than specifying an action for each branch of the possible tree of circumstances, use a heuristic rule that is applied in all circumstances.

Among the simplest strategies are ‘open-loop’ rules, where the current state of the player’s and the opponent’s stamina does not inform the choice of whether to use luck. We may also consider ‘feedback’ rules, where the current state of staminas informs this choice. We parameterise luck strategies by a threshold ; luck will only be used if the player’s current luck score is greater than or equal to .

We consider the following luck strategies:

  1. [noitemsep]

  2. Never use luck ( as above);

  3. Use luck in every non-draw round (i.e. to both ameliorate damage and enhance hits, );

  4. Use luck defensively (i.e. only to ameliorate damage, );

  5. Use luck offensively (i.e. only to enhance hits, );

  6. Use luck defensively if and ;

  7. Use luck offensively if and ;

  8. Use luck if and ;

  9. Use luck if and ;

  10. Use luck if and .

Strategy 0 is the trivial absence of luck use; strategies 1-3 are ‘open-loop’ and 4-8 are ‘feedback’. Strategies 2 and 4 are defensive; strategies 3 and 5 are offensive; strategies 1, 6, 7, 8 are both offensive and defensive.

The analytic structure determined above is largely visible in the simulation results (Supplementary Fig. 7). The main differences arise from the construction that the same strategy is retained throughout combat in the simulation study. For low , high , the favouring of strategy 3 except at low (strategy 1) and (strategy 2) is clear. At low , some feedback strategies gain traction: for example, strategy 8, which is identical to strategy 1 when is high and , but avoids depleting luck if ever , is more conservative than always using strategy 1. Similarly, strategy 5 provides a feedback alternative to strategy 3, using luck only at , which is favoured at intermediate and high where it is beneficial to preserve luck. These feedback-controlled options mimic the state-specific application of strategy found in the optimal cases in Fig. 4.

Decreasing introduces the sparsification of the strategy 3 regions – an effect which, as with the analysis, is more pronounced at higher . In simulations, at higher , the increasingly small relative increases in victory probability mean that several regions do not display substantial advantages to using strategy. However, at moderate to positive , the increased takeover of strategy 2 is observed for high , low . Strategy 4, a more conservative feedback version of strategy 2, also experiences support in these regions, competing with strategy 2 particularly in bands where , as described in the analytic case. The above banded structures of strategies 2 and 3, and the ubiquitous strategy 2 (or equivalent) at , are also observed for intermediate .

Under these simplified rulesets, what is the optimal threshold above which we should use luck? Straightforward application of Eqns. 16-17 are supported by simulation results (Supplementary Fig. 8), where the probability that a given appears in the highest strategy peaked at for strategy 1 and for strategy 3.

A heuristic approach harnessing these insights, at the broad level of Fig. 5, might be as follows. Employ strategy 3 (offensive luck use), unless: (i) initial luck is high or is low, in which case strategy 1 (unconditional luck use); (ii) and (i) is not true, in which case strategy 2 (defensive luck use); or (iii) and and (i) is not true, in which case strategy 2 (defensive luck use). At the next level of detail, feedback control in the form of strategies 4-8 may be beneficial in some cases, notably when luck depletion is likely to become an issue; regions where this is likely to be the case can be read off from Supplementary Fig. 7. Finally, the optimal strategy for any given state can be read off from Supplementary Fig. 6.

Discussion

We have examined the probability of victory in an iterated, probabilistic game that plays a central role in a well-known and widespread interactive fiction series. The game can be played with or without ‘strategy’, here manifest by the consumption of a limited resource to probabilistically influence the outcome of each round.

Several interesting features of the FF combat system make it potentially noteworthy with respect to similar systems [Canbolat et al., 2012, Neller and Presser, 2004, Smith, 2007]. The allocation of resource is dynamic and depends on system state [Blanning, 1981, Hopp, 1987, Posner and Zuckerman, 1990]. The use of resource can both increase the probability of a positive outcome for the player, or a negative outcome for the opponent. Use of this resource does not guarantee a positive outcome: its use is a gamble [Maitra and Sudderth, 2012, Dubins and Savage, 1965] that may negatively affect the player. The probability of this negative effect increases as more resource is used, providing an important consideration in the decision of whether to invoke this policy in the face of ‘diminishing returns’ [Deckro and Hebert, 2003].

The fact that gambles are dynamically used to influence iterated stochastic dynamics complicates the analysis of the game. To employ dynamic programming, state space was labelled both by the current statistics of the players and by the outcome of the current stochastic round. The policy decision of whether to gamble thus depends both on current statistics and the outcome of a round.

This analysis reveals several structural properties of the system that are not specific to the FF context of this study. The case of a limited resource being gambled against to both amplify successes and mitigate failures (with the probability of a positive outcome diminishing as resource is used) bears some resemblance to, for example, a conceptual picture of espionage and counterespionage [Solan and Yariv, 2004]. For example, resource could be allocated to covert operations with some probability of amplifying thefts and mitigating losses of information, with increased probability of negative outcomes due to discovery as these covert activites are employed more. Our analysis then reveals the strategies to be employed in different scenarios of information security (skill) and robustness to information loss (stamina).

In the absence of strategy, we find a closed-form expression for victory probability that takes an intuitive form. When strategy is included, dramatic increases in victory probability are found. The strong advantages provided by successful use of resource towards the ‘endgame’, where a successful gamble will produce instant victory or avoid instant defeat, shapes the structure of the optimal policy landscape. When little resource is available, complex structures emerge in the optimal landscape that depend on the tradeoff between using resource in the current state or ‘saving’ it in case of a more beneficial state later (‘risking points is not the same as risking the probability of winning’ [Neller and Presser, 2004]). When default victory is unlikely, using resource to reinforce rare success probabilities is a favoured strategy; when default victory is likely, using resource to mitigate rare loss probabilities is favoured. The specific optimal policy in a given state is solved and can be reasonably approximated by more heuristic strategies [Smith, 2007]. Interestingly, there is little performance loss when these heuristics are ‘open-loop’, in the sense that policy choice only depends on a round’s outcome and coarsely on the amount of current resource. ‘Feedback’ strategies additionally based on the statistics of the two players did not provide a substantial advantage as long as endgame dynamics were covered.

Numerous FF gamebooks embellish the basic combat system, where weapons, armour, and other circumstances led to different stamina costs or different rules. In the notoriously hard ‘Crypt of the Sorcerer’ [Livingstone, 1987], if the final opponent (with ) wins two successive rounds the player is instantly killed, altering the Markovian nature of the combat system (and substantially decreasing victory probability). A simulation study [Fitti, 2016], while not employing an optimised luck

strategy, estimated a 95% probability of this instant death occurring (and a 0.11% probability of overall victory through the book). Further expansion of this analysis will generalise the potential rulesets, and hence allow the identification of optimal strategies in more situations.

Another route for expansion involves optimising victory probability while preserving some statistics, for example enforcing that or at victory, so that some resource is retained for the rest of the adventure after this combat. Such constraints could readily be incorporated through an initial reallocating over system states in the dynamic programming approach (Fig. 3(i)), or by expanding the definition of the score being optimised to include some measure of desired retention in addition to .

In an era of artificial intelligence approaches providing effective but essentially uninterpretable strategies for complex games

[Campbell et al., 2002, Lee et al., 2016, Gibney, 2016], more targetted analyses still have the potential to inform more deeply about the mechanisms involved in these strategies. Further, mechanistic understanding makes successful strategies readily available and simple to implement in the absence of computational resource. We hope that this increased interpretability and accessibility both contribute to the demonstration of the general power of these approaches, and help improve the experience of some of the millions of FF players worldwide.

Acknowledgments

The author is grateful to Steve Jackson, Ian Livingstone, and the many other FF authors for creating this system and the associated worlds. The author also thanks Daniel Gibbs and Ellen Røyrvik for productive discussions.

References

  • [Adlakha et al., 2015] Adlakha, S., Johari, R., and Weintraub, G. Y. (2015). Equilibria of dynamic games with many players: Existence, approximation, and market structure. Journal of Economic Theory, 156:269–316.
  • [Baye and Hoppe, 2003] Baye, M. R. and Hoppe, H. C. (2003). The strategic equivalence of rent-seeking, innovation, and patent-race games. Games and economic behavior, 44(2):217–226.
  • [BBC, 2014] BBC (2014). Retrieved from https://www.bbc.co.uk/news/magazine-28865399, june 2019.
  • [Bellman, 1957] Bellman, R. (1957). A Markovian decision process. Journal of Mathematics and Mechanics, pages 679–684.
  • [Bellman, 1965] Bellman, R. (1965). On the application of dynamic programing to the determination of optimal play in chess and checkers. Proceedings of the National Academy of Sciences of the United States of America, 53(2):244.
  • [Blanning, 1981] Blanning, R. W. (1981). Variable-base budgeting for r&d. Management Science, 27(5):547–558.
  • [Campbell et al., 2002] Campbell, M., Hoane Jr, A. J., and Hsu, F.-h. (2002). Deep blue. Artificial intelligence, 134(1-2):57–83.
  • [Campbell, 2002] Campbell, P. J. (2002). Farmer klaus and the mouse. The UMAP Journal, 23(2):121–134.
  • [Canbolat et al., 2012] Canbolat, P. G., Golany, B., Mund, I., and Rothblum, U. G. (2012). A stochastic competitive r&d race where “winner takes all”. Operations Research, 60(3):700–715.
  • [Clark and Harvell, 1992] Clark, C. W. and Harvell, C. D. (1992). Inducible defenses and the allocation of resources: a minimal model. The American Naturalist, 139(3):521–539.
  • [Clark and Riis, 1998] Clark, D. J. and Riis, C. (1998). Contest success functions: an extension. Economic Theory, 11(1):201–204.
  • [Clarke and Norman, 2003] Clarke, S. R. and Norman, J. M. (2003). Dynamic programming in cricket: Choosing a night watchman. Journal of the Operational Research Society, 54(8):838–845.
  • [Costikyan, 2007] Costikyan, G. (2007). Games, storytelling, and breaking the string. Second Person: Roleplaying and Story in Playable Media, pages 5–14.
  • [Deckro and Hebert, 2003] Deckro, R. F. and Hebert, J. E. (2003). Modeling diminishing returns in project resource planning. Computers & industrial engineering, 44(1):19–33.
  • [Dubins and Savage, 1965] Dubins, L. E. and Savage, L. J. (1965). Inequalities for stochastic processes: How to gamble if you must. Dover Publications.
  • [Fitti, 2016] Fitti (2016). Retrieved from https://www.youtube.com/watch?v=-kMwT8QWZnw, june 2019.
  • [Gascoigne, 1985] Gascoigne, M. (1985). Out of the Pit. Puffin Books.
  • [Gerchak and Parlar, 1999] Gerchak, Y. and Parlar, M. (1999). Allocating resources to research and development projects in a competitive environment. Iie Transactions, 31(9):827–834.
  • [Gibney, 2016] Gibney, E. (2016). Google AI algorithm masters ancient game of Go. Nature News, 529(7587):445.
  • [Golany et al., 2015] Golany, B., Goldberg, N., and Rothblum, U. G. (2015). Allocating multiple defensive resources in a zero-sum game setting. Annals of Operations Research, 225(1):91–109.
  • [Golany et al., 2009] Golany, B., Kaplan, E. H., Marmur, A., and Rothblum, U. G. (2009). Nature plays with dice–terrorists do not: Allocating resources to counter strategic versus probabilistic risks. European Journal of Operational Research, 192(1):198–208.
  • [Green, 2014] Green, J. (2014). You are the Hero. Snowbooks Ltd.
  • [Gygax and Arneson, 1974] Gygax, G. and Arneson, D. (1974). Dungeons and dragons, volume 19. Tactical Studies Rules Lake Geneva, WI.
  • [Heidenberger and Stummer, 1999] Heidenberger, K. and Stummer, C. (1999). Research and development project selection and resource allocation: a review of quantitative modelling approaches. International Journal of Management Reviews, 1(2):197–224.
  • [Hopp, 1987] Hopp, W. J. (1987). A sequential model of r&d investment over an unbounded time horizon. Management Science, 33(4):500–508.
  • [Jackson and Livingstone, 2002] Jackson, S. and Livingstone, I. (2002). The Warlock of Firetop Mountain. Wizard Books.
  • [Johnston, 2016] Johnston, I. G. (2016). Endless love: On the termination of a playground number game. Recreational Mathematics Magazine, 3(5):61–78.
  • [Kallenberg, 2003] Kallenberg, L. (2003). Finite state and action MDPs. In

    Handbook of Markov decision processes

    , pages 21–87. Springer.
  • [Kirk, 2012] Kirk, D. E. (2012). Optimal control theory: an introduction. Courier Corporation.
  • [Kostreva and Hartman, 2003] Kostreva, M. M. and Hartman, R. (2003). Multiple objective solutions for tetris. Journal of Recreational Mathematics, 32(3):214.
  • [Kuykendall and Mackenzie, 1999] Kuykendall, C. and Mackenzie, D. (1999). Analyzing solitaire. Science, 283(5403):794–794.
  • [Lee et al., 2016] Lee, C.-S., Wang, M.-H., Yen, S.-J., Wei, T.-H., Wu, I.-C., Chou, P.-C., Chou, C.-H., Wang, M.-W., and Yan, T.-H. (2016). Human vs. computer go: Review and prospect [discussion forum]. IEEE Computational intelligence magazine, 11(3):67–72.
  • [Lee, 2012] Lee, J. D. (2012). The Comparison of Strategies used in the game of RISK via Markovian Analysis and Monte-Carlo Simulation. Technical Report AFIT/IOA/ENS/12-02, Air Force Inst of Tech Wright-Patterson AFB OH Graduate School of Engineering and Management.
  • [Little et al., 1963] Little, J. D., Murty, K. G., Sweeney, D. W., and Karel, C. (1963). An algorithm for the traveling salesman problem. Operations research, 11(6):972–989.
  • [Livingstone, 1987] Livingstone, I. (1987). Crypt of the Sorcerer. Puffin Books.
  • [Maitra and Sudderth, 2012] Maitra, A. P. and Sudderth, W. D. (2012). Discrete gambling and stochastic games, volume 32. Springer Science & Business Media.
  • [Neller and Presser, 2004] Neller, T. W. and Presser, C. G. (2004). Optimal play of the dice game pig. The UMAP Journal, 25(1).
  • [OEIS, 2019] OEIS (2019). The On-Line Encyclopedia of Integer Sequences, published electronically at https://oeis.org.
  • [Österberg, 2008] Österberg, A. (2008). The Rise and Fall of the Gamebook. Retrieved from http://www.outspaced.fightingfantasy.net/Hosted/Anders_-_The_Rise_and_Fall_of_the_Gamebook.pdf, June 2019.
  • [Percy, 2015] Percy, D. F. (2015). Strategy selection and outcome prediction in sport using dynamic learning for stochastic processes. Journal of the Operational Research Society, 66(11):1840–1849.
  • [Perea and Puerto, 2007] Perea, F. and Puerto, J. (2007). Dynamic programming analysis of the tv game “who wants to be a millionaire?”. European Journal of Operational Research, 183(2):805–811.
  • [Philips, 1994] Philips, B. (1994). Role-playing games in the English as a Foreign Language Classroom. Proceedings of the Tenth National Conferences on English Teaching and Learning in the Republic of China, pages 624–648.
  • [Posner and Zuckerman, 1990] Posner, M. and Zuckerman, D. (1990). Optimal r&d programs in a random environment. Journal of Applied Probability, 27(2):343–350.
  • [Rabb, 1988] Rabb, A. (1988). A probabilistic analysis of the game of solitaire. Undergraduate honors thesis, Harvard University.
  • [Shapley, 1953] Shapley, L. S. (1953). Stochastic games. Proceedings of the national academy of sciences, 39(10):1095–1100.
  • [Smith, 2007] Smith, D. K. (2007). Dynamic programming and board games: A survey. European Journal of Operational Research, 176(3):1299–1318.
  • [Solan and Yariv, 2004] Solan, E. and Yariv, L. (2004). Games with espionage. Games and Economic Behavior, 47(1):172–199.
  • [Thomas, 2003] Thomas, L. (2003). The best banking strategy when playing the weakest link. Journal of the Operational Research Society, 54(7):747–750.
  • [Trick, 2001] Trick, M. A. (2001). Building a better game through dynamic programming: A flip analysis. INFORMS Transactions on Education, 2(1):50–58.
  • [Valenzuela et al., 2015] Valenzuela, M., Szidarovszky, F., and Rozenblit, J. (2015). A multiresolution approach for optimal defense against random attacks. International Journal of Information Security, 14(1):61–72.
  • [Zagal and Lewis, 2015] Zagal, J. P. and Lewis, C. (2015). Fighting Fantasies: Authoring RPG Gamebooks for Learning Game Writing and Design. Proceedings of the 2015 RPG Summit-DiGRA 2015 Conference: Diversity of Play.

Appendix

Detailed structure of optimal policy space

Lower . In this regime, for , there is a band of strategy 2, reflecting the fact that luck is best spent attempting to avoid the fatality of a lost attack round. For certain regions, strategy 3 is optimal. For example, when is low, strategy 3 appears when or . Here, the probability of winning an attack round is very low, so even a small chance of a successful luck test amplifying the outcome to a killing blow is worth taking. These bands of strategy 3 propagate as increases, occupying strips between and . One value is omitted from each band, at . It will be noticed that these values are those where, if luck is not employed, a successful round () will decrease , whereas if luck is employed and is not successful (), that next band will not be reached. When luck is a limited resource it does not pay to ‘spend’ it in these cases.

Where strategy 2 and 3 bands overlap, intuitively, strategy 1 appears, using luck for either outcome. As increases further () other regions of stamina space display a preference for strategy 2, reflecting the fact that substantial luck is now available relative to the likely number of remaining attack rounds, so that is pays to invest luck defensively as well as offensively. As increases further still, the expanding regions of strategy 2 overlap more with existing strategy 3 regions, leading to an expansion of strategy 1 in low regions.

Column-wise, beginning at , some horizontal bands in the the strategy 3 region sparsify into bands. This banding reflects the ubiquitous presence of strategy 2 when , to potentially defer the final fatal loss and allow one more attack round. If , this region will not be reached unless the player uses a (non-optimal) defensive strategy elsewhere, so luck can be freely invested in offensive strategy. If , the battle has some probability (high if is low) of encountering the state, so there is an advantage to preserving luck for this circumstance.

Higher . Patterns of strategies 2-3 remains largely unchanged as increases, until the regions of strategy 3 at lower start to become sparsified, breaking up both row-wise and column-wise. Row-wise breaks, as above, occur at , so that low luck is not used if the outcome will not cross a band. Eventually these breaks are joined by others, so that for e.g. , strategy 3 only appears in bands of . Here, a successful luck test shifts the system to the next band, while an unsuccessful luck test means that the next band can be reached with a successful attack round without using luck. Column-wise, strategy 2 expands in bands at higher , and the ongoing sparsification of strategy 3 regions for low as increases.

For higher , as increases, the sparsification of strategy 3 regions occurs in tandem with an expansion of strategy 2 regions. Strategy 2 becomes particularly dominant in high , low regions. Here, the player expects to win most attack rounds, and the emphasis shifts to minimising losses from the rare attack rounds that the opponent wins, so that the player has time to win without their stamina running out. A banded structure emerges (for example, ) where for strategy 2 is favoured and for strategies 1 and 3 are favoured. This structure emerges because of the relatively strong advantage of a successful luck test at and , leading to immediate victory, and the propagating advantage of successful luck tests that lead to this region (wherever ). In the absence of this advantage (), the best strategy is to minimise damage from rare lost rounds until the advantageous bands are obtained by victories without depleting luck.

At higher yet , victory is very likely, and the majority of stamina space for high is dominated by the defensive strategy 2 – minimising the impact of the rare attack rounds where the opponent wins. At intermediate , strategies 2 and 3 appear in banded formation, where defensive luck use is favoured when is even, to increase the number of steps needed to reach .

Supplementary Figure 6: (magnification recommended) Complete set of optimal strategies for all states. The optimal strategy for any given state (shero), (sopp), (lhero), (kdiff). 1 – use luck regardless of the outcome of this round; 2 – use luck if this round is lost; 3 – use luck if this round is won. No number means that the optimal strategy is not to employ luck regardless of outcome. Colour gives victory probability .
Supplementary Figure 7: Simulated heuristic successful strategies and victory probabilities throughout state space. Here, a given strategy is used in every round until combat ends. The strategy with the most positive effect on in stochastic simulation for a given state is given by the number at the corresponding point (labels in the main text). No number means that no strategy had a positive influence on . Colour gives victory probability .
Supplementary Figure 8: Probability of luck threshold appearing in an optimal strategy. Histogram shows values appearing in heuristic approaches to optimise in stochastic simulation, featuring strategies 1-3 from the text.