# Duty to Warn in Strategic Games

The paper investigates the second-order blameworthiness or duty to warn modality "one coalition knew how another coalition could have prevented an outcome". The main technical result is a sound and complete logical system that describes the interplay between the distributed knowledge and the duty to warn modalities.

## Authors

• 18 publications
• 8 publications
• ### Intelligence in Strategic Games

The article considers strategies of coalitions that are based on intelli...
10/16/2019 ∙ by Pavel Naumov, et al. ∙ 0

• ### Comprehension and Knowledge

The ability of an agent to comprehend a sentence is tightly connected to...
12/11/2020 ∙ by Pavel Naumov, et al. ∙ 0

• ### Blameworthiness in Strategic Games

There are multiple notions of coalitional responsibility. The focus of t...
09/14/2018 ∙ by Pavel Naumov, et al. ∙ 0

• ### The Limits of Morality in Strategic Games

A coalition is blameable for an outcome if the coalition had a strategy ...
01/22/2019 ∙ by Rui Cao, et al. ∙ 0

• ### Knowledge and Blameworthiness

Blameworthiness of an agent or a coalition of agents is often defined in...
11/05/2018 ∙ by Pavel Naumov, et al. ∙ 0

• ### Strategic Coalitions with Perfect Recall

The paper proposes a bimodal logic that describes an interplay between d...
07/13/2017 ∙ by Pavel Naumov, et al. ∙ 0

• ### Strategic Coalitions in Stochastic Games

The article introduces a notion of a stochastic game with failure states...
10/10/2019 ∙ by Pavel Naumov, et al. ∙ 0

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1. Introduction

On October 27, 1969, Prosenjit Poddar, an Indian graduate student from the University of California, Berkeley, came to the parents’ house of Tatiana Tarasoff, an undergraduate student who recently immigrated from Russia. After a brief conversation, he pulled out a gun and unloaded it into her torso, then stabbed her eight times with a 13-inch butcher knife, walked into the house and called the police. Tarasoff was pronounced dead on arrival at the hospital a17timeline.

In this paper we study the notion of blameworthiness. This notion is usually defined through the principle of alternative possibilities: an agent (or a coalition of agents) is blamable for if is true and the agent had a strategy to prevent it f69tjop; w17. This definition is also referred to as the counterfactual definition of blameworthiness c15cop. In our case, Poddar is blamable for the death of Tatiana because he could have taken actions (to refrain from shooting and stabbing her) that would have prevented her death. He was found guilty of second-degree murder and sentenced to five years a17timeline. The principle of alternative possibilities is also a part of Halpern-Pearl definition of causality as a relation between sets of variables h16. A sound and complete axiomatization of modality “statement is true and coalition had a strategy to prevent ” is proposed in nt19aaai. In related works, Xu x98jpl and Broersen, Herzig, and Troquard bht09jancl axiomatized modality “took actions that unavoidably resulted in ” in the cases of single agents and coalitions respectively.

According to the principle of alternative possibilities, Poddar is not the only one who is blamable for Tatiana’s death. Indeed, Tatiana’s parents could have asked for a temporary police protection, hired a private bodyguard, or taken Tatiana on a long vacation outside of California. Each of these actions is likely to prevent Tatiana’s death. Thus, by applying the principle of alternative possibilities directly, we have to conclude that her parents should be blamed for Tatiana’s death. However, the police is unlikely to provide life-time protection; the parents’ resources can only be used to hire a bodyguard for a limited period time; and any vacation will have to end. These measures would only work if they knew an approximate time of a likely attack on their daughter. Without this crucial information, they had a strategy to prevent her death, but they did not know what this strategy was. If an agent has a strategy to achieve a certain outcome, knows that it has a strategy, and knows what this strategy is, then we say that the agent has a know-how strategy. Axiomatic systems for know-how strategies have been studied before aa16jlc; fhlw17ijcai; nt17aamas; nt18ai; nt18aaai; nt19ai. In a setting with imperfect information, it is natural to modify the principle of alternative possibilities to require an agent or a coalition to have a know-how strategy to prevent. In our case, parents had many different strategies that included taking vacations in different months. They did not know that a vacation in October would have prevented Tatiana’s death. Thus, they cannot be blamed for her death according to the modified version of the principle of alternative possibilities. We write this as

Although Tatiana’s parents did not know how to prevent her death, Dr. Lawrence Moore did. He was a psychiatrist who treated Poddar at the University of California mental clinic. Poddar told Moore how he met Tatiana at the University international student house, how they started to date and how depressed Poddar became when Tatiana lost romantic interest in him. Less than two months before the tragedy, Poddar shared with the doctor his intention to buy a gun and to murder Tatiana. Dr. Moore reported this information to the University campus police. Since the University knew that Poddar was at the peak of his depression, they could estimate the possible timing of the attack. Thus, the University knew what actions the parents could take to prevent the tragedy. In general, if a coalition

knows how a coalition can achieve a certain outcome, then coalition has a second-order know-how strategy to achieve the outcome. This class of strategies and a complete logical system that describes its properties were proposed in nt18aamas. We write if is true and coalition knew how coalition could have prevented . In our case,

After Tatiana’s death, her parents sued the University. In 1976 the California Supreme Court ruled that “When a therapist determines, or pursuant to the standards of his profession should determine, that his patient presents a serious danger of violence to another, he incurs an obligation to use reasonable care to protect the intended victim against such danger. The discharge of this duty may require the therapist to take one or more of various steps, depending upon the nature of the case. Thus it may call for him to warn the intended victim or others likely to apprise the victim of the danger, to notify the police, or to take whatever other steps are reasonably necessary under the circumstances.” t76opinion. In other words, the California Supreme Court ruled that in this case the duty to warn is not only a moral obligation but a legal one as well. In this paper we propose a sound and complete logical system that describes the interplay between the distributed knowledge modality and the second-order blameworthiness or duty to warn modality . The (first-order) blameworthiness modality mentioned earlier could be viewed as an abbreviation for .

The paper is organized as follows. In the next section we introduce the formal syntax and semantics of our logical system. Section 3 discusses the choices we made when defining semantics. In Section 4 we list axioms and compare them to those in the related logical systems. Section 5 gives examples of formal proofs in our system. Section 6 and Section 7 contain the proofs of the soundness and the completeness, respectively. Section 8 concludes.

## 2. Syntax and Semantics

In this section we introduce the formal syntax and semantics of our logical system. We assume a fixed set of propositional variables and a fixed set of agents . By a coalition we mean any (possibly infinite) subset of . The language of our logical system is defined by grammar:

 φ:=p|¬φ|φ→φ|KCφ|BDCφ,

where and are arbitrary coalitions. Boolean connectives , , and are defined through and in the usual way. By we denote the formula and by the set of all functions from set to set .

###### Definition .

A game is a tuple , where

1. is a set of “initial states”,

2. is an “indistinguishability” equivalence relation on the set of initial states , for each agent ,

3. is a set of “actions”,

4. is a set of “outcomes”,

5. a set of “plays” is an arbitrary set of tuples such that , , and . Furthermore, we assume that for each and each function , there is at least one such that ,

6. for each propositional variable .

By a complete (action) profile we mean any function that maps agents in into actions in . By an (action) profile of a coalition we mean any function from set .

Figure 1 depicts a diagram of the game for the Tarasoff case. It shows two possible initial states: October and November that represent two possible months with the peak of Poddar’s depression. The actual initial state was October, which was known to the University, but not to Tatiana’s parents. In other words, the University could distinguish these two states, but the parents could not. We show the indistinguishability relation by dashed lines. At the peak of his depression, agent Poddar might decide not to attack Tatiana (action 0) or to attack her (action 1). Parents, whom we represent by a single agent for the sake of simplicity, might decide to take vacation in October (action 0) or November (action 1). Thus, in our example, . Set consists of outcomes and . Recall that a complete action profile is a function from agents into actions. Since in our case there are only two agents (Poddar and parents), we write action profiles as where is an action of Poddar and is an action of the parents. The plays of the game are all possible valid combinations of an initial state, a complete action profile, and an outcome. The plays are represented in the diagram by directed edges. For example, the directed edge from initial state October to outcome is labeled with action profile . This means that . In other words, if the peak of depression is in October, Poddar decides to attack (1), and the parents take vacation in November (1), then Tatiana is dead. Multiple labels on the same edge of the diagram represent multiple plays with the same initial state and the same outcome.

Next is the core definition of this paper. Its item 5 formally defines the semantics of modality . We write if for each agent . We also write if for each element .

###### Definition .

For any game , any formula , and any play , the satisfiability relation is defined recursively as follows:

1. if ,

2. if ,

3. if or ,

4. if for each play such that ,

5. if and there is a profile such that for each play , if and , then .

Going back to our running example,

 (October,11,dead)⊩B% \scriptsize parents\scriptsize university(Tatiana is killed'')

because and

 (m,x0,ω)⊩¬(Tatiana is killed'')

for each , each initial state , and each outcome such that and .

## 3. Discussion

Traditionally, in modal logic the satisfiability is defined as a relation between a state and a formulae. This approach is problematic in the case of the blameworthiness modality because this modality refers to two different states: if statement is true in the current state and coalition knew how coalition could have prevented in the previous state. In other words, the meaning of formula depends not only on the current state, but on the previous one as well. We resolve this issue by defining the satisfiability as a relation between a play and a formula, where a play is a triple consisting of the previous state , the complete action profile , and an outcome (state) . We distinguish initial states from outcomes to make the presentation more elegant. Otherwise, this distinction is not significant.

Because the satisfiability is defined as a relation between plays and formulae, one can potentially talk about two forms of knowledge in our system: a priori knowledge in the initial state and a posteriori knowledge in the outcome. The knowledge captured by the modality as well as the knowledge implicitly referred to by the modality , see item (5) of Definition 2, is a priori knowledge. In order to define a posteriori knowledge in our setting, one would need to add an indistinguishability relation on outcomes to Definition 2. We do not consider a posteriori knowledge because one should not be blamed for something that she only knows how to prevent post-factum.

Since we define the second-order blameworthiness using distributed knowledge, if a coalition is blamable for not warning coalition , then any superset could be blamed for not warning . One might argue that the definition of blameworthiness modality should include a minimality condition on the coalition . We do not include this condition in item (5) of Definition 2, because there are several different ways to phrase the minimality, all of which could be expressed through our basic modality .

First of all, we can say that is the minimal coalition among those coalitions that knew how could have prevented . Let us denote this modality by . It can be expressed through as:

 [1]DCφ≡BDCφ∧¬⋁E⊊CBDEφ.

Second, we can say that is the minimal coalition that knew how somebody could have prevented :

 [2]DCφ≡BDCφ∧¬⋁E⊊C⋁F⊆ABFEφ.

Third, we can say that is the minimal coalition that knew how the smallest coalition could have prevented :

 [3]DCφ≡BDCφ∧¬⋁E⊆A⋁F⊊DBFEφ∧¬⋁E⊊CBDEφ.

Finally, we can say that is the minimal coalition that knew how some smallest coalition could have prevented :

 [4]Cφ≡⋁D⊆A⎛⎝BDCφ∧¬⋁E⊆A⋁F⊊DBFEφ∧¬⋁E⊊CBDEφ⎞⎠.

The choice of the minimality condition depends on the specific situation. Instead of making a choice between several possible alternatives, in this paper we study the basic blameworthiness modality without a minimality condition through which modalities , , , , and possibly others could be defined.

## 4. Axioms

In addition to the propositional tautologies in language , our logical system contains the following axioms:

1. Truth: and ,

2. Distributivity: ,

3. Negative Introspection: ,

4. Monotonicity: and ,
where and ,

5. None to Act: ,

6. Joint Responsibility: if , then
,

7. Strict Conditional: ,

8. Introspection of Blameworthiness: .

The Truth, the Distributivity, the Negative Introspection, and the Monotonicity axioms for modality are the standard axioms from the epistemic logic S5 for distributed knowledge fhmv95. The Truth axiom for modality states that a coalition can only be blamed for something that has actually happened. The Monotonicity axiom for modality captures the fact that both distributed knowledge and coalition power are monotonic.

The None to Act axiom is true because the empty coalition has only one action profile. Thus, if the empty coalition can prevent , then would have to be false on the current play. This axiom is similar to the None to Blame axiom in nt19aaai.

The Joint Responsibility axiom shows how the blame of two separate coalitions can be combined into the blame of their union. This axiom is closely related to Marc Pauly p02 Cooperation axiom, which is also used in coalitional modal logics of know-how aa16jlc; nt17aamas; nt18ai; nt18aaai and second-order know-how nt18aamas. We formally prove the soundness of this axiom in Lemma 6.

Strict conditional states that formula is known to to imply . By contraposition, coalition knows that if is prevented, then is also prevented. The Strict Conditional axiom states that if could be second-order blamed for , then it should also be second-order blamed for as long as is true. A similar axiom is present in nt19aaai.

Finally, the Introspection of Blameworthiness axiom says that if coalition is second-order blamed for , then knows that it is second-order blamed for as long as is true. A similar Strategic Introspection axiom for second-order know-how modality is present in nt18aamas.

We write if formula is provable from the axioms of our system using the Modus Ponens and the Necessitation inference rules:

 φ,φ→ψψ,φKCφ.

We write if formula is provable from the theorems of our logical system and an additional set of axioms using only the Modus Ponens inference rule.

If , then .

###### Proof.

By the deduction lemma applied times, assumption implies that Thus, by the Necessitation inference rule,

 ⊢KC(φ1→(φ2→…(φn→ψ)…)).

Hence, by the Distributivity axiom and the Modus Ponens rule,

 ⊢KCφ1→KC(φ2→…(φn→ψ)…).

Then, again by the Modus Ponens rule,

 KCφ1⊢KC(φ2→…(φn→ψ)…).

Therefore, by applying the previous steps more times. ∎

The next lemma capture a well-known property of S5 modality. Its proof could be found, for example, in nt18aamas.

.

## 5. Examples of Derivations

The soundness of logical system is established in the next section. Here we prove several lemmas about our formal system that will be used later in the proof of the completeness.

.

###### Proof.

Note that by the Introspection of Blameworthiness axiom. Thus, , by the law of contrapositive. Then, by the Necessitation inference rule. Hence, by the Distributivity axiom and the Modus Ponens inference rule,

 ⊢KC¬KC(φ→BDCφ)→KC¬BDCφ.

At the same time, by the Negative Introspection axiom:

 ⊢¬KC(φ→BDCφ)→KC¬KC(φ→BDCφ).

Then, by the laws of propositional reasoning,

 ⊢¬KC(φ→BDCφ)→KC¬BDCφ.

Thus, by the law of contrapositive,

 ⊢¬KC¬BDCφ→KC(φ→BDCφ).

Since is an instance of the Truth axiom, by propositional reasoning,

 ⊢¬KC¬BDCφ→(φ→BDCφ).

Therefore, by the definition of . ∎

If , then .

###### Proof.

By the Strict Conditional axiom,

 ⊢KC(ψ→φ)→(BDCφ→(ψ→BDCψ)).

Assumption implies by the laws of propositional reasoning. Hence, by the Necessitation inference rule. Thus, by the Modus Ponens rule, Then, by the laws of propositional reasoning,

 ⊢(BDCφ→ψ)→(BDCφ→BDCψ). (1)

Observe that by the Truth axiom. Also, by the assumption of the lemma. Then, by the laws of propositional reasoning, . Therefore, by the Modus Ponens inference rule from statement (1). ∎

.

###### Proof.

By the Truth axioms, . Hence, by the law of contrapositive, . Thus, by the definition of the modality . Therefore, by the Modus Ponens inference rule. ∎

The next lemma generalizes the Joint Responsibility axiom from two coalitions to multiple coalitions.

###### Lemma

For any integer ,

 {¯¯¯¯KEiBFiEiχi}ni=1,χ1∨⋯∨χn⊢BF1∪⋯∪FnE1∪⋯∪En(χ1∨⋯∨χn),

where sets are pairwise disjoint.

###### Proof.

We prove the lemma by induction on . If , then disjunction is Boolean constant false . Hence, the statement of the lemma, , is provable in the propositional logic.

Next, assume that . Then, from Lemma 5 using Modus Ponens rule twice, we get .

Assume now that . By the Joint Responsibility axiom and the Modus Ponens inference rule,

 ¯¯¯¯KE1∪⋯∪En−1BF1∪⋯∪Fn−1E1∪⋯∪En−1(χ1∨⋯∨χn−1),¯¯¯¯KEnBFnEnχn, χ1∨⋯∨χn−1∨χn⊢BF1∪⋯∪Fn−1∪FnE1∪⋯∪En−1∪En(χ1∨⋯∨χn−1∨χn).

Hence, by Lemma 5,

 BF1∪⋯∪Fn−1E1∪⋯∪En−1(χ1∨⋯∨χn−1),¯¯¯¯KEnBFnEnχn,χ1∨⋯∨χn−1∨χn ⊢BF1∪⋯∪Fn−1∪FnE1∪⋯∪En−1∪En(χ1∨⋯∨χn−1∨χn).

At the same time, by the induction hypothesis,

 {¯¯¯¯KEiBFiEiχi}n−1i=1,χ1∨⋯∨χn−1⊢BF1∪⋯∪Fn−1E1∪⋯∪En−1(χ1∨⋯∨χn−1).

Thus,

 {¯¯¯¯KEiBFiEiχi}ni=1,χ1∨⋯∨χn−1,χ1∨⋯∨χn−1∨χn ⊢BF1∪⋯∪Fn−1∪FnE1∪⋯∪En−1∪En(χ1∨⋯∨χn−1∨χn).

Note that is provable in the propositional logic. Thus,

 {¯¯¯¯KEiBFiEiχi}ni=1,χ1∨⋯∨χn−1 ⊢BF1∪⋯∪Fn−1∪FnE1∪⋯∪En−1∪En(χ1∨⋯∨χn−1∨χn). (2)

Similarly, by the Joint Responsibility axiom and the Modus Ponens inference rule,

 ¯¯¯¯KE1BF1E1χ1,¯¯¯¯KE2∪⋯∪EnBF2∪⋯∪FnE2∪⋯∪En(χ2∨⋯∨χn), χ1∨(χ2∨⋯∨χn)⊢BF1∪⋯∪Fn−1∪FnE1∪⋯∪En−1∪En(χ1∨(χ2∨⋯∨χn)).

Because formula is provable in the propositional logic, by Lemma 5,

 ¯¯¯¯KE1BF1E1χ1,¯¯¯¯KE2∪⋯∪EnBF2∪⋯∪FnE2∪⋯∪En(χ2∨⋯∨χn), χ1∨χ2∨⋯∨χn⊢BF1∪⋯∪Fn−1∪FnE1∪⋯∪En−1∪En(χ1∨χ2∨⋯∨χn).

Hence, by Lemma 5,

 ¯¯¯¯KE1BF1E1χ1,BF2∪⋯∪FnE2∪⋯∪En(χ2∨⋯∨χn),χ1∨χ2∨⋯∨χn ⊢BF1∪⋯∪Fn−1∪FnE1∪⋯∪En−1∪En(χ1∨χ2∨⋯∨χn).

At the same time, by the induction hypothesis,

 {¯¯¯¯KEiBFiEiχi}ni=2,χ2∨⋯∨χn⊢BF2∪⋯∪FnE2∪⋯∪En(χ2∨⋯∨χn).

Thus,

 {¯¯¯¯KEiBFiEiχi}ni=1,χ2∨⋯∨χn,χ1∨χ2∨⋯∨χn ⊢BF1∪⋯∪Fn−1∪FnE1∪⋯∪En−1∪En(χ1∨χ2∨⋯∨χn).

Note that is provable in the propositional logic. Thus,

 {¯¯¯¯KEiBFiEiχi}ni=1,χ2∨⋯∨χn ⊢BF1∪⋯∪Fn−1∪FnE1∪⋯∪En−1∪En(χ1∨χ2∨⋯∨χn). (3)

Finally, note that the following statement is provable in the propositional logic for ,

 ⊢χ1∨⋯∨χn→(χ1∨⋯∨χn−1)∨(χ2∨⋯∨χn).

Therefore, from statement (2) and statement (3),

 {¯¯¯¯KEiBFiEiχi}ni=1,χ1∨⋯∨χn⊢BF1∪⋯∪FnE1∪⋯∪En(χ1∨⋯∨χn).

by the laws of propositional reasoning. ∎

Our last example rephrases Lemma 5 into the form which is used in the proof of the completeness.

###### Lemma

For any , any sets , and any pairwise disjoint sets ,

 {¯¯¯¯KEiBFiEiχi}ni=1,KC(φ→χ1∨⋯∨χn)⊢KC(φ→BDCφ).
###### Proof.

Let . Then, by Lemma 5,

 X,χ1∨⋯∨χn⊢BF1∪⋯∪FnE1∪⋯∪En(χ1∨⋯∨χn).

Hence, by the Monotonicity axiom,

 X,χ1∨⋯∨χn⊢BDC(χ1∨⋯∨χn).

Thus,

 X,φ,φ→χ1∨⋯∨χn⊢BDC(χ1∨⋯∨χn)

by the Modus Ponens inference rule. Hence, by the Truth axiom,

 X,φ,KC(φ→χ1∨⋯∨χn)⊢BDC(χ1∨⋯∨χn).

The following formula is an instance of the Strict Conditional axiom . Thus, by the Modus Ponens applied twice,

 X,φ,KC(φ→χ1∨⋯∨χn)⊢φ→BDCφ.

Then, by the Modus Ponens. Thus,

 X,KC(φ→χ1∨⋯∨χn)⊢φ→BDCφ

by the deduction lemma. Hence,

 {KC¯¯¯¯KEiBFiEiχi}ni=1,KCKC(φ→χ1∨⋯∨χn)⊢KC(φ→BDCφ)

by Lemma 4 and the definition of set . Then,

 {KEi¯¯¯¯KEiBFiEiχi}ni=1,KCKC(φ→χ1∨⋯∨χn)⊢KC(φ→BDCφ)

by the Monotonicity axiom, the Modus Ponens inference rule, and the assumption . Thus,

 {¯¯¯¯KEiBFiEiχi}ni=1,KCKC(φ→χ1∨⋯∨χn)⊢KC(φ→BDCφ)

by the definition of modality , the Negative Introspection axiom, and the Modus Ponens rule. Therefore, by Lemma 4 and the Modus Ponens inference rule, the statement of the lemma is true. ∎

## 6. Soundness

The soundness of the Truth, the Distributivity, the Negative Introspection, the Monotonicity, and the None to Blame axioms is straightforward. Below we prove the soundness of the Joint Responsibility, the Strict Conditional, and the Introspection of Blameworthiness axioms as separate lemmas.

###### Lemma

If , , , and , then .

###### Proof.

By Definition 2 and the definition of modality , assumption implies that there is a play such that and . Thus, again by Definition 2, there is an action profile such that for each play , if and , then . Recall that . Thus, for each play ,

 α∼Cα′∧s1=Dδ′→(α′,δ′,ω′)⊮φ. (4)

Similarly, assumption implies that there is a profile such that for each play ,

 α∼Eα′∧s2=Fδ′→(α′,δ′,ω′)⊮ψ. (5)

Let be the action profile:

 s(a)={s1(a),if a∈D,s2(a),if a∈F. (6)

Action profile is well-defined because . Statements (4), (5), and (6) by Definition 2 imply that for each play if and , then . Recall that . Therefore, by Definition 2. ∎

###### Lemma

If , , and , then .

###### Proof.

By Definition 2, assumption implies that for each play of the game if , then .

By Definition 2, assumption implies that there is an action profile such that for each play , if and , then .

Hence, for each play , if and , then . Therefore, by Definition 2 and the assumption of the lemma. ∎

If , then .

###### Proof.

By Definition 2, assumption implies that there is an action profile such that for each play , if and , then .

Let be a play where and . By Definition 2, it suffices to show that .

Consider any play such that and . Then, since is an equivalence relation, assumptions and imply . Thus, by the choice of action profile . Therefore, by Definition 2 and the assumption . ∎

## 7. Completeness

The standard proof of the completeness for individual knowledge modality defines states as maximal consistent sets fhmv95. Two such sets are indistinguishable to an agent if these sets have the same -formulae. This construction does not work for distributed knowledge because if two sets share -formulae and -formulae, they do not necessarily have to share -formulae. To overcome this issue, we use the Tree of Knowledge construction, similar to the one in nt18ai. An important change to this construction proposed in the current paper is placing elements of a set on the edges of the tree. This change is significant for the proof of Lemma 7.

Let be an arbitrary set of cardinality larger than that of the set . Next, for each maximal consistent set of formulae , we define the canonical game