In this article we study blameworthiness of agents and their coalitions in multiagent systems. Throughout centuries, blameworthiness, especially in the context of free will and moral responsibility, has been at the focus of philosophical discussions se13eb . These discussions continue in the modern time f94p ; fr00 ; nk07nous ; m15ps ; w17 . Frankfurt f69tjop acknowledges that a dominant role in these discussions has been played by what he calls a principle of alternate possibilities: “a person is morally responsible for what he has done only if he could have done otherwise”. Following the established tradition w17 , we refer to this principle as the principle of alternative possibilities. Cushman c15cop talks about counterfactual possibility: “a person could have prevented their harmful conduct, even though they did not”.
Others refer to an alternative possibility as a counterfactual possibility c15cop ; h16 . Halpern and Pearl proposed several versions of a formal definition of causality as a relation between sets of variables that include a counterfactual requirement h16 . Halpern and Kleiman-Weiner hk18aaai used a similar setting to define degrees of blameworthiness. Batusov and Soutchanski bs18aaai gave a counterfactual-based definition of causality in situation calculus. Alechina, Halpern, and Logan ahl17aamas applied counterfactual definition of causality to team plans. In nt19aaai , we proposed a logical system that describes properties of coalition blameworthiness in strategic games as a modal operator whose semantics is also based on the principle of alternative possibilities.
Although the principle of alternative possibilities makes sense in the settings with perfect information, it needs to be adjusted for settings with imperfect information. Indeed, consider a traffic situation depicted in Figure 1. A self-driving truck and a regular car are approaching an intersection at which truck must stop to yield to car . The truck is experiencing a sudden brake failure and it cannot stop, nor can it slow down at the intersection. The truck turns on flashing lights and sends distress signals to other self-driving cars by radio. The driver of car can see the flashing lights, but she does not receive the radio signal. She can also observe that the truck does not slow down. The driver of car has two potential strategies to avoid a collision with the truck: to slow down or to accelerate.
The driver understands that one of these two strategies will succeed, but since she does not know the exact speed of the truck, she does not know which of the two strategies will succeed. Suppose that the collision could be avoided if the car accelerates, but the car driver decides to slow down. The vehicles collide. According to the principle of alternative possibilities, the driver of the car is responsible for the collision because she had a strategy to avoid the collision but did not use it.
It is not likely, however, that a court will find the driver of car responsible for the accident. For example, US Model Penal Code ali62 distinguishes different forms of legal liability as different combinations of “guilty actions” and “guilty mind”. The situation in our example falls under strict liability (pure “guilty actions” without an accompanied “guilty mind”). In many situations, strict liability does not lead to legal liability.
In this article we propose a formal semantics of blameworthiness in strategic games with imperfect information. According to this semantics, an agent (or a coalition of agents) is blamable for if is true and the agent knew how to prevent . In our example, since the driver of the car does not know that she must accelerate in order to avoid the collision, she cannot be blamed for the collision. We write this as: Now, consider a similar traffic situation in which car is a self-driving vehicle. The car receives the distress signal from truck , which contains the truck’s exact speed. From this information, car determines that it can avoid the collision if it accelerates. However, if the car slows down, then the vehicles collide and the self-driving car is blameable for the collision:
The main technical result of this article is a bimodal logical system that describes the interplay between knowledge and blameworthiness of coalitions in strategic games with imperfect information.
The article is organized as follows. In the next section we review the literature. Section 3 presents the formal syntax and semantics of our logical system. Section 4 introduces our axioms and compares them to those in the related works. Section 5 gives examples of formal derivations in the proposed logical system. Sections 6 and 7 prove the soundness and the completeness of our system. Section 8 concludes with a discussion of future work.
2 Related Literature
Although the study of responsibility and blameworthiness has a long history in philosophy, the use of formal logical systems to capture these notions is a recent development. Xu x98jpl proposed a complete logical system for reasoning about responsibility of individual agents in multiagent systems. His approach was extended to coalitions by Broersen, Herzig, and Troquard bht09jancl . The definition of responsibility in these works is different from ours. They assume that an agent or a coalition of agents is responsible for an outcome if the actions that they took unavoidably lead to the outcome. Xu x98jpl also requires a possibility that the outcome might not happen. However, he does not require that the agent has a strategy to prevent the outcome. Thus, their definitions are not based on the principle of alternative possibilities.
Halpern and Pearl gave several versions of a formal definition of causality between sets of variables using counterfactuals h16 . Lorini and Schwarzentruber ls11ai observed that a variation of this definition can be captured in STIT logic bp90krdr ; h01 ; h95jpl ; hp17rsl ; ow16sl . They said that there is a counterfactual dependence between actions of a coalition and an outcome if is true and the complement of the coalition had no strategy to force . In their notations: where is the set of all agents. They also observed that many human emotions (regret, rejoice, disappointment, elation) can be expressed through a combination of the modality and the knowledge modality.
The game-like setting of this article closely resembles the semantics of Mark Pauly’s logic of coalition power p01illc ; p02 . His approach has been widely investigated in the literature g01tark ; vw05ai ; b07ijcai ; sgvw06aamas ; abvs10jal ; avw09ai ; b14sr ; gjt13jaamas ; alnr11jlc ; ga17tark ; ge18aamas ; nr18kr . Logics of coalition power study modality that express what a coalition can do. In nt19aaai we modified Mark Pauly’s semantics to express what a coalition could have done. We axiomatized a logic that combines statements “ is true” and “coalition could have prevented ” into a single modality .
In this article we replace “coalition could have prevented ” in nt19aaai with “coalition knew how it could have prevented ”. The distinction between an agent having a strategy, knowing that a strategy exists, and knowing what the strategy is has been studied before. While Jamroga and Ågotnes ja07jancl talked about “knowledge to identify and execute a strategy”, Jamroga and van der Hoek jv04fm discussed “difference between an agent knowing that he has a suitable strategy and knowing the strategy itself”. Van Benthem v01ber called such strategies “uniform”. Broersen b08deon talked about “knowingly doing”, while Broersen, Herzig, and Troquard bht09jancl discussed modality “know they can do”. We used term “executable strategy” nt17aamas . Wang w15lori ; w17synthese talked about “knowing how”.
The properties of know-how as a modality have been previously axiomatized in different settings. Ågotnes and Alechina introduced a complete axiomatization of an interplay between single-agent knowledge and coalition know-how modalities to achieve a goal in one step aa16jlc . A modal logic that combines the distributed knowledge modality with the coalition know-how modality to maintain a goal was axiomatized by us in nt17aamas . A sound and complete logical system in a single-agent setting for know-how strategies to achieve a goal in multiple steps rather than to maintain a goal is developed by Fervari, Herzig, Li, and Wang fhlw17ijcai . In nt17tark ; nt18ai , we developed a trimodal logical system that describes an interplay between the (not know-how) coalition strategic modality, the coalition know-how modality, and the distributed knowledge modality. In nt18aaai , we proposed a logical system that combines the coalition know-how modality with the distributed knowledge modality in the perfect recall setting. In nt18aamas , we introduced a logical system for second-order know-how. Wang proposed a complete axiomatization of “knowing how” as a binary modality w15lori ; w17synthese , but his logical system does not include the knowledge modality.
The axioms of the logical system proposed in this article are very similar to our axioms in nt19aaai for blameworthiness in games with perfect information and so are the proofs of soundness of these axioms. The most important contribution of this article is the proof of completeness, in which the construction from nt19aaai is significantly modified to incorporate distributed knowledge. These modifications are discussed in the beginning of Section 7.
3 Syntax and Semantics
In this article we assume a fixed set of agents and a fixed set of propositional variables. By a coalition we mean an arbitrary subset of set .
is the minimal set of formulae such that
for each propositional variable ,
for all formulae ,
, for each coalition and each .
In other words, language is defined by grammar:
Formula is read as “coalition distributively knew before the actions were taken that statement would be true” and formula as “coalition is blamable for ”.
Boolean connectives , , and as well as constants and are defined in the standard way. By formula we mean . For the disjunction of multiple formulae, we assume that parentheses are nested to the left. That is, formula is a shorthand for . As usual, the empty disjunction is defined to be . For any two sets and , by we denote the set of all functions from to .
The formal semantics of modalities and is defined in terms of models, which we call games. These are one-shot strategic games with imperfect information. We specify the set of actions by all agents, or a complete action profile, as a function from the set of all agents to the set of all actions .
A game is a tuple , where
is a set of “initial states”,
is an “indistinguishability” equivalence relation on set ,
is a nonempty set of “actions”,
is a set of “outcomes”,
the set of “plays” is an arbitrary set of tuples where for each initial state and each complete action profile , there is at least one outcome such that ,
is a function that maps propositional variables into subsets of .
In the introductory example, the set has two states high and low, corresponding to the truck going at a high or low speed, respectively. The driver of the regular car cannot distinguish these two states while these states can be distinguished by a self-driving version of car . For the sake of simplicity, assume that there are two actions that car can take: and two possible outcomes: . Vehicles collide if either the truck goes with a low speed and the car decides to slow-down or the truck goes with a high speed and the car decides to accelerate. In our case there is only one agent (car ), so the complete action profile can be described by giving just the action of this agent. We refer to the two complete action profiles in this situation simply as profile slow-down and profile speed-up. The list of all possible scenarios (or “plays”) is given by the set
Note that in our example an initial state and an action profile uniquely determine the outcome. In general, just like in nt19aaai , we allow nondeterministic games where this does not have to be true. However, unlike nt19aaai , we do require that for each initial state and each action profile there is at least one outcome. As we discuss in Section 4, this requirement captures better the intuitive notion of blameworthiness.
Whether statement is true or false depends not only on the outcome but also on the initial state of the game. Indeed, coalition might have known how to prevent in one initial state but not in the other. For this reason, we assume that all statements are true or false for a particular play of the game. For example, propositional variable can stand for “car slowed down and collided with truck going at a high speed”. As a result, function in the definition above maps into subsets of rather than subsets of .
By an action profile of a coalition we mean an arbitrary function that assigns an action to each member of the coalition. If and are action profiles of coalitions and , respectively, and is any coalition such that , then we write to denote that for each agent . We write if for each . In particular, it means that for any two initial states .
Next is the key definition of this article. Its item 5 formally specifies blameworthiness using the principle of alternative possibilities. In order for a coalition to be blamable for , not only must be true and the coalition should have had a strategy to prevent , but this strategy should work in all initial states that the coalition cannot distinguish from the current state. In other words, the coalition should have known the strategy.
For any game , any formula , and any play , the satisfiability relation is defined recursively as follows:
if , where is a propositional variable,
if or ,
if for each play such that ,
if and there is an action profile of coalition such that for each play , if and , then .
Since modality represents a priori (before the actions) knowledge of coalition , only the initial states in plays and are indistinguishable in item 4 of Definition 3. Similarly, since item 5 of the above definition refers to indistinguishability relation on initial states, not outcomes, the knowledge of the strategy to prevent captured by the modality is also a priori knowledge of coalition .
For formula to be true, item 5 of Definition 3 requires coalition to know a strategy to prevent , but it does not require the coalition to know that is true. This captures a common belief, for example, that a murder is blameable for a death even if the murder does not know that the victim died.
Note that in item 5 of the above definition we do not assume that coalition is a minimal one that knew how to prevent the outcome. This is different from the definition of blameworthiness in h17 . Our approach is consistent with how word “blame” is often used in English. For example, the sentence “Millennials being blamed for decline of American cheese” g18foxnews does not imply that no one in the millennial generation likes American cheese.
In addition to the propositional tautologies in language , our logical system contains the following axioms.
Truth: and ,
Negative Introspection: ,
Monotonicity: and , where ,
None to Blame: ,
Blamelessness of Truth: ,
Joint Responsibility: , where ,
Blame for Known Cause: ,
Knowledge of Fairness: .
We write if formula is provable from the axioms of our system using the Modus Ponens and the Necessitation inference rules:
We write if formula is provable from the theorems of our logical system and an additional set of axioms using only the Modus Ponens inference rule. Note that if set is empty, then statement is equivalent to . We say that set is consistent if .
The Truth, the Distributivity, the Negative Introspection, and the Monotonicity axioms for epistemic modality are the standard S5 axioms from the logic of distributed knowledge. The Truth axiom for blameworthiness modality states that a coalition could only be blamed for something true. The Monotonicity axiom for the blameworthiness modality states that if a part of a coalition is blamable for something, then the whole coalition is also blamable for the same thing. The None to Blame axiom says that an empty coalition can be blamed for nothing. The Blamelessness of Truth axiom states that no coalition can be blamed for a tautology. This is a new axiom that does not have an equivalent in nt19aaai . The soundness of this axiom relies on our assumption in item 4 of Definition 2 that any combination of an initial state and a complete action profile has at least one outcome. Without this assumption, a coalition might be able to terminate the game without reaching an outcome. In other words, coalition might have a strategy to “prevent” .
The remaining three axioms describe the interplay between knowledge and blameworthiness modalities. The Joint Responsibility axiom says that if a coalition cannot exclude a possibility of being blamable for , a coalition cannot exclude a possibility of being blamable for , and the disjunction is true, then the joint coalition is blamable for the disjunction. This axiom resembles Xu’s axiom for the independence of individual agents x98jpl ,
where modality is an abbreviation for and formula stands for “formula is universally true in the given model”. Broersen, Herzig, and Troquard bht09jancl captured the independence of disjoint coalitions and in their Lemma 17:
In spite of certain similarity, the definition of responsibility used in x98jpl and bht09jancl does not assume the principle of alternative possibilities. The Joint Responsibility axiom is also similar to Marc Pauly’s Cooperation axiom for the logic of coalitional power p01illc ; p02 :
where coalitions and are disjoint and stands for “coalition has a strategy to achieve ”. Finally, The Joint Responsibility axiom in this article is a generalization of the Joint Responsibility axiom for games with perfect information nt19aaai :
where coalitions and are disjoint.
Informally, if , then we say that is a cause of known to coalition . Note that if a coalition has a strategy to prevent a known cause , then the coalition also has a strategy to prevent . However, it is not true that the coalition should be blamed for if it can be blamed for because “the known cause” might not be true. If the known cause is true, then the blameworthiness for implies the blameworthiness for . This is captured in the Blame for Known Cause axiom. A similar axiom, but without knowledge, appeared in nt19aaai .
Our last axiom also goes back to one of the axioms for the games with perfect information. The Fairness axiom for these games
states “if a coalition is blamed for , then it should be blamed for whenever is true” nt19aaai . The Knowledge of Fairness axiom in the current article states that if a coalition is blamable for in an imperfect information game, then it knows that it is blamable for whenever is true.
Next, we state the deduction and Lindenbaum lemmas for our logical system. These lemmas are used later in the proof of the completeness.
Lemma 1 (deduction)
If , then .
Proof. Suppose that sequence is a proof from set and the theorems of our logical system that uses the Modus Ponens inference rule only. In other words, for each , either
is equal to , or
there are such that formula is equal to .
It suffices to show that for each . We prove this by induction on through considering the four cases above separately.
Case 1: . Note that is a propositional tautology, and thus, is an axiom of our logical system. Hence, by the Modus Ponens inference rule. Therefore, .
Case 2: . Then, .
Case 3: formula is equal to . Thus, is a propositional tautology. Therefore, .
Case 4: formula is equal to for some . Thus, by the induction hypothesis, and . Note that formula is a propositional tautology. Therefore, by applying the Modus Ponens inference rule twice.
Note that it is important for the above proof that stands for derivability only using the Modus Ponens inference rule. For example, if the Necessitation inference rule is allowed, then the proof will have to include one more case where is formula for some coalition , and some integer . In this case we will need to prove that if , then , which is not true.
Lemma 2 (Lindenbaum)
Any consistent set of formulae can be extended to a maximal consistent set of formulae.
Proof. The standard proof of Lindenbaum’s lemma applies here (m09, , Proposition 2.14).
5 Examples of Derivations
We prove the soundness of the axioms of our logical system in the next section. Here we prove several lemmas about our formal system that are used later in the proof of the completeness.
Proof. Note that by the Knowledge of Fairness axiom. Thus, , by the law of contrapositive. Then, by the Necessitation inference rule. Hence, by the Distributivity axiom and the Modus Ponens inference rule,
At the same time, by the Negative Introspection axiom:
Then, by the laws of propositional reasoning,
Thus, by the law of contrapositive,
Since is an instance of the Truth axiom, by propositional reasoning,
Therefore, by the definition of .
If , then .
Proof. Assumption implies by the laws of propositional reasoning. Hence, by the Necessitation inference rule. Thus, by the Blame for Known Cause axiom and the Modus Ponens inference rule. Hence, by propositional reasoning. Then, again by propositional reasoning,
Observe that by the Truth axiom. Also, by the assumption of the lemma. Then, by the laws of propositional reasoning, . Therefore, by the Modus Ponens inference rule from statement (1).
The next lemma states a well-known S5 principle that we use several times in the proofs that follow.
Proof. By the Truth axioms, . Hence, by the law of contrapositive, . Thus, by the definition of the modality . Therefore, by the Modus Ponens inference rule.
The next lemma generalizes the Joint Responsibility axiom from two coalitions to multiple coalitions. Informally, it says that if the disjunction is true and each of the disjoint coalitions cannot exclude a possibility of being blamed for the corresponding disjunct, then together they should be blamed for the disjunction.
For any integer and any pairwise disjoint sets ,
Proof. We prove the lemma by induction on . If , then disjunction is Boolean constant false . Hence, the statement of the lemma, , is provable in the propositional logic.
Next, assume that . Then, from Lemma 3 using Modus Ponens rule twice, we get .
Assume that . By the assumption of the lemma that sets are pairwise disjoint, the Joint Responsibility axiom, and the Modus Ponens inference rule,
Hence, by Lemma 5,
At the same time, by the induction hypothesis,
Note that is provable in the propositional logic. Thus,
Similarly, by the Joint Responsibility axiom and the Modus Ponens inference rule,
Because formula is provable in the propositional logic, by Lemma 4,
Hence, by Lemma 5,
At the same time, by the induction hypothesis,
Note that is provable in the propositional logic. Thus,
Finally, note that the following statement is provable in the propositional logic for ,
by the laws of propositional reasoning.
If , then .
Proof. By Lemma 1 applied times, assumption implies that Thus, by the Necessitation inference rule,
Hence, by the Distributivity axiom and the Modus Ponens rule,
Then, again by the Modus Ponens rule,
Therefore, by applying the previous steps more times.
The following lemma states a well-known principle in epistemic logic.
Lemma 8 (Positive Introspection)
Proof. Formula is an instance of the Truth axiom. Thus, by contraposition. Hence, taking into account the following instance of the Negative Introspection axiom: , we have
At the same time, is an instance of the Negative Introspection axiom. Thus, by the law of contrapositive in the propositional logic. Hence, by the Necessitation inference rule, . Thus, by the Distributivity axiom and the Modus Ponens inference rule, The latter, together with statement (4), implies the statement of the lemma by propositional reasoning.
Our last example rephrases Lemma 6 into the form which is used in the proof of the completeness.
For any and any disjoint sets ,
Proof. By Lemma 6,
Hence, by the Monotonicity axiom,
Thus, by the Modus Ponens inference rule,
By the Truth axiom and the Modus Ponens inference rule,
The following formula is an instance of the Blame for Known Cause axiom . Hence, by the Modus Ponens inference rule applied twice,
By the Modus Ponens inference rule,
By Lemma 1,
By Lemma 7,
By the Monotonicity axiom, the Modus Ponens inference rule, and the assumption ,
By the definition of modality , the Negative Introspection axiom, and the Modus Ponens inference rule,
Therefore, by Lemma 8 and the Modus Ponens inference rule, the statement of the lemma follows.
The epistemic part of the Truth axiom as well as the Distribitivity, the Negative Introspection, and the Monotonicity axioms are the standard axioms of epistemic logic S5 for distributed knowledge. Their soundness follows from the assumption that is an equivalence relation in the standard way fhmv95 . The soundness of the blameworthiness part of the Truth axiom and of the Monotonicity axiom immediately follows from Definition 3. In this section, we prove the soundness of each of the remaining axioms as a separate lemma. In these lemmas, are coalitions, are formulae, and is a play of a game .
Proof. Assume that . Hence, by Definition 3, we have and there is an action profile such that for each play , if and , then .
Let , , and . Since and , by the choice of action profile we have . Then, , which leads to a contradiction.
Proof. Suppose that . Thus, by Definition 3, there is an action profile of coalition such that for each play , if and , then .
Recall that the set of actions is not empty by Definition 2. Let be any action from set . Define a complete action profile as follows:
By item 5 of Definition 2, there is an outcome such that . Note that because relation is an equivalence relation. Also by the choice of the complete action profile . Therefore, by the choice of the action profile , we have , which contradicts Definition 3, taking into account the definition of the constant .
If , , , and , then .
Proof. Suppose that and . Hence, by Definition 3 and the definition of modality , there are plays and such that , , and .
Statement , by Definition 3, implies that there is a profile such that for each play , if and , then .
Similarly, statement , by Definition 3, implies that there is an action profile such that for each play , if and , then .
Consider an action profile of coalition such that