# Representing and Aggregating Conflicting Beliefs

We consider the two-fold problem of representing collective beliefs and aggregating these beliefs. We propose modular, transitive relations for collective beliefs. They allow us to represent conflicting opinions and they have a clear semantics. We compare them with the quasi-transitive relations often used in Social Choice. Then, we describe a way to construct the belief state of an agent informed by a set of sources of varying degrees of reliability. This construction circumvents Arrow's Impossibility Theorem in a satisfactory manner. Finally, we give a simple set-theory-based operator for combining the information of multiple agents. We show that this operator satisfies the desirable invariants of idempotence, commutativity, and associativity, and, thus, is well-behaved when iterated, and we describe a computationally effective way of computing the resulting belief state.

## Authors

• 2 publications
• 26 publications
• ### Aggregating incoherent agents who disagree

In this paper, we explore how we should aggregate the degrees of belief ...
09/12/2017 ∙ by Richard Pettigrew, et al. ∙ 0

• ### Aggregating Learned Probabilistic Beliefs

We consider the task of aggregating beliefs of severalexperts. We assume...
01/10/2013 ∙ by Pedrito Maynard-Reid II, et al. ∙ 0

• ### On Limited Non-Prioritised Belief Revision Operators with Dynamic Scope

The research on non-prioritized revision studies revision operators whic...
08/17/2021 ∙ by Kai Sauerwald, et al. ∙ 0

• ### Belief Revision and Trust

Belief revision is the process in which an agent incorporates a new piec...
04/30/2014 ∙ by Aaron Hunter, et al. ∙ 0

• ### Feasible Joint Posterior Beliefs

We study the set of possible joint posterior belief distributions of a g...
02/26/2020 ∙ by Itai Arieli, et al. ∙ 0

• ### How do you revise your belief set with

In the classic AGM belief revision theory, beliefs are static and do not...
04/21/2015 ∙ by Ryuta Arisaka, et al. ∙ 0

• ### Learning Triadic Belief Dynamics in Nonverbal Communication from Videos

Humans possess a unique social cognition capability; nonverbal communica...
04/07/2021 ∙ by Lifeng Fan, et al. ∙ 0

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

We are interested in the multi-agent setting where agents are informed by sources of varying levels of reliability, and where agents can iteratively combine their belief states. This setting introduces three problems: (1) Finding an appropriate representation for collective beliefs; (2) Constructing an agent’s belief state by aggregating the information from informant sources, accounting for the relative reliability of these sources; and, (3) Combining the information of multiple agents in a manner that is well-behaved under iteration.

The Social Choice community has dealt extensively with the first problem (although in the context of representing collective preferences rather than beliefs) (cf. [Sen1986]). The classical approach has been to use quasi-transitive relations (of which total pre-orders are a special subclass) over the set of possible worlds. However, these relations do not distinguish between group indifference and group conflict, and this distinction can be crucial. Consider, for example, a situation in which all members of a group are indifferent between movie and movie . If some passerby expresses a preference for , the group may very well choose to adopt this opinion for the group and borrow . However, if the group was already divided over the relative merits of and , we would be wise to hesitate before choosing one over the other just because a new supporter of appears on the scene. We propose a representation in which the distinction is explicit. We also argue that our representation solves some of the unpleasant semantical problems suffered by the earlier approach.

The second problem addresses how an agent should actually go about combining the information received from a set of sources to create a belief state. Such a mechanism should favor the opinions held by more reliable sources, yet allow less reliable sources to voice opinions when higher ranked sources have no opinion. True, under some circumstances it would not be advisable for an opinion from a less reliable source to override the agnosticism of a more reliable source, but often it is better to accept these opinions as default assumptions until better information is available. [Maynard-Reid II and Shoham2000] provides a solution to this problem when belief states are represented as total pre-orders, but runs into Arrow’s Impossibility Theorem [Arrow1963] when there are sources of equal reliability. As we shall see, the generalized representation allows us to circumvent this limitation.

To motivate the third problem, consider the following dynamic scenario: A robot controlling a ship in space receives from a number of communication centers on Earth information about the status of its environment and tasks. Each center receives information from a group of sources of varying credibility or accuracy (e.g., nearby satellites and experts) and aggregates it. Timeliness of decision-making in space is often crucial, so we do not want the robot to have to wait while each center sends its information to some central location for it to be first combined before being forwarded to the robot. Instead, each center sends its aggregated information directly to the robot. Not only does this scheme reduce dead time, it also allows for “anytime” behavior on the robot’s part: the robot incorporates new information as it arrives and makes the best decisions it can with whatever information it has at any given point. This distributed approach is also more robust since the degradation in performance is much more graceful should information from individual centers get lost or delayed.

In such a scenario, the robot needs a mechanism for combining or fusing the belief states of multiple agents potentially arriving at different times. Moreover, the belief state output by the mechanism should be invariant with respect to the order of agent arrivals. We will describe such a mechanism.

The paper is organized as follows: After some preliminary definitions and a discussion of the approach to aggregation taken in classical Social Choice, we introduce modular, transitive relations for representing generalized belief states. We then describe how to construct the belief state of an agent given the belief states of its informant sources when these sources are totally pre-ordered. Finally, we describe a simple set-theory-based operator for fusing agent belief states that satisfies the desirable invariants of idempotence, commutativity, and associativity, and we describe a computationally effective way of computing this belief state.

## 2 Preliminaries

We begin by defining various well-known properties of binary relations111We only use binary relations in this paper, so we will refer to them simply as relations.; they will be useful to us throughout the paper.

###### Definition 1

Suppose is a relation over a finite set , i.e., . We shall use to denote and to denote . The relation is:

1. reflexive iff for . It is irreflexive iff for .

2. symmetric iff for . It is asymmetric iff for . It is anti-symmetric iff for .

3. the strict version of a relation over iff for .

4. total iff for .

5. modular iff for .

6. transitive iff for .

7. quasi-transitive iff its strict version is transitive.

8. the transitive closure of a relation over iff for some integer , for .

9. acyclic iff implies for all integers , where is the strict version of .

10. a total pre-order iff it is total and transitive. It is a total order iff it is also anti-symmetric.

11. an equivalence relation iff it is reflexive, symmetric, and transitive.

###### Proposition 1

1. The transitive closure of a modular relation is modular.

2. Every transitive relation is quasi-transitive.

3. [Sen1986] Every quasi-transitive relation is acyclic.

Given a relation over a set of alternatives and a subset of these alternatives, we often want to pick the subset’s “best” elements with respect to the relation. We define this set of “best” elements to be the subset’s choice set:

###### Definition 2

If is a relation over a finite set , is its strict version, and , then the choice set of with respect to is

 C(X,≤)={x∈X:∄x′∈X. x′

A choice function is one which assigns to every subset a non-empty subset of :

###### Definition 3

A choice function over a finite set is a function such that for every .

Now, every acyclic relation defines a choice function, one which assigns to each subset its choice set:

###### Proposition 2

[Sen1986] Given a relation over a finite set , the choice set operation defines a choice function iff is acyclic.222Sen’s uses a slightly stronger definition of choice sets, but the theorem still holds in our more general case.

If a relation is not acyclic, elements involved in a cycle are said to be in a conflict because we cannot order them:

###### Definition 4

Given a relation over a finite set , and are in a conflict wrt iff there exist such that , where .

## 3 Aggregation in Social Choice

We are interested in belief aggregation, but the community historically most interested in aggregation has been that of Social Choice theory. The aggregation is over preferences rather than beliefs, so the discussion in this subsection will focus on representing preferences; however, as we shall see, the results are equally relevant to representing beliefs. In the Social Choice community, the standard representation of an agent’s preferences is a total pre-order. Each total pre-order is interpreted as describing the weak preferences of an individual , so that means considers alternative to be at least as preferable as alternative .333The direction of the relation symbol is unintuitive, but standard practice in the belief revision community. If and , then is indifferent between and .

Unfortunately, Arrow’s Impossibility Theorem [Arrow1963] showed that no aggregation operator over total pre-orders exists satisfying the following small set of desirable properties:

###### Definition 5

Let be an aggregation operator over the preferences , …, of individuals, respectively, over a finite set of alternatives , and let .

• Restricted Range: The range of is the set of total pre-orders over .

• Unrestricted Domain: The domain of is the set of -tuples of total pre-orders over .

• Pareto Principle: If for all , then .

• Independence of Irrelevant Alternatives (IIA): Suppose . If, for , iff for all , then iff .

• Non-Dictatorship: There is no individual such that, for every tuple in the domain of and every , implies .

###### Proposition 3

[Arrow1963] There is no aggregation operator that satisfies restricted range, unrestricted domain, (weak) Pareto principle, independendence of irrelevant alternatives, and nondictatorship.

This impossibility theorem led researchers to look for weakenings to Arrow’s framework that would circumvent the result. One was to weaken the restricted range condition, requiring that the result of an aggregation only satisfy totality and quasi-transitivity rather than the full transitivity of a total pre-order. This weakening was sufficient to guarantee the existence of an aggregation function satisfying the other conditions, while still producing relations that defined choice functions [Sen1986]. However, this solution was not without its own problems.

First, total, quasi-transitive relations have unsatisfactory semantics. If is total and quasi-transitive but not a total pre-order, its indifference relation is not transitive:

###### Proposition 4

Let be a relation over a finite set and let be its symmetric restriction (i.e., iff and ). If is total and quasi-transitive but not transitive, then is not transitive.

There has been much discussion as to whether or not indifference should be transitive; in many cases one feels indifference should be transitive. If Deb enjoys plums and mangoes equally and also enjoys mangoes and peaches equally, we would conclude that she also enjoys plums and peaches equally. It seems that total quasi-transitive relations that are not total pre-orders cannot be understood easily as preference or indifference.

Since the existence of a choice function is generally sufficient for classical Social Choice problems, this issue was at least ignorable. However, in iterated aggregation, the result of the aggregation must not only be usable for making decisions, but must be interpretable as a new preference relation that may be involved in later aggregations; consequently, it must maintain clean semantics.

Secondly, the totality assumption is excessively restrictive for representing aggregate preferences. In general, a binary relation can express four possible relationships between a pair of alternatives and : and , and , and , and and . Totality reduces this set to the first three which, under the interpretation of relations as representing weak preference, correspond to the two strict orderings of and , and indifference. However, consider the situation where a couple is trying to choose between an Italian and an Indian restaurant, but one strictly prefers Italian food to Indian food, whereas the second strictly prefers Indian to Italian. The couple’s opinions are in conflict, a situation that does not fit into any of the three remaining categories. Thus, the totality assumption is essentially an assumption that conflicts do not exist. This, one may argue, is appropriate if we want to represent preferences of one agent (but see [Kahneman and Tversky1979] for persuasive arguments that individuals are often ambivalent). However, the assumption is inappropriate if we want to represent aggregate preferences since individuals will almost certainly have differences of opinion.

## 4 Generalized Belief States

Let us turn to the domain of belief aggregation. A total pre-order over the set of possible worlds is a fairly well-accepted representation for a belief state in the belief revision community [Grove1988, Katsuno and Mendelzon1991, Lehmann and Magidor1992, Gärdenfors and Makinson1994]. Instead of preference, relations represent relative likelihood, instead of indifference, equal likelihood. For the remainder of the paper, assume we are given some language with a satisfaction relation for . Let be a finite, non-empty set of possible worlds (interpretations) over . Suppose is a total pre-order on . The belief revision literature maintains that the conditional belief “if then ” (where and are sentences in ) holds if all the worlds in the choice set of those satisfying also satisfy ; we write . The individual’s unconditional beliefs are all those where is the sentence . If neither the belief nor its negation hold in the belief state, it is said to be agnostic with respect to , written .

It should come as no surprise that belief aggregation is formally similar to preference aggregation and, as a result, is also susceptible to the problems described in the previous section. We propose a solution to these problems which generalizes the total pre-order representation so as to capture information about conflicts.

### 4.1 Modular, transitive states

We take strict likelihood as primitive. Since strict likelihood is not necessarily total, it is possible to represent agnosticism and conflicting opinions in the same structure. This choice deviates from that of most authors, but are similar to those of Kreps [Kreps1990, p. 19] who is interested in representing both indifference and incomparability. Unlike Kreps, rather than use an asymmetric relation to represent strict likelihood (e.g., the strict version of a weak likelihood relation), we impose the less restrictive condition of modularity.

We formally define generalized belief states:

###### Definition 6

A generalized belief state is a modular, transitive relation over . The set of possible generalized belief states over is denoted .

We interpret to mean “there is reason to consider as strictly more likely than .” We represent equal likelihood, which we also refer to as “agnosticism,” with the relationship defined such that if and only if and . We define the conflict relation corresponding to , denoted , so that iff and . It describes situations where there are reasons to consider either of a pair of worlds as strictly more likely than the other. In fact, one can easily check that precisely represents conflicts in a belief state in the sense of Definition 4.

For convenience, we will refer to generalized belief states simply as belief states for the remainder of the paper except when to do so would cause confusion.

### 4.2 Discussion

Let us consider why our choice of representation is justified. First, we agree with the Social Choice community that strict likelihood should be transitive.

As we discussed in the previous section, there is often no compelling reason why agnosticism/indifference should not be transitive; we also adopt this view. However, transitivity of strict likelihood by itself does not guarantee transitivity of agnosticism. A simple example is the following: , so that . However, if we buy that strict likelihood should be transitive, then agnosticism is transitive identically when strict likelihood is also modular:

###### Proposition 5

Suppose a relation is transitive and is the corresponding agnosticism relation. Then is transitive iff is modular.

In summary, transitivity and modularity are necessary if strict likelihood and agnosticism are both required to be transitive.

We should point out that conflicts are also transitive in our framework. At first glance, this may appear undesirable: it is entirely possible for a group to disagree on the relative likelihood of worlds and , and and , yet agree that is more likely than . However, we note that this transitivity follows from the cycle-based definition of conflicts (Definition 4), not from our belief state representation. It highlights the fact that we are not only concerned with conflicts that arise from simple disagreements over pairs of alternatives, but those that can be inferred from a series of inconsistent opinions as well.

Now, to argue that modular, transitive relations are sufficient to capture relative likelihood, agnosticism, and conflicts among a group of information sources, we first point out that adding irreflexivity would give us the class of relations that are strict versions of total pre-orders, i.e., conflict-free. Let be the set of total pre-orders over , , the set of their strict versions.

###### Proposition 6

The set of irreflexive relations in is isomorphic to and, in fact, equals .

Secondly, the following representation theorem shows that each belief state partitions the possible worlds into sets of worlds either all equally likely or all potentially involved in a conflict, and totally orders these sets; worlds in distinct sets have the same relation to each other as do the sets.

###### Proposition 7

iff there is a partition of such that:

1. For every and , implies iff .

2. Every is either fully connected ( for all ) or fully disconnected ( for all ).

Figure 1 shows three examples of belief states: one which is a total pre-order, one which is the strict version of a total pre-order, and one which is neither.

Thus, generalized belief states are not a big change from the strict versions of total pre-orders. They merely generalize these by weakening the assumption that sets of worlds not strictly ordered are equally likely, allowing for the possibility of conflicts. Now we can distinguish between agnostic and conflicting conditional beliefs. A belief state is agnostic about conditional belief (i.e., ) if the choice set of worlds satisfying contains both worlds which satisfy and and is fully disconnected. It is in conflict about this belief, written , if the choice set is fully connected.

Finally, we compare the representational power of our definitions to those discussed in the previous section. First, subsumes the class of total pre-orders:

###### Proposition 8

and is the set of reflexive relations in .

Secondly, neither subsumes nor is subsumed by the set of total, quasi-transitive relations, and the intersection of the two classes is . Let be the set of total, quasi-transitive relations over , and , the set of their strict versions.

###### Proposition 9

1. .

2. .

3. if has at least three elements.

4. if has one or two elements.

Because modular, transitive relations represent strict preferences, it is probably fairer to compare them to the class of strict versions of total, quasi-transitive relations. Again, neither class subsumes the other, but this time the intersection is

:

###### Proposition 10

1. .

2. .

3. if has at least three elements.

4. if has one or two elements.

In the next section, we define a natural aggregation policy based on this new representation that admits clear semantics and obeys appropriately modified versions of Arrow’s conditions.

## 5 Single-agent belief state construction

Suppose an agent is informed by a set of sources, each with its individual belief state. Suppose further that the agent has ranked the sources by level of credibility. We propose an operator for constructing the agent’s belief state by aggregating the belief states of the sources in while accounting for the credibility ranking of the sources.

###### Example 1

We will use a running example from our space robot domain to help provide intuition for our definitions. The robot sends to earth a stream of telemetry data gathered by the spacecraft, as long as it receives positive feedback that the data is being received. At some point it loses contact with the automatic feedback system, so it sends a request for information to an agent on earth to find out if the failure was caused by a failure of the feedback system or by an overload of the data retrieval system. In the former case, it would continue to send data, in the latter, desist. As it so happens, there has been no overload, but the computer running the feedback system has hung. The agent consults the following three experts, aggregates their beliefs, and sends the results back to the robot:

1. , the computer programmer that developed the feedback program, believes nothing could ever go wrong with her code, so there must have been an overload problem. However, she admits that if her program had crashed, the problem could ripple through to cause an overload.

2. , the manager for the telemetry division, unfortunately has out-dated information that the feedback system is working. She was also told by the engineer who sold her the system that overloading could never happen. She has no idea what would happen if there was an overload or the feedback system crashed.

3. , the technician working on the feedback system, knows that the feedback system crashed, but doesn’t know whether there was a data-overload. Not being familiar with the retrieval system, she is also unable to speculate whether the data retrieval system would have overloaded if the feedback system had not failed.

Let and be propositional variables representing that the feedback and data retrieval systems, respectively, are okay. The belief states for the three sources are shown in Figure 2.

Let us begin the formal development by defining sources:

###### Definition 7

is a finite set of sources. With each source is associated a belief state .

We denote the agnosticism and conflict relations of a source by and , respectively. It is possible to assume that the belief state of a source is conflict free, i.e., acyclic. However, this is not necessary if we allow sources to suffer from the human malady of “being torn between possibilities.”

We assume that the agent’s credibility ranking over the sources is a total pre-order:

###### Definition 8

is a totally ordered finite set of ranks.

###### Definition 9

assigns to each source a rank.

###### Definition 10

is the total pre-order over induced by the ordering over . That is, iff ; we say is as credible as . is the restriction of to .

We use and to denote the asymmetric and symmetric restrictions of , respectively.444Note that, unlike the relations representing belief states, and are read in the intuitive way, that is, “greater” corresponds to “better.” The finiteness of () ensures that a maximal source (rank) always exists, which is necessary for some of our results. Weaker assumptions are possible, but at the price of unnecessarily complicating the discussion.

We are ready to consider the source aggregation problem. In the following, assume an agent is informed by a set of sources . We look at two special cases—equal-ranked and strictly-ranked source aggregation—before considering the general case.

### 5.1 Equal-ranked sources aggregation

Suppose all the sources have the same rank so that is fully connected. Intuitively, we want take all offered opinions seriously, so we take the union of the relations:

###### Definition 11

If , then is the relation .

By simply taking the union of the source belief states, we may lose transitivity. However, we do not lose modularity:

###### Proposition 11

If , then is modular but not necessarily transitive.

Thus, we know from Proposition 1 that we need only take the transitive closure of to get a belief state:

###### Definition 12

If , then is the relation .

###### Proposition 12

If , then .

Not surprisingly, by taking all opinions of all sources seriously, we may generate many conflicts, manifested as fully connected subsets of .

###### Example 2

Suppose all three sources in the space robot scenario of Example 1 are considered equally credible, then the aggregate belief state will be the fully connected relation indicating that there are conflicts over every belief.

### 5.2 Strictly-ranked sources aggregation

Next, consider the case where the sources are strictly ranked, i.e., is a total order. We define an operator such that lower-ranked sources refine the belief states of higher ranked sources. That is, in determining the ordering of a pair of worlds, the opinions of higher-ranked sources generally override those of lower-ranked sources, and lower-ranked sources are consulted when higher-ranked sources are agnostic:

###### Definition 13

If , then is the relation

.

The definition of the operator does not rely on being a total order, and we will use it in this more general setting in the following sub-section. However, in the case that is a total order, the result of applying is guaranteed to be a belief state.

###### Proposition 13

If and is a total order, then .

###### Example 3

Suppose, in the space robot scenario of Example 1, the technician is considered more credible than the manager who, in turn, is considered more credible than the programmer. The aggregate belief state, shown in Figure 3, informs the robot correctly that the feedback system has crashed, but that it shouldn’t worry about an overload problem and should keep sending data.

Note that this case of strictly-ranked sources is almost exactly that considered in [Maynard-Reid II and Shoham2000], except that the authors are not able to allow for conflicts in belief states. A surprising result they show is that standard AGM belief revision [Alchourrón et al.1985] can be modeled as the aggregation of two sources, the informant and the informee, where the informant is considered more credible than the informee.

### 5.3 General aggregation

In the general case, we may have several ranks represented and multiple sources of each rank. It will be instructive to first consider the following seemingly natural strawman operator, : First combine equi-rank sources using , then aggregate the strictly-ranked results using what is essentially :

###### Definition 14

Let . For any , let and , the corresponding agnosticism relation. Also, let . is the relation

indeed defines a legitimate belief state:

###### Proposition 14

If , then .

Unfortunately, a problem with this “divide-and-conquer” approach is it assumes the result of aggregation is independent of potential interactions between the individual sources of different ranks. Consequently, opinions that will eventually get overridden may still have an indirect effect on the final aggregation result by introducing superfluous opinions during the intermediate equi-rank aggregation step, as the following example shows:

###### Example 4

Let . Suppose such that with belief states and , and where . Then is . All sources are agnostic over and , yet and are in the result because of the transitive closure in the lower rank involving opinions ( and ) which actually get overridden in the final result.

Because of these undesired effects, we propose another aggregation operator which circumvents this problem by applying refinement (as defined in Definition 13) to the set of source belief states before infering new opinions via closure:

###### Definition 15

The rank-based aggregation of a set of sources is .

Encouragingly, outputs a valid belief state:

If , then .

###### Example 5

Suppose, in the space robot scenario of Example 1, the technician is still considered more credible than the manager and the programmer, but the latter two are considered equally credible. The aggregate belief state, shown in Figure 5, still gives the robot the correct information about the state of the system. The robot also learns for future reference that there is some disagreement over whether or not there would have been a data overload if the feedback system were working.

We observe that , when applied to the set of sources in Example 4, does indeed bypass the problem described above of extraneous opinion introduction:

###### Example 6

Assume , , and are as in Example 4. .

We also observe that behaves well in the special cases we’ve considered, reducing to when all sources have equal rank, and to when the sources are totally ranked:

###### Proposition 16

Suppose .

1. If is fully connected, .

2. If is a total order, .

### 5.4 Arrow, revisited

Finally, a strong argument in favor of is that it satisfies appropriate modifications of Arrow’s conditions. Let be an operator which aggregates the belief states , …, over of sources , respectively, and let . We consider each condition separately.

##### Restricted range

The output of the aggregation function will be a modular, transitive belief state rather than a total pre-order.

###### Definition 16

(modified) Restricted Range: The range of is .

##### Unrestricted domain

Similarly, the input to the aggregation function will be modular, transitive belief states of sources rather than total pre-orders.

###### Definition 17

(modified) Unrestricted Domain: For each , can be any member of .

##### Pareto principle

Generalized belief states already represent strict likelihood. Consequently, we use the actual input and output relations of the aggregation function in place of their strict versions to define the Pareto principle. Obviously, because we allow for the introduction of conflicts, will not satisfy the original formal Pareto principle which essentially states that if all sources have an unconflicted belief that one world is strictly more likely than another, this must also be true of the aggregated belief state. Neither condition is necessarily stronger than the other.

###### Definition 18

(modified) Pareto Principle: If for all , then .

##### Independence of irrelevant alternatives

Conflicts are defined in terms of cycles, not necessarily binary. By allowing the existence of conflicts, we effectively have made it possible for outside worlds to affect the relation between a pair of worlds, viz., by involving them in a cycle. As a result, we need to weaken IIA to say that the relation between worlds should be independent of other worlds unless these other worlds put them in conflict.

###### Definition 19

(modified) Independence of Irrelevant Alternatives (IIA): Suppose such that for all , and . If, for , iff for all , , and , then iff .

##### Non-dictatorship

As with the Pareto principle definition, we use the actual input and output relations to define non-dictatorship since belief states represent strict likelihood. From this perspective, our setting requires that informant sources of the highest rank be “dictators” in the sense considered by Arrow. However, the setting originally considered by Arrow was one where all individuals are ranked equally. Thus, we make this explicit in our new definition of non-dictatorship by adding the pre-condition that all sources be of equal rank. Now, treats a set of equi-rank sources equally by taking all their opinions seriously, at the price of introducing conflicts. So, intuitively, there are no dictators. However, because Arrow did not account for conflicts in his formulation, all the sources will be “dictators” by his definition. We need to modify the definition of non-dictatorship to say that no source can always push opinions through without them ever being contested.

###### Definition 20

(modified) Non-Dictatorship: If for all , then there is no such that, for every combination of source belief states and every , and implies and .

We now show that indeed satisfies these conditions:

###### Proposition 17

Let and . satisfies (the modified versions of) restricted range, unrestricted domain, Pareto principle, IIA, and non-dictatorship.

## 6 Multi-agent fusion

So far, we have only considered the case where a single agent must construct or update her belief state once informed by a set of sources. Multi-agent fusion is the process of aggregating the belief states of a set of agents, each with its respective set of informant sources. We proceed to formalize this setting.

An agent is informed by a set of sources . Agent ’s induced belief state is the belief state formed by aggregating the belief states of its informant sources, i.e., . Assume the set of agents to fuse agree upon (and, consequently, ).555We could easily extend the framework to allow for individual rankings, but we felt that the small gain in generality would not justify the additional complexity and loss of perspicuity. Similarly, we could consider each agent as having a credibility ordering only over its informant sources. However, it is unclear how, for example, crediblity orderings over disjoint sets of sources should be combined into a new credibility ordering since their union will not be total. We define the fusion of this set to be an agent informed by the combination of informant sources:

###### Definition 21

Let be a set of agents such that each agent is informed by . The fusion of , written , is an agent informed by .

Not surprisingly given its set-theoretic definition, fusion is idempotent, commutative, and associative. These properties guarantee the invariance required in multi-agent belief aggregation applications such as our space robot domain.

In the multi-agent space robot scenario described in Section 1, we only have a direct need for the belief states that result from fusion. We are only interested in the belief states of the original sources in as far as we want the fused belief state to reflect its informant history. An obvious question is whether it is possible to compute the belief state induced by the agents’ fusion solely from their initial belief states, that is, without having to reference the belief states of their informant sources. This is highly desirable because of the expense of storing—or, as in the case of our space robot example, transmitting—all source belief states; we would like to represent each agent’s knowledge as compactly as possible.

In fact, we can do this if all sources have equal rank. We simply take the transitive closure of the union of the agents’ belief states:

###### Proposition 18

Let and be as in Definition 21, , agent ’s induced belief state, and , fully connected. If , then is ’s induced belief state.

Unfortunately, the equal rank case is special. If we have sources of different ranks, we generally cannot compute the induced belief state after fusion using only the agent belief states before fusion, as the following simple example demonstrates:

###### Example 7

Let . Suppose two agents and are informed by sources with belief state and with belief state , respectively. ’s belief state is the same as ’s and ’s is the same as ’s. If , then the belief state induced by is , whereas if , then it is . Thus, just knowing the belief states of the fused agents is not sufficient for computing the induced belief state. We need more information about the original sources.

However, if sources are totally pre-ordered by credibility, we can still do much better than storing all the original sources. It is enough to store for each opinion of the rank of the highest-ranked source supporting it. We define pedigreed belief states which enrich belief states with this additional information:

###### Definition 22

Let be an agent informed by a set of sources . ’s pedigreed belief state is a pair where and such that . We use to denote the restriction of ’s pedigreed belief state to , that is, .

We verify that a pair’s label is, in fact, the rank of the source used to determine the pair’s membership in , not that of some higher-ranked source:

###### Proposition 19

Let be an agent informed by a set of sources and with pedigreed belief state . Then

 x≺Ary

iff

 ∃s∈S. x

The belief state induced by a pedigreed belief state is, obviously, the transitive closure of .

Now, given only the pedigreed belief states of a set of agents, we can compute the new pedigreed belief state after fusion. We simply combine the labeled opinions using our refinement techniques.

###### Proposition 20

Let and be as in Definition 21, , a total pre-order, and . If

1. is the relation

 ⎧⎨⎩(x,y):∃Ai∈A,r∈R. x≺Airy∧(∀Aj∈A,r′>r∈R. x∼Ajr′y)⎫⎬⎭

over ,

2. such that , and

then is ’s pedigreed belief state.

From the perspective of the induced belief states, we are essentially discarding unlabeled opinions (i.e., those derived by the closure operation) before fusion. Intuitively, we are learning new information so we may need to retract some of our inferred opinions. After fusion, we re-apply closure to complete the new belief state. Interestingly, in the special case where the sources are strictly-ranked, the closure is unnecessary:

###### Proposition 21

If and are as in Definition 21, is a total order, and is the pedigreed belief state of , then .

###### Example 8

Let’s look once more at the space robot scenario considered in Example 1. Suppose the arrogant programmer is not part of the telemetry team, but instead works for a company on the other side of the country. Then the robot has to request information from two separate agents, one to query the manager and technician and one to query the programmer. Assume that the agents and the robot all rank the sources the same, assigning the technician rank 2 and the other two agents rank 1, which induces the same credibility ordering used in Example 5. The agents’ pedigreed belief states and the result of their fusion are shown in Figure 5.

The first agent does not provide any information about overloading and the second agent provides incorrect information. However, we see that after fusing the two, the robot has a belief state that is identical to what it computed in Example 5 when there was only one agent informed by all three sources (we’ve only separated the top set of worlds so as to show the labeling). Consequently, it now knows the correct state of the system. And, satisfyingly, the final result does not depend on the order in which the robot receives the agents’ reports.

The savings obtained in required storage space by this scheme can be substantial. Whereas explicitly storing all of an agent’s informant sources requires amount of space in the worst case (when all the sources’ belief states are fully connected relations), storing a pedigreed belief state only requires space in the worst case. Moreover, not only does the enriched representation allow us to conserve space, but it also provides for potential savings in the efficiency of computing fusion since, for each pair of worlds, we only need to consider the opinions of the agents rather than those of all the sources in the combined set of informants.

Incidentally, if we had used as the basis for our general aggregation, simply storing the rank of the maximum supporting sources would not give us sufficient information to compute the induced belief state after fusion. To demonstrate this, we give an example where two pairs of sources induce the same annotated agent belief states, yet yield different belief states after fusion:

###### Example 9

Let , , and be as in Example 4. Suppose agents , , , and are informed by sets of sources , , , and , respectively, where , , and . dictates that the pedigreed belief states of all four agents equal with all opinions annotated with . In spite of this indistinguishability, if and , then ’s induced belief state equals , i.e., , whereas ’s is .

## 7 Conclusion

We have described a semantically clean representation for aggregate beliefs which allows us to represent conflicting opinions without sacrificing the ability to make decisions. We have proposed an intuitive operator which takes advantage of this representation so that an agent can combine the belief states of a set of informant sources totally pre-ordered by credibility. Finally, we have described a mechanism for fusing the belief states of different agents which iterates well.

The aggregation methods we have discussed here are just special cases of a more general framework based on voting. That is, we account not only for the ranking of the sources supporting or disagreeing with an opinion (i.e., the quality of support), but also the percentage of sources in each camp (the quantity of support). Such an extension allows for a much more refined approach to aggregation, one much closer to what humans often use in practice. Exploring this richer space is the subject of further research.

Another problem which deserves further study is developing a fuller understanding of the properties of the , , and operators and how they interrelate.

#### Acknowledgements

Pedrito Maynard-Reid II was partly supported by a National Physical Science Consortium Fellowship. The final version of this paper was written with the financial support of the Jean et Hélène Alfassa Fund for Research in Artificial Intelligence.

## References

• [Alchourrón et al.1985] Carlos E. Alchourrón, Peter Gärdenfors, and David Makinson. On the logic of theory change: Partial meet contraction and revision functions. Journal of Symbolic Logic, 50:510–530, 1985.
• [Arrow1963] Kenneth J. Arrow. Social Choice and Individual Values. Wiley, New York, 2nd edition, 1963.
• [Gärdenfors and Makinson1994] Peter Gärdenfors and David Makinson. Nonmonotonic inference based on expectations. Artificial Intelligence, 65(1):197–245, January 1994.
• [Grove1988] Adam Grove. Two modellings for theory change. Journal of Philosophical Logic, 17:157–170, 1988.
• [Kahneman and Tversky1979] D. Kahneman and A. Tversky. Prospect theory: An analysis of decision under risk. Econometrica, 47(2):263–291, March 1979.
• [Katsuno and Mendelzon1991] Hirofumi Katsuno and Alberto O. Mendelzon. Propositional knowledge base revision and minimal change. Artificial Intelligence, 52(3):263–294, 1991.
• [Kreps1990] David M. Kreps. A Course in Microeconomic Theory. Princeton University Press, 1990.
• [Lehmann and Magidor1992] Daniel Lehmann and Menachem Magidor. What does a conditional knowledge base entail? Artificial Intelligence, 55(1):1–60, May 1992.
• [Maynard-Reid II and Shoham2000] Pedrito Maynard-Reid II and Yoav Shoham. Belief fusion: Aggregating pedigreed belief states. Journal of Logic, Language, and Information, 2000. To appear.
• [Sen1986] Amartya Sen. Social choice theory. In K. J. Arrow and M. D. Intriligator, editors, Handbook of Mathematical Economics, volume III, chapter 22, pages 1073–1181. Elevier Science Publishers, 1986.

## Appendix A Proofs

###### Proposition 1

1. The transitive closure of a modular relation is modular.

2. Every transitive relation is quasi-transitive.

3. [Sen1986] Every quasi-transitive relation is acyclic.

1. Suppose a relation over finite set is modular, and is the transitive closure of . Suppose and . Then there exist such that . Since is modular and , either or . In the former case, , so . In the latter case, , so .

2. Suppose is a finite set, , is a transitive relation over , and is its strict version. Suppose and . Then , , , and . and imply , and and imply , both by transitivity. So .

###### Proposition 2

[Sen1986] Given a relation over a finite set , the choice set operation defines a choice function iff is acyclic.

###### Proposition 3

[Arrow1963] There is no aggregation operator that satisfies restricted range, unrestricted domain, (weak) Pareto principle, independendence of irrelevant alternatives, and nondictatorship.

###### Proposition 4

Let be a relation over a finite set and let be its symmetric restriction (i.e., iff and ). If is total and quasi-transitive but not transitive, then is not transitive.

• Let be a total, quasi-transitive, non-transitive relation. First, such a relation exits: if , it is easily verified that the relation is total, quasi-transitive, but not transitive.

Suppose and but . By totality, , so . If , then by quasi-transitivity, a contradiction. Thus, . Similarly, if , then , a contradiction, so . But , so . Therefore, is not transitive.

###### Proposition 5

Suppose a relation is transitive and is the corresponding agnosticism relation. Then is transitive iff is modular.

• Suppose is transitive and suppose , . We prove by contradiction: Suppose and . By transitivity, and , so and . By assumption, , so , a contradiction.

Suppose, instead, is modular and suppose and , . Then , , , and . By modularity, and , so .

###### Proposition 6

The set of irreflexive relations in is isomorphic to and, in fact, equals .

• Let . Suppose is irreflexive. Let be defined as iff . We first show that is the strict version of . Suppose is the strict version of . If , then and , so . If, instead, , then . By totality, , so .

We show that . If then . Otherwise, . But since is irreflexive, (otherwise by transitivity), so and is total. Next, suppose and . Then and . By modularity, , so , so is transitive.

Now suppose and is its strict version. First we show that is modular. Suppose . Then and . Since is total, or . Suppose . Whether or , by transitivity. Suppose, instead, . Then and , both by transitivity. We conclude that and , or and , so or . Second, transitivity of follows immediately from Proposition 1 and the transitivity of . Finally, is irreflexive since it is asymmetric.

###### Proposition 7

iff there is a partition of such that:

1. For every and , implies iff .

2. Every is either fully connected ( for all ) or fully disconnected ( for all ).

• We refer to the conditions in the proposition as conditions 1 and 2, respectively. We prove each direction of the proposition separately.

Suppose , that is, is a modular and transitive relation over . We use a series of definitions and lemmas to show that a partition of exists satisfying conditions 1 and 2. We first define an equivalence relation by which we will partition . Two elements will be equivalent if they “look the same” from the perspective of every element of :

###### Definition 23

iff for every , iff and iff .

###### Lemma 7.1

is an equivalence relation over .

• Suppose . For every , iff and iff , so . Therefore, is reflexive.

Suppose and . Then for every , iff and iff . But then for every , iff and iff . Therefore, , so is symmetric.

Suppose , , and . Suppose further that . By definition of , iff and iff , and iff and iff . Therefore, iff and iff . Since is arbitrary, , so is transitive.

partitions into its equivalence classes. We use to denote the equivalence class containing , that is, the set . Observe that two worlds in conflict always appear in the same equivalence class:

###### Lemma 7.2

If and , then .

• Suppose and . Since is an equivalence class, it suffices to show that , that is, . Suppose . By transitivity, if , then ; if , then ; if , then ; and, if then . Thus, iff and iff , and since is arbitrary, .

We now define a total order over these equivalence classes:

###### Definition 24

For all , iff or .

###### Lemma 7.3

is well-defined, that is, if and , then iff , for all .

• Suppose and , . By the definition of , for every , iff . In particular, iff . Also by the definition of , for every , iff . In particular, iff . Therefore, iff .

###### Lemma 7.4

is a total order over the equivalence classes of defined by .

• Suppose . We first show that is total. By definition of , if or , then or , respectively. Suppose and , and suppose . By modularity of , implies , implies , implies , and implies , so . Therefore, , so by the definition of .

Next, we show that is anti-symmetric. Suppose and . Then or and . In the former case we are done, in the latter, the result follows from Lemma 7.2.

Finally, we show that is transitive. Suppose and . Obviously, if or , then . Suppose not. Then and , so by the transitivity of . Therefore, by the definition of .

We name the members of the partition such that iff , where is an integer. Such a naming exists since every finite, totally ordered set is isomorphic to some finite prefix of the integers.

We now check that this partition satisfies the two conditions. For the first condition, suppose , , and . We want to show that iff . Since , . Suppose . Then , so . Since , by the definition of . Now suppose, instead, that . Then by the definition of , so . Since , by Lemma 7.2. Since and , by the definition of , so . Thus, .

Finally, we show that each is either fully connected or fully disconnected. Suppose so that . It suffices to show that iff . By the definition of , iff , and iff . Suppose . Then, and , so by transitivity of . Suppose now, . Then, and , so by modularity of .

Suppose is a partition of and is a relation over satisfying the given conditions. We want to show that is modular and transitive. We first give the following lemma:

###### Lemma 7.5

Suppose is a partition of and is a relation over satisfying condition 1. If , , , and , then .

• If , we’re done. Suppose . Then, since , by condition 1.

We now show is modular. Suppose , , and . Then by Lemma 7.5. Suppose . Then or by the modularity of . Suppose or . Then or by condition 1. Otherwise , so . Since , is fully connected by condition 2, so (and ).

Finally, we show that is transitive. Suppose , , , , and . By Lemma 7.5, and , so by the transitivity of . Suppose . Then by condition 1. Otherwise , so . Since , is fully connected by condition 2, so .

(END OF PROPOSITION 7 PROOF)

###### Proposition 8

and is the set of reflexive relations in .

• We first show that . Let and . By definition, is transitive. Suppose . Since is total, or . If , then by transitivity, so is modular. On the other hand, the empty relation over is modular and transitive, but not total and, consequently, not in .

Now we show that is in if and only if it is reflexive. If , it is total, so it is reflexive. If, instead, is reflexive, then so, by modularity, or . Thus, is total. And, since , it is transitive.

###### Proposition 9

1. .

2. .

3. if has at least three elements.

4. if has one or two elements.

1. Suppose . Then is total and transitive and, hence, in . Suppose . By definition, is total. Also by definition, it is transitive, so by Proposition 1, it is quasi-transitive and, thus, in . By Proposition 8, and, so, in .

2. The empty relation is modular and transitive, but not total and, so, not in .

3. Suppose and are distinct elements of . The relation is total, and, since the strict version is which is transitive, it is also quasi-transitive. However, if there are at least three elements in , it is not transitive and, so, not in .

4. Suppose has one element. Then contains both possible relations over , whereas contains only the fully connected relation over .

Suppose has two elements and . Then contains empty relation, the fully connected relation, and all the remaining eight relations which contain either or , but not both. , on the other hand, only contains the three reflexive relations containing either or .

1. .