Conditional probability and improper priors

06/07/2020 ∙ by Gunnar Taraldsen, et al. ∙ 0

The purpose of this paper is to present a mathematical theory that can be used as a foundation for statistics that include improper priors. This theory includes improper laws in the initial axioms and has in particular Bayes theorem as a consequence. Another consequence is that some of the usual calculation rules are modified. This is important in relation to common statistical practice which usually include improper priors, but tends to use unaltered calculation rules. In some cases the results are valid, but in other cases inconsistencies may appear. The famous marginalization paradoxes exemplify this latter case. An alternative mathematical theory for the foundations of statistics can be formulated in terms of conditional probability spaces. In this case the appearance of improper laws is a consequence of the theory. It is proved here that the resulting mathematical structures for the two theories are equivalent. The conclusion is that the choice of the first or the second formulation for the initial axioms can be considered a matter of personal preference. Readers that initially have concerns regarding improper priors can possibly be more open toward a formulation of the initial axioms in terms of conditional probabilities. The interpretation of an improper law is given by the corresponding conditional probabilities. Keywords: Axioms of statistics, Conditional probability space, Improper prior, Projective space

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Statistical developments are driven by applications, theory, and most importantly the interplay between applications and theory. The following is intended for readers that can appreciate the importance of the theoretical foundation for probability theory as given by the axioms of

Kolmogorov (1933). According to the recipe of Kolmogorov a random element in a set is identified with a measurable function where is the basic probability space that the whole theory is based upon.

The use of improper priors is common in the statistical literature, but most often without reference to a corresponding theoretical foundation. It will here be explained how the theory of conditional probability spaces as developed by Renyi (1970a) is related to a theory for statistics that includes improper priors. This theory has been presented in a simplified form with some elementary examples by Taraldsen and Lindqvist (2010). The idea is to use the above recipe given by Kolmogorov, but generalized by assuming that is defined by the use of a -finite measure. The underlying law itself is not a -finite measure, but it is an equivalence class of -finite measures. A more precise formulation is given by Definition 2 below.

In oral presentations of the theory related to improper priors it is quite common that someone in the audience makes the following claim: Improper priors are just limits of proper priors, so we need not consider improper priors. We strongly disagree with this even though it is true that improper priors can be obtained as limits of proper priors. The reason is perhaps best explained by analogy with a more familiar example: It is true that the real number system is obtained as a limit of the rational numbers. A precise construction, and this is important, is found by the aid of equivalence classes of Cauchy sequences. Nonetheless, most people prefer to think of real numbers as such without reference to rational numbers. Real numbers are just limits of rational numbers, but we use them with properties as given by the axioms of the real number system.

The reader will not find any new algorithms or methods for the solution of practical problems here. The presented theory can, however, be used to put many known solutions to practical problems on a more solid theoretical foundation. This additional effort is necessary to avoid and explain contra-dictionary results as exemplified by for instance the famous marginalization paradoxes (Stone and Dawid, 1972; Dawid et al., 1973). The theory also gives a natural frame for the proof of optimality of inference based on fiducial theory (Taraldsen and Lindqvist, 2013a) and a proof of coincidence of a fiducial distribution and certain Bayesian posteriors based on improper priors (Taraldsen and Lindqvist, 2013b)

. The theory has also been used for a rigorous specification of intrinsic conditional autoregression models

(Lavine and Hodges, 2012). These models are widely used in spatial statistics, dynamic linear models, and elsewhere.

2 Conditional measures

The aim in the following is to formulate a theory that can be used to provide a foundation for statistics that includes improper priors. A less technical presentation of this with some elementary examples has already been provided by Taraldsen and Lindqvist (2010). They show in particular by examples that this theory is different from the alternative theory for improper priors provided by Hartigan (1983). This section gives a condensed presentation of the mathematical ingredients. Most definitions are standard as presented by for instance Rudin (1987), but some are not standard and are emphasized in the text. An example is the concept of a C-measure as introduced below.

Let be a nonempty set, and a family of subsets that includes the empty set . The family is a -field if it is closed under formation of countable unions and complements. A set is measurable if it belongs to . A measurable space is a set equipped with a -field . The same symbol is here used to denote both the set and the space. The notation is also used to denote the measurable space. The convention here is to use the term space to denote a set with some additional structure.

A measure space is a measurable space equipped with a measure . A measure is a function that is countably additive: The equality holds for disjoint measurable .

Definition 1.

Let be a measure space. An admissible condition is a measurable set with .

The measure space is -finite if there exists a sequence of admissible conditions such that . A probability space is a measure space such that , and is then said to be a probability measure.

Consider the set of all -finite measures on a fixed measurable space . The set includes in particular all probability measures, and the following gives new concepts also when restricted to probability measures. Two elements and in are defined to be equivalent if for some positive number . This defines an equivalence relation and the quotient space defines the set of C-measures. It should in particular be observed that any topology on induces a corresponding quotient topology on . Convergence of C-measures is an important topic, but the study of this is left for the future. Some further discussion will, however, be provided in Section 4. A C-measure space is a measurable space equipped with a C-measure . This means that is an equivalence class of -finite measures. The term conditional probability space will here be used as an equivalent term. This convention will be motivated next, and is further elaborated in Section 4.

Let be a conditional probability space, and let be an admissible condition. The conditional law, or equivalently the conditional measure, is then well defined by where . It is well defined since it does not depend on the choice of , and the resulting conditional law is a probability measure. This argument gives:

Proposition 1.

A conditional probability space defines a unique family of probability spaces indexed by the admissible sets: .

For ease of reference we restate the following definition.

Definition 2.

Let be a measurable space. A conditional measure is a set of -finite measures defined by a -finite measure defined on . Let be an admissible condition and let be measurable. The formula defines the conditional probability of given . A conditional measure space is a measurable space equipped with a conditional measure .

In general probability and statistics it is most useful to extend the definition of conditional probability and expectation to include conditioning on -fields and statistics. This can be done also in the more general context here. The main new ingredient is given by the definition of -finite -fields and -finite measurable functions.

Let be a measure space. Assume that is a -finite -field in the sense that is -finite where is the restriction of to . This implies that is also -finite. Let . The conditional measure is defined by the measurable function uniquely determined by the relation

(1)

which is required to hold for all measurable subsets .

The existence and uniqueness proof follows by observing: (i) defines a measure on . (ii) is dominated by the measure . (iii) The Radon-Nikodym theorem gives existence and uniqueness of the conditional as the density of with respect to so the claim follows. The uniqueness is only as a measurable function defined on the measure space and the conditional probability is more properly identified with an equivalence class of measurable functions.

The defining equation (1) shows that for all . It can be concluded that the conditional measure is well defined if is a conditional probability space. An immediate consequence is , so the conditional measures are all normalized. The term conditional probability will motivated by this be used as equivalent to the term conditional measure. This is as above for the elementary conditional measure.

The following example demonstrates that this conditional probability generalizes the elementary conditional probabilities . Let be the -field generated by a countable partition of into disjoint admissible sets . It follows that is -finite. Assuming this the conditional expectation is given by for .

The definitions presented so far lead naturally to the definition of the category of conditional probability spaces with a corresponding class of arrows. The study of this, and functors to related categories, will not be pursued here. This more general theory gives, however, alternative motivation for some of the concepts presented next.

A function is measurable if the inverse image of every measurable set is measurable: is measurable for all measurable . Let be a measure on . The image measure is defined by

(2)

A measurable function is by definition -finite if is -finite. A direct verification shows that a -finite function pushes a conditional probability space structure on into a conditional probability space structure on . This follows from the above and the identity . Consequently, if is a C-measure, then the C-measure  is well defined if is -finite. The definition given here of a

-finite function is a generalization of the concept of a regular random variable as defined by

Renyi (1970a, p.73). The definition of -finite -fields and -finite measurable functions can be reformulated as follows.

Definition 3.

Let be a -finite measure space. A -field is -finite if restricted to is -finite. Let be a measurable space. A measurable is -finite if the -field is -finite.

The previous arguments show that the -finite functions can play the role as arrows in the category of conditional probability spaces. The -finite functions can also be used to define conditional probabilities just as -finite -fields did in equation (1). It will be a generalization since the previous definition is obtained by consideration of the function as a function taking values in the space equipped with the -finite -field in the construction that follows.

Assume that is -finite. The conditional probability is defined by the relation

(3)

which is required to hold for all measurable subsets . The existence and uniqueness proof follows by an argument similar to the argument after equation (1). The identity holds and the conditional measure is well defined for a C-measure .

Composition of the functions and defines the conditional probability as a measurable function defined on . This function is measurable with respect to the initial -field generated by . The -finiteness of is equivalent with the -finiteness of . Direct verification shows that the definitions of conditional probability given by equations (1) and (3) coincide in the sense that .

The conclusion is that a conditional probability space is not only equipped with the family of elementary conditional probabilities , but also a family of conditional probabilities for each -finite .

The conditional probability on is defined by

(4)

The function must be -finite, but it is not required that is -finite. It follows that

(5)

The case and gives the defining equation (3) as a special case of the more general factorization given by equation (5).

The previous discussion can be summarized by:

Proposition 2.

A conditional measure space , a -finite , and a measurable define a unique measurable family of conditional measures defined on .

It does not follow in general that there exists a version of such that this is a measure for almost all . A sufficient condition for this is that is a Borel space (Schervish, 1995, p.618). This is the case if is in one-one measurable correspondence with a measurable subset of an uncountable complete separable metric space (Royden, 1989, p.406). The corresponding version of the conditional probability is then said to be a regular conditional probability. Integration with respect to can nonetheless be defined without any regularity conditions, and the factorization given by equation (5) holds in the most general case as stated. The possibility of this more general integral with respect to conditional probabilities was indicated already by Kolmogorov (1933, eq.10 on p.54).

3 Statistics with Improper Priors

Let be the observed result of an experiment. It will be assumed that can be identified with an element of a mathematically well defined set . The set should include all other possible outcomes that could have been observed as a result of the experiment. The observed result can be a number, a collection of numbers organized in some way, a continuous function, a self-adjoint linear operator, a closed subset of a topological space, or any other element of a well defined set corresponding to the experiment under consideration.

Assume that the sample space is equipped with a -field so that is a measurable space. Assume furthermore that is a probability space for each in the model parameter space . The family specifies a statistical model for the experiment.

A predominant family of example in the applied statistical literature is given by letting

be the multivariate Gaussian distribution on

with covariance matrix and mean where . The simplest special case is given by and which corresponds to independent sampling from the univariate Gaussian with unknown mean

and variance equal to

. Other examples included are given by ANOVA models with fixed and random effects, more general regression models, structured equations models with latent variables from the fields of psychology and economy (Jöreskog, 1970), and a variety of models from the statistical signal processing literature (Van Trees, 2002). These models correspond to specific choices of the functional dependence on in and .

The contents so far coincides with the definition of a statistical model as found in standard statistical literature. One exception is the choice of the notation for the statistical model. This choice indicates the connection to the theory of conditional probability spaces as will be explained by the introduction of further assumptions.

It is assumed that the statistical model is based upon an underlying abstract conditional probability space . This includes the case of an underlying abstract probability space as formulated by Kolmogorov (1933) as a special case, but seen as a conditional probability space. It is abstract in the sense that it is assumed to exist, but it is not specified. It is assumed that the model parameter space is a measurable space , that there exists a -finite measurable , and that there exists a measurable so that the resulting conditional probability as defined in equation (4) coincides with the specified statistical model.

Existence of , , , and can be proved in many concrete cases by consideration of the product space equipped with the -finite measure obtained from the choice of a -finite measure . This includes in particular the multivariate Gaussian example indicated above. As soon as existence is established it is assumed for the further theoretical development that , , , and are abstract unspecified objects with the required resulting statistical model as a consequence.

It can be observed that it is required that the mapping is measurable for all measurable for the above construction to be possible. This condition is trivially satisfied in most examples found in applications, and is furthermore a typical assumption in theoretical developments. A good example of the latter is given by the mathematical proof of the factorization theorem for sufficient statistics (Halmos and Savage, 1949).

The basis for frequentist inference is then the observation and the specified statistical model based on the underlying abstract conditional probability space .

The basis for Bayesian inference is as for frequentist inference, but the prior distribution

is also specified. The basis for Bayesian inference is hence the observation

and the joint distribution

. The conclusions of Bayesian inference are derived from the posterior distribution , which is well defined by equation (4) if is -finite. This result can be considered to be a very general version of Bayes theorem as promised in the Abstract. A discussion of a more elementary version involving densities is given by Taraldsen and Lindqvist (2010).

The importance of the -finiteness of has also been observed by others, but then as a requirement on the prior. Berger et al. (2009, p.911) includes this requirement as a part of the definition of a permissible prior. The definition as formulated in this section can be used as a generalization of this part to cases not defined by densities.

A summary of the contents in this section is given by:

Definition 4 (Statistical model).

A statistical model is specified by a family of probability spaces indexed by the model parameter space with the additional structure defined in the following.

It is assumed that all objects are defined based on the underlying conditional probability space . The observation is given by a measurable function and the model parameter is given by a -finite measurable function . It is assumed that the family of probability measures is given by the conditional law, so .

A Bayesian statistical model is specified by a statistical model together with a specification of the prior law

. It is assumed that is -finite, and then the resulting marginal law is a conditional measure and the resulting posterior law is well defined.

In the previous the prior , the marginal , and the joint distribution are C-measures with corresponding conditional probability spaces , , and . The interpretation of the prior is in terms of the corresponding elementary conditional laws . The same holds for the other improper laws.

Bayesian inference is essentially unique. This is in contrast to frequentist inference which most often offer many different possible inference procedures for a given problem. An analogous situation occurs in applied metrology where it is common to have many different measurement instruments available for the measurement of a physical quantity. The choice of instrument depends on the actual situation and purpose of the experiment at hand.

The previous gives a mathematical definition of a statistical model and a Bayesian statistical model based on the concept of a conditional measure. The concept of a fiducial statistical model can also be defined based on the same theory. The necessary ingredients and further discussion of this have been presented by Taraldsen and Lindqvist (2013a, b).

4 Renyi Conditional Probability Spaces

Renyi (1970a, p.38-) gives a definition of a conditional probability space based on a family of objects . A condensed summary of the initial ingredients in this theory is presented next, but with some extensions and minor changes in the choice naming conventions. The purpose is to show the close connection to the concept of a conditional measure space as discussed in the previous two section. The final words of Renyi on this subject are recommended for a more thorough (Renyi, 1970a) and pedagogical presentation (Renyi, 1970b) of the theory as formulated and motivated by Renyi.

Definition 5 (Bunch).

Let be a measurable space. A family is a bunch in if

  1. .

  2. There exist such that .

  3. The empty set does not belong to .

Example 1 Let be the real line equipped with the Borel -field. Let be the set of finite nonempty unions of open intervals on the form where are integers. The family is then a countable bunch.

Definition 6 (Renyi space).

A Renyi space is a measurable space equipped with a family of probability measures indexed by a bunch which fulfill and , and the identity

(6)

A Renyi space extends a Renyi space by definition if and for all . The extension is strict if and . A Renyi space is maximal if a strict extension does not exist.

Example 1 (continued) Let be the uniform probability law on for each . This gives a Renyi space where is the real line with the Borel -field. The family is in this case a countable family of probability measures.

Let be the C-measure given by Lebesgue measure on the real line. The elementary conditional measures for admissible defines a Renyi space which contains the Renyi space in the sense that and for all . It follows from the results presented next that is a maximal extension of .

It follows generally that a C-measure space generates a unique Renyi space through the elementary conditional measures . The same symbol is here used for two different concepts. Further excuse for this abuse of notation is given by the following structure result:

Proposition 3.

A Renyi space generates a unique conditional measure space. The corresponding resulting Renyi space is a maximal extension of the initial Renyi space.

Proof.

Let be the Renyi space. It will be proved that there exists a -finite measure such that for all , and that the C-measure  is unique.

The first step in the proof is to pick an arbitrary and define for . This choice gives the normalization . This definition is extended to measurable by . An arbitrary measurable can be written as a disjoint union of measurable where each is contained in some set from the bunch. The measure is then finally defined by .

Equation (6) can be used to prove that the previous definition of based on a for does not depend on the choice of . This, and further proof of consistency and uniqueness of is left to the reader. An alternative is to consult the proof of a corresponding result given by Renyi (1970a, p.40-43). ∎

Two different Renyi spaces can generate the same C-measure space. A concrete example is provided by consideration of the bunch generated by the intervals in addition to the two bunches considered in Example 1. It follows generally that the set of Renyi spaces based on a given measurable space is strictly larger than the set of C-measures on the measurable space.

Corollary 1.

A Renyi space has a unique extension to a maximal Renyi space. The set of maximal Renyi spaces is in one-one correspondence with the set of C-measure spaces.

Proof.

Let be a Renyi space and let be the corresponding generated C-measure space. The Renyi space given by the set of admissible conditions is then a unique maximal extension. Uniqueness and maximality follows since any Renyi space that contains will generate the C-measure space by the construction given in the proof of Proposition 3. ∎

A more general concept of a conditional probability space was originally introduced by Renyi (1955), and a corresponding more general structure theorem was proved by Császár (1955). Renyi (1970a, p.95) refers to these more general spaces as generalized conditional probability spaces. They are truly more general and a generalized conditional probability space is not necessarily generated by a single -finite measure.

5 Discussion

The distinction between a -finite measure space and the corresponding C-measure space could at first sight seem trivial. For a -finite measure the corresponding C-measure  is an equivalence class of -finite measures in the set of all -finite measures on the measurable space . It follows, as stated earlier, that any topology on the set of -finite measures gives a corresponding quotient topology on the set of C-measures. Convergence of -finite measures is different from convergence of the corresponding conditional measures. This is also true if the initial -finite measure is a probability measure.

An alternative is to consider the C-measure space as a maximal Renyi space, and this is a concept more clearly distinct from that of a -finite measure space. Convergence concepts for Renyi spaces can be studied directly, and initial work on this has been done by Renyi (1970a). He shows in particular that any conditional measure can be obtained as a limit of conditional measures corresponding to probability measures in a reasonable topology. The study of convergence concepts exemplify an important difference between -finite measures and C-measures. This is left for the future.

The distinction between a -finite measure space and the corresponding C-measure space can also be seen by analogy with the construction of projective spaces. The projective space as a set is the set of lines through the origin in . It is hence equal to the set of equivalence classes in . The set of C-measures on a measurable space is hence different from the set of -finite measures just as a projective space is different from the space on which it is constructed.

The presented theory is in line with the arguments given by Jeffreys (1939). He argues that improper priors are necessary to get the Bayesian machine up and running. This point of view can be disputed, but it is indisputable that usage of improper priors flourish in the statistical literature. There is hence a need for a theory that includes improper priors.

Lindley (1965), apparently in line with the view of the current authors, found that the theory of Renyi is a natural starting point for statistical theory. In the Preface to his classical text on probability he writes:

The axiomatic structure used here is not the usual one associated with the name of Kolmogorov. Instead one based on the ideas of Renyi has been used. The essential difference between the two approaches is that Renyi’s is stated in terms of conditional probabilities, whereas Kolmogorov’s is in terms of absolute probabilities, and conditional probabilities are defined in terms of them. Our treatment always refers to the probability of A, given B, and not simply to the probability of A. In my experience students benefit from having to think of probability as a function of two arguments, A and B, right from the beginning. The conditioning event, B, is then not easily forgotten and misunderstandings are avoided. These ideas are particularly important in Bayesian inference where one’s views are influenced by the changes in the conditioning event.

Lindley (1965) refers to an earlier German edition of the book cited here (Renyi, 1962). The two books (Renyi, 1970b, a) represent the final view of Renyi regarding conditional probability spaces, but the basis for the theory development are found in earlier articles (Renyi and Turan, 1976; Renyi, 1955). The extension given by conditioning on -finite statistics and -finite -fields is not treated by Renyi.

The structure theorem shows in general that a family of conditional probabilities that satisfies the axioms of a Renyi space given in Definition 6 can be extended so that it corresponds to a unique maximal Renyi space which can be identified with a C-measure space. The family of conditional probabilities gives intuitive motivation and interpretation for usage of improper laws in probability and statistics. In this theory any marginal law corresponds to a conditional probability space. All probabilities are conditional probabilities.

References

  • Berger et al. (2009) James O. Berger, José M. Bernardo, and Dongchu Sun. The Formal Definition of Reference Priors. The Annals of Statistics, 37(2), 2009.
  • Császár (1955) Ákos Császár. Sur la structure des espaces de probabilité conditionnelle. Acta Mathematica Academiae Scientiarum Hungarica, 6:337–361, 1955.
  • Dawid et al. (1973) A. P. Dawid, M. Stone, and J. V. Zidek. Marginalization Paradoxes in Bayesian and Structural Inference. Journal of the Royal Statistical Society Series B-Statistical Methodology, 35(2):189–233, 1973.
  • Halmos and Savage (1949) P. Halmos and L. J. Savage. Application of the Radon-Nikodym theorem to the theory of sufficient statistics. Annals of Mathematical Statistics, 20:225–241, 1949.
  • Hartigan (1983) J. Hartigan. Bayes theory. Springer, 1983.
  • Jeffreys (1939) H. Jeffreys. Theory of probability (1966 ed). Oxford, third edition, 1939.
  • Jöreskog (1970) K. G. Jöreskog. A general method for analysis of covariance structures. Biometrika, 57(2):239–251, August 1970.
  • Kolmogorov (1933) A. Kolmogorov. Foundations of the theory of probability. Chelsea edition (1956), second edition, 1933.
  • Lavine and Hodges (2012) Michael L. Lavine and James S. Hodges. On Rigorous Specification of ICAR Models. The American Statistician, 66(1):42–49, 2012.
  • Lindley (1965) D. V. Lindley. Introduction to Probability and Statistics from a Bayesian Viewpoint(Cambridge 2008 reprint), volume I-II. Cambridge University Press, 1965.
  • Renyi (1962) A. Renyi. Wahrscheinlichkeitsrechnung (Probability theory 1970). Deutscher Verlag der Wissenschaften, Berlin, 1962.
  • Renyi (1970a) A. Renyi. Foundations of Probability. Holden-Day, 1970a.
  • Renyi and Turan (1976) A. Renyi and P. Turan. Selected papers of Alfred Renyi, volume I-III. Akademiai Kiado, Budapest, 1976.
  • Renyi (1955) Alfred Renyi. On a new axiomatic theory of probability. Acta Mathematica Hungarica, 6(3):285–335, September 1955.
  • Renyi (1970b) Alfred Renyi. Probability Theory. Dover Publications, dover ed (2007) edition, May 1970b.
  • Royden (1989) H. L. Royden. Real Analysis. Macmillan, third edition, 1989.
  • Rudin (1987) W. Rudin. Real and Complex Analysis. McGraw-Hill, 1987.
  • Schervish (1995) M. J. Schervish. Theory of Statistics. Springer, 1995.
  • Stone and Dawid (1972) M. Stone and A. P. Dawid. Un-Bayesian Implications of Improper Bayes Inference in Routine Statistical Problems. Biometrika, 59(2):369–375, 1972.
  • Taraldsen and Lindqvist (2010) G. Taraldsen and B. H. Lindqvist. Improper Priors Are Not Improper. The American Statistician, 64(2):154–158, 2010.
  • Taraldsen and Lindqvist (2013a) G. Taraldsen and B. H. Lindqvist. Fiducial theory and optimal inference. Annals of Statistics, 41(1):323–341, 2013a.
  • Taraldsen and Lindqvist (2013b) G. Taraldsen and B. H. Lindqvist. Fiducial and Posterior Sampling. Communications in Statistics:Theory and Methods (accepted), 2013b.
  • Van Trees (2002) H. L. Van Trees.

    Optimum Array Processing (Detection, Estimation, and Modulation Theory, Part IV)

    .
    Wiley-Interscience, 1 edition, March 2002.