# Identifying the consequences of dynamic treatment strategies: A decision-theoretic overview

We consider the problem of learning about and comparing the consequences of dynamic treatment strategies on the basis of observational data. We formulate this within a probabilistic decision-theoretic framework. Our approach is compared with related work by Robins and others: in particular, we show how Robins's 'G-computation' algorithm arises naturally from this decision-theoretic perspective. Careful attention is paid to the mathematical and substantive conditions required to justify the use of this formula. These conditions revolve around a property we term stability, which relates the probabilistic behaviours of observational and interventional regimes. We show how an assumption of 'sequential randomization' (or 'no unmeasured confounders'), or an alternative assumption of 'sequential irrelevance', can be used to infer stability. Probabilistic influence diagrams are used to simplify manipulations, and their power and limitations are discussed. We compare our approach with alternative formulations based on causal DAGs or potential response models. We aim to show that formulating the problem of assessing dynamic treatment strategies as a problem of decision analysis brings clarity, simplicity and generality.

## Authors

• 5 publications
• 10 publications
05/23/2018

### Identification in Nonparametric Models for Dynamic Treatment Effects

This paper develops a nonparametric model that represents how sequences ...
05/09/2012

### Effects of Treatment on the Treated: Identification and Generalization

Many applications of causal analysis call for assessing, retrospectively...
09/20/2021

### Deep Bayesian Estimation for Dynamic Treatment Regimes with a Long Follow-up Time

Causal effect estimation for dynamic treatment regimes (DTRs) contribute...
08/18/2016

### A Bayesian Nonparametric Approach for Estimating Individualized Treatment-Response Curves

We study the problem of estimating the continuous response over time to ...
09/01/2021

### On Estimation and Cross-validation of Dynamic Treatment Regimes with Competing Risks

The optimal moment to start renal replacement therapy in a patient with ...
10/21/2016

### Learning Cost-Effective Treatment Regimes using Markov Decision Processes

Decision makers, such as doctors and judges, make crucial decisions such...
03/05/2021

The parametric g-formula is an approach to estimating causal effects of ...
##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

Many important practical problems involve sequential decisions, each chosen in the light of the information available at the time, including in particular the observed outcomes of earlier decisions. As an example, consider long-term anticoagulation treatment, as often given after events such as stroke, pulmonary embolism or deep vein thrombosis. The aim is to ensure that the patient’s prothrombin time (INR) is within a target range (which may depend on the diagnosis). Patients on this treatment are monitored regularly, and when their INR is outside the target range the dose of anticoagulant is increased or decreased, so that the dose at any given time is a function of the previous INR observations. Despite the availability of limited guidelines for adjusting the dose, the quality of anticoagulation control achieved is often poor (Rosthøj et al., 2006). Another example is the question of when to initiate antiretroviral therapy for an HIV-1-infected patient. The CD4 cell count at which therapy should be started is a central unresolved issue. Preliminary findings indicate that treatment should be initiated when the CD4 cell count drops below a certain level, i.e. treatment should be a function of the patient’s previous CD4 count history (Sterne et al., 2009).

In general, any well-specified way of adjusting the choice of the next decision (treatment or dose to administer) in the light of previous information constitutes a dynamic decision (or treatment) strategy. There will typically be an enormous number of strategies that could be thought of. Researchers would like to be able to evaluate and compare these and, ideally, choose a strategy that is optimal according to a suitable criterion (Murphy, 2003). In many applications, such as the examples given above, it is unlikely that we will have access to large random samples of patients treated under each one of the strategies under consideration. At best, the data available will have been gathered in controlled clinical trials, but often we will have to content ourselves with data from uncontrolled observational studies, with, for example, the treatments being selected by doctors according to informal criteria that we do not know. The key question we address in the present paper is: Under what conditions, and how, could the available data be used to evaluate, compare, and hence choose among, the various decision strategies? When a given strategy can be evaluated from available data it will be termed identifiable.

In principle, our problem can be formulated, represented and solved using the machinery of sequential decision theory, including decision trees and influence diagrams

(Raiffa, 1968; Oliver and Smith, 1990)

— and this is indeed the approach that we shall take in this paper. However, this machinery does not readily provide us with an answer to the question of when data obtained, for example, from an observational study will be sufficiently informative to identify a given strategy. Here, we shall be concerned only with issues around potential biases in the data, rather than their completeness. Thus wherever necessary we suppose that the quantity of data available is sufficient to estimate, to any desired precision, the parameters of the process that actually produced those data. However, that process might still differ from that in the new decision problem at hand. We shall therefore propose simple and empirically meaningful conditions (which can thus be meaningfully criticised) under which it is appropriate and possible to make use of the available parameter estimates, and we shall develop formulae for doing this. These conditions will be termed

stability due to the way they relate observational and interventional regimes. We shall further discuss how one might justify this stability condition by including unobservable variables into the decision theoretic framework, and by using influence diagrams.

Our proposal is closely related to the seminal work of Robins (Robins, 1986, 1987, 1989, 1997). Much of Robins (1986) takes an essentially decision theoretic approach, while also using the framework of structured tree graphs as well as potential responses (and later using causal direct acyclic graphs (DAGs), see Robins (1997)). He shows that under conditions linking hypothetical studies, where the different treatment strategies to be compared are applied, identifiability can be achieved. Robins calls these conditions sequential randomization (and later no unmeasured confounding, see e.g. Robins (1992)). While these are often formalised using potential responses, a closer inspection of Robins (1986) (or especially Robins (1997)) reveals that all that is needed is an equality of conditional distributions under different regimes, which is what our stability conditions state explicitly. Furthermore, Robins (1986) introduces the -computation algorithm as a method to evaluate a sequential strategy, and contrasts it with traditional regression approaches that yield biased results even when stability or sequential randomization holds (Robins, 1992). We shall demonstrate below that, assuming stability, this -computation algorithm arises naturally out of our decision-theoretic analysis, where it can be recognized as a version of the fundamental ‘backward induction’ recursion algorithm of dynamic programming.

### 1.1 Conditional independence

The technical underpinning for our decision-theoretic formulation is the application of the language and calculus of conditional independence (Dawid, 1979, 2002)

to relate observable variables of two types: ‘random’ variables and ‘decision’ (or ‘intervention’) variables. This formalism is used to express relationships that may be assumed between the probabilistic behaviour of random variables under differing regimes (

e.g.

, observational and interventional). Nevertheless, although it does greatly clarify and simplify analysis, this particular language is not indispensable: everything we do could, if so desired, be expressed directly in terms of relationships between probability distributions for observable variables. Thus no essential additional ingredients are being added to the standard formulation of statistical decision theory.

In many cases the conditional independence relations we work with can be represented by means of a graphical display: the influence diagram (ID). Once again, although enormously helpful this is, in a formal sense, only an optional extra. Moreover, although we pay special attention to problems that can be represented by influence diagrams, there are yet others, still falling under our general approach, where this is not possible.

Inessential though these ingredients are, we nevertheless suggest that it is well worth the effort of mastering the basic language and properties, both algebraic and graphical, of conditional independence. In particular, these allow very simple derivations of the logical consequences of assumptions made (Dawid, 1979; Lauritzen et al., 1990).

### 1.2 Overview

In §§ 2 and 3 we set out the basic ingredients of our problem and our notation. Section 4 identifies a simple recursion that can be used to calculate the consequence of applying a given treatment regime when the appropriate probabilistic ingredients are available. In § 5 we consider how these ingredients might be come by, and show that the simple stability condition mentioned above allows estimation of these ingredients — and thus, by application of the procedure of -recursion, of the overall consequence. In §§ 6 and 7 we consider how one might justify this stability condition, starting from a position (‘extended stability’) that might sometimes be more defensible, and relate various sets of sufficient conditions for this to properties of influence diagrams. Section 8 develops more general conditions, similar to Robins (1987) and Robins (1997), under which -recursion can be justified, while § 9 addresses the question of finding an ordering of the involved variables suitable to carry out -recursion. Finally §10 shows how analyses based on the alternative formalism of potential responses can be related mathematically to our own development.

## 2 A multistage decision problem

We are concerned with a sequential data-gathering and decision-making process, progressing through a discrete sequence of stages. The archetypical context is that of a sequence of medical treatments applied to a patient over time, each taking into account any interim responses or adverse reactions to earlier treatments, such as the anticoagulation treatment for stroke patients or the decision of when to start antiretroviral therapy for HIV patients. We shall sometimes use this language.

Associated with each patient are two sets of variables: , the set of observable variables, and , the set of action variables. The variables in can, in principle, be manipulated by external intervention, while those in are generated and revealed by Nature. The variables in are termed domain variables. There is a distinguished variable , the response variable, of special concern.

A specified sequence , where and the are disjoint subsets of , defines the information base. The interpretation is that the variables arise or are observed in that order; represents (possibly multivariate, generally time-dependent) patient characteristics or other variables over which we have no control, observable between times and ; describes the treatment action applied to the patient at time ; and is the final ‘response variable’ of primary interest.

For simplicity we suppose throughout that all these variables exist and can be observed for every patient. Thus we do not directly consider cases where, e.g., is time to death, which might occur before some of the ’s and ’s have had a chance to materialize. However our analyses could readily be elaborated to handle such extensions.

When the aim is to control through appropriate choices for the action variables , any principled approach will involve making comparisons, formal or informal, between the implied distributions of under a variety of possible strategies for choosing the . For example, we might have specified a loss associated with each outcome of , and desire to minimise its expectation .111 Realistically the loss could also depend on the values of intermediate variables, e.g. if these relate to adverse drug reactions. Such problems can be treated by redefining as the overall loss suffered (at any rate so long as this loss does not depend on other, unobserved, variables.) Any such decision problem can be solved as soon as we know the relevant distributions for (Dawid, 2000, Section 6).

The simplest kind of strategy is to apply some fixed pre-defined sequence of actions, irrespective of any observations on the patient: we call this a static or unconditional strategy (Pearl (2009) terms it atomic). However in realistic contexts static strategies, which do not take any account of accruing information, will be of little interest. In particular, under a decision-theoretically optimal strategy the action to be taken at any stage must typically be chosen to respond appropriately to the data available at that stage (Robins, 1989; Murphy, 2003).

A non-randomized dynamic treatment strategy (with respect to a given information base ) is a rule that determines, for each stage and each configuration (or partial history) for the variables available prior to that stage, the value of that is then to be applied.

Any decision-theoretically optimal strategy can always be chosen to be non-randomized. Nevertheless, for added generality we shall also consider randomized222More correctly, these correspond to what are termed behavioral rules in decision theory (Ferguson, 1967) dynamic treatment strategies. Such a strategy determines, for each stage and associated partial history , a probability distribution for , describing the random way in which the next action is to be generated. When every such randomization distribution is degenerate at a single action this reduces to a non-randomized strategy.

Suppose now we wish to compare a number of such strategies. If we knew or could estimate the full probabilistic structure of all the variables under each of these, we could simply calculate and compare directly the various distributions for the response . As outlined in the introduction, our principal concern in this paper is how to obtain such distributional knowledge, when in many cases the only data available will have been gathered under purely observational or other circumstances that might be very different from the strategies we want to compare. To clarify the potential difficulties, consider a statistician or scientist S, who has obtained data on a collection of variables for a large number of patients. She wishes to use her data, if possible, to identify and compare the consequences of various treatment interventions or policies that might be contemplated for some new patient. A major complication, and the motivation for much work in this area, is that S’s observational data will often be subject to ‘confounding’. For example, S’s observations may include actions that have been determined by a doctor D, partly on the basis of additional private information D has about the patient, over and above the variables S has measured. Then knowledge of the fact that D has selected an act , by virtue of that being correlated with unobserved private information D has that may also be predictive of the response , could affect the distribution of in this observational regime in a way different from what would occur if D had no such private information, or if S had herself chosen the value of . In particular, without giving careful thought to the matter we cannot simply assume that probabilistic behaviour seen under the observational regime will be directly relevant to other, e.g. interventional, regimes of interest.

## 3 Regimes and consequences

In general, we consider the distribution of all the variables in the problem under a variety of different regimes, possibly but not necessarily involving external intervention. For example, these might describe different locations, time-periods, or contexts in which observations can be made. For simplicity we suppose that the domain variables are the same for all regimes. Formally, we introduce a regime indicator, , taking values in some set

, which specifies which regime is under consideration — and thus which (known or unknown) joint distribution over the domain variables

is operating. Thus has the logical status of a parameter or decision variable, rather than a random variable. We think of the value of as being determined externally, before any observations are made; all probability statements about the domain variables must then be explicitly or implicitly conditional on the value of . We use e.g. to denote the conditional density for , at , given , under regime . In order to side-step measure-theoretic subtleties, we shall confine attention to the case that all variables considered are discrete; in particular, the terms ‘distribution’ or ‘density’ should be interpreted as denoting a probability mass function. However, the basic logic of our arguments does extend to more general cases (albeit with some non-trivial technical complications to handle null events.)

If we know for all , we can determine, for any function , the expectation . Often we shall be interested in one or a small number of such functions, e.g.

. For definiteness we henceforth consider a fixed given function , and use the term consequence of to denote the expectation of when regime is followed.

More generally we might wish to focus attention on a subgroup (typically defined in terms of the pre-treatment information ), and compare the various ‘conditional consequences’, given membership of the subgroup. Although we do not address this directly here, it is straightforward to extend our unconditional analysis to this case.

### 3.1 Inference across regimes

In the most usual and useful situation, , where is a particular observational regime under which data have been gathered, and is a collection of contemplated interventional strategies with respect to the information base . We wish to use data collected under the observational regime to identify the consequence of following any of the strategies . This means we need to make inference strictly beyond the available data to what would happen, in future cases, under regimes that we have not been able to observe in the past.

It should be obvious, but nonetheless deserves emphasis, that we can not begin to address this problem without assuming some relationships between the probabilistic behaviour of the variables across the differing regimes, both observed and unobserved. Inferences across regimes will typically be highly sensitive to the assumptions made, and the validity of our conclusions will depend on their reasonableness. Although in principle any such assumptions are open to empirical test, using data gathered under all the regimes involved, this will often be impossible in practice. In this case, while it is easy to make assumptions, it can be much harder to justify them. Any justification must involve context-dependent considerations, which we can not begin to address here. Instead we simply aim to understand the logical consequences of making certain assumptions. One message that could be drawn is: if you don’t like the consequences, rethink your assumptions.

## 4 Evaluation of consequences

Writing e.g.  for , we denote by , with similar conventions for other variables in the problem.

For any fixed regime , we can specify the joint distribution of , when , in terms of its sequential conditional distributions for each variable, given all earlier variables. These comprise:

1. for .

2. for .

3. .

Note that (iii) can also be considered as the special case of (i) for .

With , we can factorize the overall joint density as:

 p(y,¯l,¯¯¯a;s)={N+1∏i=1p(li∣¯li−1,¯¯¯ai−1;s)}×{N∏i=1p(ai∣¯li,¯¯¯ai−1;s)}. (1)

If we know all the terms in (1), we can simply sum out over all variables but to obtain the desired distribution of under regime , from which we can in turn compute the consequence .

Alternatively, and more efficiently, this calculation can be implemented recursively, as follows. Let denote a partial history, of the form or (. We also include the ‘null’ history , and ‘full’ histories . We denote the set of all partial histories by . Fixing the regime , define a function on by:

 f(h):=E{k(Y)∣h;s}. (2)

Simple application of the laws of probability yields:

 f(¯li,¯¯¯ai−1) = ∑aip(ai∣¯li,¯¯¯ai−1;s)×f(¯li,¯¯¯ai) (3) f(¯li−1,¯¯¯ai−1) = ∑lip(li∣¯li−1,¯¯¯ai−1;s)×f(¯li,¯¯¯ai−1). (4)

For a full history , we have . Using these as starting values, by successively implementing (3) and (4) in turn, starting with (4) for and ending with (4) for , we step down through ever shorter histories until we have computed , the consequence of regime .333More generally (see footnote 1), we could consider a function of . Starting now with , we can apply the identical steps to arrive at . In particular we can evaluate the expected overall loss under , even when the loss function depends on the full sequence of variables.

The recursion expressed by (3) and (4) is exactly that underlying the ‘extensive form’ analysis of sequential decision theory (see e.g. Raiffa (1968)). In particular, under suitable further conditions we can combine this recursive method for evaluation of consequences with the selection of an optimal strategy, when it becomes dynamic programming. This ‘step-down histories’ approach also applies just as readily to more general probability or decision trees, where the length of the history, and even the variables entering into it, can vary with the path followed. We do not consider such extensions here, but they raise no new issues of principle.

When is a non-randomized strategy, the distribution of given , when , is degenerate, at , say, and the only randomness left is for the variables . We can now consider as a function of only the appearing in , since, under , these then determine the . Then (3) holds automatically, while (4) becomes:

 f(¯li−1)=∑lip(li∣¯li−1,¯¯¯gi−1;s)×f(¯li). (5)

When, further, the regime is static, each in the above expressions reduces to the fixed action specified by .

We remark that the conditional distributions in (i)(iii) and (2) are undefined when the conditioning event has probability 0 under . The overall results of recursive application of (3) and (4) will not depend on how such ambiguities are resolved. However, for later convenience we henceforth assume that in (2) is defined as 0 whenever . Note that this property is preserved under (3) and (4).

## 5 Identifying the ingredients

In order for the statistician S to be able to apply the above recursive method to calculate the consequence of some contemplated regime , she needs to know all the ingredients (i), (ii) and (iii). How might such knowledge be attained?

### 5.1 Control strategies

Consider first the term in (ii), as needed for (3). It will often be the case that for the regimes of interest this is known a priori to the statistician S for all . For instance we might be interested in strategies for initiating antiretroviral treatment of HIV patients as soon as the CD4 count has dropped below a given value . The strategy therefore fully determines the value of the binary given the previous covariate history as long as this includes information on the CD4 counts. In such a case we shall call a control strategy (with respect to the information base ). In particular this will typically be the case when is a (possibly randomized) dynamic strategy, as introduced in §2.

### 5.2 Stability

More problematic is the source of knowledge of the conditional density in (i) as required for (4) (including, as a special case, that of in (iii)).

If we observed many instances of regime , we may be able to estimate this directly; but typically we will be interested in assessing the consequences of various contemplated regimes (e.g. control strategies) that we have never yet observed. The problem then becomes: under what conditions can we use probability distributions assessed under one regime to deduce the required conditional probabilities, (i) and (iii), under another?

In the application of most interest, we have , where is an observational regime under which data have been gathered, and is a collection of contemplated interventional strategies. If we can use data collected under the observational regime to identify the consequence of following any of the strategies , we will be in a position to compare the consequences of different interventional strategies (and thus, if desired, choose an optimal one) on the basis of data collected in the single regime .

In general, the distribution of given will depend on which regime is in operation. Even application of a control strategy might well have effects on the joint distribution of all the variables, beyond the behaviour it directly specifies for the actions. For example, consider an educational experiment in which we can select certain pupils to undergo additional home tutoring. Such an intervention can not be imposed without subjecting the pupil and his family to additional procedures and expectations, which would probably be different if the decision to undergo extra tutoring had come directly from the pupil, and possibly different again if it had come from the parents. Consequently we can not necessarily assume that the distribution of given assessed under the observational regime will be the same as that for an interventional strategy, or that it would be the same for different interventional strategies.

It will clearly be helpful when we can impose this assumption — and so be able to identify the required interventional distributions of given with those assessed under the observational regime. We formalize this assumption as follows:

###### Definition 5.1

We say that the problem exhibits simple stability, with respect to the information base and the set of regimes if, with denoting the non-random regime indicator taking values in :

 Li⊥⊥σ∣(¯¯¯¯Li−1,¯¯¯¯Ai−1)(i=1,…,N+1). (6)

Here and throughout, we use the notation and theory of conditional independence introduced by Dawid (1979), as generalized as in Dawid (2002) to apply also to problems involving decision or parameter variables. In words, condition (6) asserts that the stochastic way in which arises, given the previous values of the ’s and ’s, should be the same, irrespective of which regime in is in operation. More precisely, expressed in terms of densities, (6) requires that, for each , there exist some common conditional density specification such that, for each ,

 p(Li=li∣¯¯¯¯Li−1=¯li−1,¯¯¯¯Ai−1=¯¯¯ai−1;s)=q(Li=li∣¯¯¯¯Li−1=¯li−1,¯¯¯¯Ai−1=¯¯¯ai−1) (7)

whenever the conditioning event has positive probability under regime .

As will be described further in §7 below, it is often helpful (though never essential) to represent conditional independence properties graphically, using the formalism of influence diagrams (IDs): such diagrams have very specific semantics, and can facilitate logical arguments by displaying implied properties in a particularly transparent form (Dawid, 2002). The appropriate graphical encoding of property (6) for 1, 2 and 3 is shown in Figure 1. The specific property (6) is represented by the absence of arrows from to , , and . For general we simply supplement the complete directed graph on with an additional regime node , and an arrow from to each .

An important question is how we should assess whether property (6) holds in any given situation. It could in principle be tested empirically, if we could collect data under all regimes. In practice this is usually impossible, and other arguments for or against its appropriateness would be brought to bear. Whether or not the simple stability property can be regarded as appropriate in any application will depend on the overall context of the problem. In particular, it will depend on the specific information base involved. For example, if is a control strategy with respect to S’s information base, and an observational regime under which the doctor D chooses the on the basis of private information not represented in S’s information base, possibly associated with , then, for , we might well expect (6) to be violated. This is often described as (potential) confounding.

The simple stability property (6) is our version of a condition termed ‘sequential randomization’ (Robins, 1986, 1997) or ‘no unmeasured confounding’ (Robins, 1992; Robins, Hernán and Brumback, 2000) or ‘sequential ignorability’ (Robins, 2000). The connexions become particularly clear when comparing (6) with the equalities derived in Theorem 3.1 of Robins (1997), which we consider in more detail in §10.1.1 below. These alternative names suggest particular situations where stability should be satisfied, such as when the data have been gathered under an observational regime where the actions were indeed physically sequentially randomized; or when S’s information base contains all the information the doctor D has used in choosing the . However, we emphasise that our property (6) can be meaningfully considered even without referring to any ‘potential confounder’ variables; and that if (as in §6 below) we do choose to introduce such further variables to help us assess whether (6) holds, nevertheless the property itself must hold or fail quite independently of which additional variables (if any) are considered.

In any case, because stability is a property of the relationship between different regimes, it can never be empirically established on the basis of data collected under only one (e.g., observational) regime, nor can it be deduced from properties assumed to hold for just one such regime.

#### 5.2.2 Positivity

The purpose of invoking simple stability (with respect to ) is to get a handle on (4) for an unobserved interventional strategy , using data obtained in the observational regime . Intuitively, under simple stability we can replace by , which is estimable from the observational data. However, some care is needed on account of the positivity qualification following (7). If, for example, we want to assess the consequence of a static interventional strategy , which always applies some pre-specified action sequence , we clearly will be unable to do so using data from an observational regime in which the probability of obtaining that particular sequence of actions is zero. (Pragmatically it may still be difficult to do so if that probability is non-zero but so small that we are unable to estimate it well from available observational data. However we ignore that difficulty here, supposing that the data are sufficiently extensive that we can indeed get good estimates of all probabilities under ).

In order to avoid this problem, we impose the positivity (absolute continuity) condition:

###### Definition 5.2

We say the problem exhibits positivity if, for any , the joint distribution of under is absolutely continuous with respect to that under , i.e.

 p(E;e)>0⇒p(E;o)>0 (8)

for any event defined in terms of . We write this as .

In our discrete set-up, it is clearly enough to demand (8) whenever comprises a single sequence . Denoting by , the sets of partial histories having positive probability under, respectively, regimes and , we can restate (8) as

 E⊆O. (9)

### 5.3 G-recursion

Let . Given enough data collected under we can identify () for . Under simple stability (7) and positivity (9), this will also give us () for all . If, further, is a control strategy, then using the known form for (), we have all the ingredients to apply (3) and (4) and thus identify the consequence of regime from data collected under .

Specifically, we have

 f(¯li,¯¯¯ai−1) = ∑aip(ai∣¯li,¯¯¯ai−1;e)×f(¯li,¯¯¯ai) (10) f(¯li−1,¯¯¯ai−1) = ∑lip(li∣¯li−1,¯¯¯ai−1;o)×f(¯li,¯¯¯ai−1). (11)

We start the recursion with

 f(¯lN,¯¯¯aN)≡E{k(Y)∣¯lN,¯¯¯aN;e}=⎧⎨⎩E{k(Y)∣¯lN,¯¯¯aN;o}if (¯lN,¯¯¯aN)∈E0otherwise

(using simple stability for ), and exit with the desired interventional consequence .

We refer to the above method as -recursion.444Cases in which simple stability may not hold but we can nevertheless still apply -recursion are considered in Section 8.

For the case that is a non-randomized strategy, -recursion can be based on (5), becoming

 f(¯li−1)=∑lip(li∣¯li−1,¯¯¯gi−1;o)×f(¯li), (12)

starting with . The -computation formula (Robins, 1986) is the algebraic formula for in terms of that results when we write out explicitly the successive substitutions required to perform this recursion.

Finally we remark that, when the simple stability property (6) holds for , it also holds for , where is any function of . For there is nothing new to show, while (6) for follows easily for when it holds for , using general properties of conditional independence (Dawid, 1979). It is also easy to see that when positivity, Definition 5.2, holds for it likewise holds for . Consequently, under the same conditions that allow -recursion to compute the interventional distribution of , we can use it to compute that of . In particular (see footnote 1), this will allow us to evaluate the expected loss of applying , even when the loss function depends on all of .

## 6 Extended stability

We have already alluded to the possibility that, in many applications, the simple stability assumption (6) might not be easy to justify directly. This might be the case, in particular, when we are concerned about the possibility of ‘confounding effects’ due to unobserved influential variables.

In such a case we might proceed by constructing a more detailed model, incorporating a collection of additional, possibly unobserved, variables; and investigate its implications. These unobserved variables might be termed ‘sequential (potential) confounders’. Under certain additional assumptions to be discussed below, we might then be able to deduce that simple stability does, after all, apply. This programme can be helpful when the assumptions involving the additional variables are easier to justify than assumptions referring only to the variables of direct interest. We here initially express these additional assumptions purely algebraically, in terms of conditional independence; in §7 we shall conduct a parallel analysis utilising influence diagrams to facilitate the expression and manipulation of the relevant conditional independencies.

Reasoning superficially similar to ours has been conducted by Pearl and Robins (1995) and Robins (1997). However, that is mostly based on the assumed existence of a ‘causal DAG’ representation of the problem. We once again emphasise that the simple stability property (6) is always meaningful of itself, and its truth or falsity can not rely on the possibility of carrying out such a programme of reduction from a more complex model including unobservable variables.

### 6.1 Preliminaries

We shall specifically investigate models having a property we term extended stability. Such a model again involves a collection of observable domain variables (including a response variable ) and a collection of action domain variables, together with a regime indicator variable taking values in . But now we also have the collection of unobservable domain variables (for simplicity we suppose throughout that which variables are observed or unobserved is the same under all regimes considered). Let denote an ordering of all these observable and unobservable domain variables (typically, though not necessarily, their time-ordering). As before we assume that comes before in this ordering. We term an extended information base. Let [resp., ] denote the set of observed [resp., unobserved] variables between and .

###### Definition 6.1

We say that the problem exhibits extended stability with respect to the extended information base and the set of regimes if, for ,

 (Ui,Li)⊥⊥σ∣(¯¯¯¯Ui−1,¯¯¯¯Li−1,¯¯¯¯Ai−1). (13)

(If the () were observable, this would be identical with the definition of simple stability.)

Under extended stability the marginal distribution of is supposed the same in both regimes, as is the conditional distribution of given , etc. Similarly, the distributions of given , of given ,…, and finally of () given , are all supposed to be independent of the regime operating.

There is a corresponding extension of Definition 5.2:

###### Definition 6.2

We say the problem exhibits extended positivity if, for any , as distributions over ; that is, and any event defined in terms of .

In many problems, though by no means universally, an extended stability assumption might be regarded as more reasonable and defensible than simple stability — so long as appropriate unobserved variables are taken into account. For example, this might be the case if we believed that, in the observational regime, the actions were chosen by a decision-maker who had been able to observe, in sequence, some or all of the variables in the problem, including possibly the ’s; and was then operating a control strategy with respect to this extended information base, so that, when choosing each action, he was taking account of all previous variables in this extended sequence, but nothing else. But even then, as discussed in §5.2, the extended stability property is a strong additional assumption, that needs to be justified in any particular problem. And again, because it involves the relationships between distributions under different regimes, it can not be justified on the basis of considerations or findings that apply only to one regime.

Unobservable variables can assist in modelling the observational regime and its relationship with the interventional control regimes under consideration. But, because they are unobserved, they can not form part of the information taken into account by such control regimes. Thus we shall still be concerned with evaluating — using -recursion when possible — a regime that is a control strategy with respect to the observable information base as introduced in §5.1. More specifically, in this more general context we define:

###### Condition 6.1 (Control strategy)

The regime is a control strategy if, for ,

 Ai⊥⊥¯¯¯¯Ui∣(¯¯¯¯Li,¯¯¯¯Ai−1;e) (14)

and, in addition, the conditional distribution of , given , under regime , is known to the analyst.

Condition 6.1 expresses the property that, under regime , the randomization distribution or other sources of uncertainty about , given all earlier variables, does not in fact depend on the earlier unobserved variables; and that this conditional distribution is known. The condition will hold, in particular, in the important common case that, under , is fully specified as a function of previous observables.

### 6.2 Stability regained

When there are unobservables in the problem, the extended positivity property of Definition 6.2 will clearly imply the simple positivity property of Definition 5.2. However, even when extended stability holds, the simple stability property, with respect to the observable information base from which (as is a pragmatic necessity) we have had to exclude the unobserved variables, will typically fail. But we can sometimes incorporate additional background knowledge, most usefully expressed in terms of conditional independence, to show that it does, after all, hold.

We now describe two sets of additional sufficient (though not necessary) conditions, either of which will, when appropriate, allow us to deduce the simple stability property (6) — and with it, the possibility of applying -recursion (ignoring the unobservable variables), as set out in §5.3. The results in this section can be regarded as extending the analysis of Dawid (2002) § 8.3 (see also Guo and Dawid (2010)) to the sequential setting.

#### 6.2.1 Sequential randomization

It has frequently been proposed (e.g., Robins (1986, 1997)) that when, under an observational regime, the actions have been physically (sequentially) randomized, then simple stability (6) will hold. Indeed, our concept of simple stability has also been termed ‘sequential randomization’ (Robins, 1986). However we shall be more specific and restrict the term sequential randomization to the special case that we have extended stability and, in addition, Condition 6.2 below holds. We shall show that these properties are indeed sufficient to imply simple stability — but they are by no means necessary.

So consider now the following condition:

###### Condition 6.2
 Ai⊥⊥¯¯¯¯Ui∣(¯¯¯¯Li,¯¯¯¯Ai−1;σ)(i=1,…,N). (15)

This is essentially a discrete-time version of Definition 2 (ii) of Arjas and Parner (2004), but with the additional vital requirement that the unobservable variables involved already be such as to allow us to assume the extended stability property (13). (Without such an underlying assumption there can be no way of relating different regimes together.)

Condition 6.2 requires that, for each regime, any earlier unobserved variables in the extended information base can have no further effect on the distribution of , once the earlier observed variables are taken into account. This will certainly be the case when, under each regime, treatment assignment, at any stage, is determined by some deterministic or randomizing device that only has the values of those earlier observed variables as inputs. While this will necessarily hold for a control strategy with respect to the observed information base, whether or not it is a reasonable requirement for the observational regime will depend on deeper consideration of the specific context and circumstances. It will typically do so if all information available to and utilised by the decision-maker (the doctor, for instance) in the observational regime is included in , or, indeed, if the actions have been physically randomized within levels of .

###### Theorem 6.1

Suppose our model exhibits extended stability. If in addition Condition 6.2 holds, then we shall also have the simple stability property (6).

Proof. Our proof will be based on universal general properties of conditional independence, as described by Dawid (1979, 1998).

Let , , denote, respectively, the following assertions:

 Ei : (Li,Ui)⊥⊥σ∣(¯¯¯¯Li−1,¯¯¯¯Ui−1,¯¯¯¯Ai−1) Ri : Ai⊥⊥¯¯¯¯Ui∣(¯¯¯¯Li,¯¯¯¯Ai−1;σ) Hi : (Li,¯¯¯¯Ui)⊥⊥σ∣(¯¯¯¯Li−1,¯¯¯¯Ai−1)

Extended stability is equivalent to holding for all , so we assume that; while is just Condition 6.2, which we are likewise assuming for all . We shall show that these assumptions imply for all , which in turn implies , i.e., simple stability.

We proceed by induction. Since and are both equivalent to , holds.

Suppose now holds. Conditioning on yields

 ¯¯¯¯Ui⊥⊥σ∣(¯¯¯¯Li,¯¯¯¯Ai−1), (16)

and this together with is equivalent to , which on conditioning on then yields

 ¯¯¯¯Ui⊥⊥σ∣(¯¯¯¯Li,¯¯¯¯Ai). (17)

Also, by we have

 (Li+1,Ui+1)⊥⊥σ∣(¯¯¯¯Li,¯¯¯¯Ui,¯¯¯¯Ai). (18)

Taken together, (17) and (18) are equivalent to , so the induction is established.

#### 6.2.2 Sequential irrelevance

Another possible condition is:

###### Condition 6.3
 Li⊥⊥¯¯¯¯Ui−1∣(¯¯¯¯Li−1,¯¯¯¯Ai−1;σ)(i=1,…,N+1). (19)

In contrast to (15), (19) does permit the unobserved variables to date, , to influence the next action (which can however only happen in the observational regime), as well as the current observable ; but they do not affect the subsequent development of the ’s (including, in particular, the response variable ).

###### Theorem 6.2

Suppose:

1. Extended stability, (13), holds.

2. Sequential irrelevance, Condition 6.3, holds for the observational regime :

 Li⊥⊥¯¯¯¯Ui−1∣(¯¯¯¯Li−1,¯¯¯¯Ai−1;σ=o)(i=1,…,N+1). (20)
3. Extended positivity, as in Definition 6.2, holds.

Then we shall have simple stability:

 Li⊥⊥σ∣(¯¯¯¯Li−1,¯¯¯¯Ai−1)(i=1,…,N+1). (21)

Moreover, sequential irrelevance holds under any regime:

 Li⊥⊥¯¯¯¯Ui−1∣(¯¯¯¯Li−1,¯¯¯¯Ai−1;σ)(i=1,…,N+1). (22)

Proof. Let be a bounded real function of , and, for each regime , let be a version of .

By (20) there exists such that

 h(¯¯¯¯Ui−1,¯¯¯¯Li−1,¯¯¯¯Ai−1;o)=f(¯¯¯¯Li−1,¯¯¯¯Ai−1)a.s. [Po] (23)

whence, from (8), for all ,

 h(¯¯¯¯Ui−1,¯¯¯¯Li−1,¯¯¯¯Ai−1;o)=f(¯¯¯¯Li−1,¯¯¯¯Ai−1)a.s. [Ps]. (24)

Also, from (13),

 Li⊥⊥σ∣¯¯¯¯Ui−1,¯¯¯¯Li−1,¯¯¯¯Ai−1 (25)

and so there exists such that, for all ,

 h(¯¯¯¯Ui−1,¯¯¯¯Li−1,¯¯¯¯Ai−1;s)=g(¯¯¯¯Ui−1,¯¯¯¯Li−1,¯¯¯¯Ai−1)a.s. [Ps]. (26)

In particular,

 h(¯¯¯¯Ui−1,¯¯¯¯Li−1,¯¯¯¯Ai−1;o)=g(¯¯¯¯Ui−1,¯¯¯¯Li−1,¯¯¯¯Ai−1)a.s. [Po], (27)

so that, again using (8),

 h(¯¯¯¯Ui−1,¯¯¯¯Li−1,¯¯¯¯Ai−1;o)=g(¯¯¯¯Ui−1,¯¯¯¯Li−1,¯¯¯¯Ai−1)a.s. [Ps]. (28)

Combining (24), (26) and (28), we obtain

 h(¯¯¯¯Ui−1,¯¯¯¯Li−1,¯¯¯¯Ai−1;s)=f(¯¯¯¯Li−1,¯¯¯¯Ai−1)a.s. [Ps]. (29)

Since this property holds for all and every bounded real function , we deduce

 Li⊥⊥(¯¯¯¯Ui−1,σ)∣(¯¯¯¯Li−1,¯¯¯¯Ai−1) (30)

from which both (21) and (22) follow.

It is worth noting that we do not need the full force of extended stability for the above proof, but only (25). In particular, we could allow arbitrary dependence of on any earlier variables, including . We note further that the above proof makes essential use of the extended positivity property of Definition 6.2: (21) can not be deduced from extended stability and Condition 6.3 making use of the standard conditional independence axioms (Dawid, 1998; Pearl and Paz, 1987; Dawid, 2001) alone.

Although we can certainly deduce simple stability when we can assume the conditions of either Theorem 6.1 or Theorem 6.2, it can also arise our of extended stability in other ways. For example, this can be so when Condition 6.2 holds for some subsets of , while Condition 6.3 holds for some subsets of . Such cases are addressed by Corollaries 4.1 and 4.2 of Robins (1997); we give examples in §7.2.3 below.

## 7 Influence diagrams

As previously mentioned, it is often helpful (though never essential) to represent and manipulate conditional independence properties graphically, using the formalism of influence diagrams (IDs). In particular, when including unobserved variables and assuming extended stability, we can often deduce directly from graph-theoretic separation properties whether simple stability holds.

### 7.1 Semantics

Here we very briefly describe the semantics of IDs, and show how they can facilitate logical arguments by displaying implied properties in a particularly transparent form. We shall use the theory and notation of Cowell et al. (1999) and Dawid (2002) in relation to directed acyclic graphs (DAGs) and IDs, and their application to probability and decision models. The reader is referred to these sources for more details.

For any DAG or ID , its moral graph, or moralization, is the undirected graph in which first an edge is inserted between any unlinked parents of a common child in , and then all directions are ignored. For any set of nodes of we denote the smallest ancestral subgraph of containing by , and its moralization by (we may omit the specification of when this is clear). For sets of nodes of we write , and say separates from (with respect to ) to mean that, in , every path joining to intersects . Let and denote the non-descendants and parents of a random node , then it can be shown (Lauritzen et al., 1990; Dawid, 2002) that, whenever a probability distribution or decision problem is represented by , in the sense that for any such the probabilistic conditional independence holds, we have

 A⊥⊥\scriptsizeDB∣C⇒A⊥⊥B∣C. (31)

This moralization criterion thus allows us to infer probabilistic independence properties from purely graph-theoretic separation properties.555An alternative, and entirely equivalent, approach can be based on the ‘-separation criterion’ (Verma and Pearl, 1990; Pearl, 2009). We have found (31) more straightforward to understand and apply.

While the above allows us to read off conditional independencies from a DAG, we can, conversely, construct an ID from a given collection of joint distributions over the domain variables (one for each regime) in the following way.

The node-set is given by . The graph has random (round) nodes for all the domain variables, and a founder decision (square) node for . The ordering given by the extended information base induces an ordering on such that any nodes in the (possibly empty) sets , come after and before , and is last. In addition we require the node to be prior to any domain variables in this ordering. With each node is associated its collection of conditional distributions, given values for all its predecessors, , in the ordering (including, in particular, specification of the relevant regime).

For each such we will have a conditional independence (CI) property of the form:

 C(ν):ν⊥⊥pre(ν)∣pa(ν)

where is some given subset of . Thus asserts that the distributions of , given all its predecessors, in fact only depends on the values of those in . Note that property will be vacuous, and can be omitted, when . Such a collection, say, of CI properties is termed recursive. We represent graphically by drawing an arrow into each node from each member of its parent set , and we associate with the ‘parent-child’ conditional probabilities of the form . The ID constructed in this way will ensure that the joint distribution of the domain variables, in each regime, satisfies any conditional independencies obtained by applying the moralization criterion (31).

From this point on, when we use the terms ‘parents’, ‘ancestors’ etc., the regime node will be excluded from these sets. Also, while in general the terms , could each refer to a collection of variables, for simplicity we shall consider only the case in which they represent just one (or sometimes none), and so can be modelled (if present at all) by a single node in the graph.

We emphasise that IDs are related to but distinct from ‘causal DAGs’ (Spirtes, Glymour and Scheines, 2000; Pearl, 1995). For a discussion see Dawid (2010) and Didelez, Kreiner and Keiding (2010).

### 7.2 Extended stability

The extended stability property (13) embodies a recursive collection of CI properties with respect to the ordering induced by the extended information base. Consequently it can be faithfully expressed by an ID satisfying:

###### Condition 7.1

The only arrows out of in are into .

For this is depicted in Figure 2. Note that the subgraph corresponding to the domain variables is complete.

#### 7.2.1 Sequential randomization

With the ordering induced by the extended information base , (13) and (15) together form a recursive collection of CI properties. Therefore the conditions of Theorem 6.1 can be faithfully represented graphically in an ID