Optimal block designs for small experiments on networks

by   Vasiliki Koutra, et al.
King's College London

We propose a method for constructing optimal block designs for experiments on networks. The response model for a given network interference structure extends the linear network effects model of Parker et al. (2016) to incorporate blocks. The optimality criteria are chosen to reflect the experimental objectives and an exchange algorithm is used to search across the design space for obtaining an efficient design when an exhaustive search is not possible. Our interest lies in estimating the direct comparisons among treatments, in the presence of nuisance spillover effects that stem from the underlying interference structure governing the experimental units, or in the spillover effects themselves. Comparisons of optimal designs under different models, including the standard treatment models, are examined by comparing the variance and bias of treatment effect estimators. We also suggest a way of defining blocks, while taking into account the interrelations of groups of experimental units within a network, using spectral clustering techniques to achieve optimal modularity. We expect connected units within closed-form communities to behave similarly to an external stimulus. We provide evidence that our approach can lead to efficiency gains over conventional designs such as randomized designs while ignoring the network structure and we illustrate its usefulness for experiments on networks.



There are no comments yet.


page 1

page 2

page 3

page 4


Designs with complex blocking structures and network effects for use in agricultural field experiments

We propose a novel model-based approach for constructing optimal designs...

Copula-based robust optimal block designs

Blocking is often used to reduce known variability in designed experimen...

A graph-theoretic framework for algorithmic design of experiments

In this paper, we demonstrate that considering experiments in a graph-th...

D-optimal Design for Network A/B Testing

A/B testing refers to the statistical procedure of conducting an experim...

Design of Order-of-Addition Experiments

In an order-of-addition experiment, each treatment is a permutation of m...

Teaching Design of Experiments using Hasse diagrams

Hasse diagrams provide a principled means for visualizing the structure ...

Adaptive Experimental Design with Temporal Interference: A Maximum Likelihood Approach

Suppose an online platform wants to compare a treatment and control poli...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Designing experiments on networks is a growing area of research mainly due to the rise and popularity of online social networks and viral marketing. The work presented here is motivated by the need to develop a practical methodology for obtaining efficient designs which control for variation among the experimental units from two sources: blocks and network interference, so that the true effects of the treatments can be detected.

Consider a commercial experiment on a social networking site to compare the effectiveness of different advertisements concerning a product. The network members are connected via virtual friendships. The responses can be the quantities purchased during the week immediately following the advertising campaign. The company’s goal is to maximize the appeal of the product by using the advertisement as a tool to affect the purchasing decisions of potential customers and the experiment will be used as a means for comparing different advertisements. The advertisements used might have an effect not only on the recipient but also on their (virtual) friends. Moreover, there may be blocking structures related to different age groups of customers or cliques of close friends who engage in similar behaviors, e.g. making similar decisions as to what quantity of product to purchase. We might expect subjects in the same block to have similar responses and subjects in different blocks to have dissimilar responses, irrespective of the presence or absence of viral effects of specific advertisements. By allowing for this in the design, we ensure more precise comparisons of the effects of advertisements.

A noteworthy work which accommodated interference is Besag and Kempton (1986), which discussed spatial techniques and provided appropriate models to adjust for response or treatment interference in the context of agricultural field experiments. With our interest in the case of treatment interference, we focus on the model of Pearce (1957) presented by Besag and Kempton (1986), which jointly takes into account ‘local’ (i.e. direct) and ‘remote’ (i.e. indirect or neighbor) effects in the model. A wide variety of candidate models has been suggested for the case of treatment interference thereafter. Important examples include the work of Druilhet (1999), Kunert and Martin (2000) and Kunert and Mersmann (2011), who provided models and efficient designs that concern experiments where units are arranged in a circle or a line and in which neighbors one unit apart can interfere with each other. More recently, Parker et al. (2016) adopted the conceptual approach of Pearce (1957) and introduced a model, called the linear network effects model (LNM). The LNM is similar to the model of Kunert and Martin (2000). However, it differs by relaxing the assumption of neighbor effects existing in only one direction and allows for a network setting in which units can be linked in a way that does not form a regular layout. Here, we extend the LNM of Parker et al. (2016) to include the notion of blocks.

After we introduce the network block model (NBM) (Section 2), we define two specific design criteria for estimating with minimum variance the direct treatment comparisons in the presence of nuisance network effects or the network effects themselves (Section 3). We describe a systematic exchange algorithm used to obtain our designs (Section 4). We then make some comparisons among optimal designs under different models and investigate issues associated with the design efficiency and bias arising in the analysis of dependent data (Section 5). When blocks arise as cliques in the network, we suggest a method to define the blocks using spectral clustering and the concept of modularity. We give an example illustrating the recommended step-by-step methodology for the special case of two unstructured treatments (Section 6). Finally, we discuss some practical concerns emerging from the suggested methodology and further issues of interest (Section 7).

2 Statistical model

Suppose that experimental units are available for experimentation and that they form a network which is represented by means of a graph , with vertex set (of size ) and edge set (of size ). The vertices might represent individuals connected through edges which might indicate some form of connection such as friendship, collaboration, or communication. The adjacency matrix of a graph is an matrix with , which is a compact way to represent the collection of edges. We focus on connected, undirected and unweighted graphs with representing the absence or presence of an edge between the vertices and (mutual connections). Thus is symmetric and binary. The diagonal elements represent the self-links, which by convention are set to zero. In an undirected network, the degree, , of the vertex denotes the number of vertices that vertex is connected to (number of neighbors), i.e.  .

We assume that there are groups of experimental units which are expected to give similar responses, e.g. because they share properties. We describe a method for finding these groups when they correspond to cliques in the network in Section 6. We label these groups and let group have experimental units within it. We model the responses of experimental units by the Network Block Model (NBM), which is an extension of the LNM (Parker et al., 2016) and is described by the equation

where , is the response from unit in the th block receiving the treatment , is the average response for the whole set of units (overall mean), is the effect of block , is the (direct) treatment effect, is the adjacency matrix indicating the edge between units and belonging to blocks and respectively, is the network effect (neighbor or indirect treatment effect) and are the errors, which we assume to be independent and identically distributed with mean and constant variance  .

To overcome the model overparametrization requires imposing some constraints, otherwise the normal equations have an infinite number of solutions and our parameters cannot be uniquely estimated. In our examples, without loss of generality, we assume the set-last-to-zero linear constraints with the th treatment effect and th block effect to be set equal to zero.

The expectation of the response for this model in matrix form is


is the vector of parameters. There are no columns corresponding to the

th treatment effect and the th block effect since we have assumed them to be zero. Following Bailey (2008, Sec. 2.7), vector corresponds to an vector where for each of the treatments () all the elements of the vector equal zero, except for those which correspond to the units which receive that treatment and are equal to one. For instance is the indicator vector with ones for the unit(s) receiving treatment and zeros elsewhere. The same holds for the vector for block . The symmetric information matrix is

where and are the incidence matrices for treatment and block effects respectively and and are the same matrices with their last columns deleted. We should note at this point that only certain types of network structures will enable us to estimate the network effects. With regular graphs the estimation of network effects is not possible.

3 Designs with block and network effects

In order to determine the optimal block designs we use the the L-optimality criterion that minimizes the average variance of the estimators of a pre-specified set of linear functions of the parameters , where corresponds to multiple vectors of known constants (Atkinson et al., 2007, Ch. 10). The design criterion to be minimized is thus a scalar function of which is proportional to , where corresponds to the information matrix and is a design chosen from the set of all possible designs. Thus the minimization of the optimality criterion function

where , leads to an L-optimal design. For the L-optimal design , , is the optimal function value.

To obtain these pairwise differences we have to explicitly define the vectors.

Definition. Let be a contrast vector formed in the following way:

  1. Form a vector of zeroes, corresponding to the intercept, treatment effects, , followed by block effects, , followed by network effects, .

  2. Let the th and th elements be 1 and -1 respectively, corresponding to the particular effects we wish to estimate in a given contrast.

  3. Delete the th and th elements, corresponding to our restriction that and to estimate the treatment and block effects uniquely.

The resulting vectors of length are pre- and post-multiplied by the matrix in the summation that defines the optimality criteria and which we now define.

We seek to minimize the average variance of all pairwise differences of treatment effects

This is proportional to

and similarly we seek to minimize the average variance of all pairwise differences of network effects

For instance, for the case of treatments and blocks the criteria become

where and . The first corresponds to the estimate of the treatment effect difference, and the second to the estimate of the difference between network effects.

Comparative measures of design performance also involve the information matrix. In particular, we often compare the performance of two designs using their relative efficiency. The relative efficiency with respect to the objective function of a design compared with a design is given by . We can also define the L-efficiency of a design as , where is the L-optimal design. We come back to these definitions when assessing the performance of different optimal designs based on different models for a number of given networks.

4 Exchange algorithm

When exhaustive search to find an optimal design is not possible due to the large number of units and/or treatments, we implement an approximate method. In the literature, there exist a number of computationally efficient algorithms for finding near-optimal designs in a practical amount of time using iterative methods, e.g., Fedorov (1972) and Cook and Nachtsheim (1980). The main steps involved in the majority of those algorithms are the following: (i) initialization of the search, e.g. random generation of a non-singular design (i.e. the matrix must be non-singular to ensure that the parameters in are estimable); (ii) modification of the current solution, e.g. make exchanges in the treatment set; (iii) assessment of new solution, e.g. design is assessed with respect to an objective function. Steps (ii)-(iii) are repeated until no change improves the design value; (iv) termination of the search process and return of the final design (which is assumed to be optimal).

We adapt the ‘Modified Fedorov Exchange Algorithm’ of Cook and Nachtsheim (1980) to obtain an exchange algorithm for finding near-optimal designs for unstructured treatments on networks. The algorithm can be described as follows:

Point Exchange on Networks (PEN) Algorithm

  • Step : Generate a random initial design (i.e. random assignment of a treatment label to each unit). Units are labeled from to .

  • Step : Calculate the optimality criterion function (, or any other defined criterion) for the arbitrary design of step .

  • Step : Iterate from to to find a better design (where corresponds to the -th subject). In the th iteration:

    1. Exchange the treatment applied to subject with another treatment.

    2. Calculate the information matrix. If it is non-singular, compute the new value of the chosen criterion. Otherwise go to d.

    3. If the new criterion function value is better than the current one, keep the exchange of treatments. Otherwise, undo the exchange and go to d.

    4. If , set .

  • Step : Repeat step until a complete pass of all the vertices yields no further improvement and then go to step .

  • Step : Rerun all the above steps for several randomly generated initial designs and return the design with the lowest criterion function value . This is the near-optimal design, which will usually be globally optimal, though this cannot be guaranteed. End.

The last step is required to obtain an efficient final design (and overcome the problem of becoming stuck in a local optimum) by using multiple random initializations. For the particular problems of interest in this work, the PEN algorithm appears to be powerful enough and simple to implement. The main advantage is that it has been appropriately adjusted to satisfy the exact needs of the particular designs of interest.

5 Comparison of optimal designs under different models

We aim to construct experimental designs on networks that are efficient for estimating treatment or network effects and that ideally lead to a small bias of the estimated model parameters. In this section we provide comparisons of optimal designs for estimating the treatment and network effects under traditional models that ignore network interference and models that account for it. This in turn leads to consideration of whether or not experiments should be randomized, when there is strong evidence of network structure underlying the experimental units. We also obtain the design bias introduced in the parameter estimates due to the model misspecification as a function of the unknown model parameters. The models we consider are the model derived from the completely randomized design , the model derived from the generalised block design , the linear network effects model and the linear network with blocks model , defined as:

In all cases, we assume that the errors are independent and randomly distributed with zero mean and constant variance. The CRM and RBM are the standard treatment models, that are derived from randomization schemes without and with blocks respectively. The designs for which these models are appropriate are the simplest forms of designs to compare different treatments by randomly assigning them to experimental units (which for RBM are additionally arranged in blocks). The LNM and NBM are extensions of CRM and RBM respectively, including a network term for capturing the connections between units. Figure 1 illustrates the hierarchy among the models by means of a Hasse diagram (see, e.g., Bailey, 2008). Here, we use the Hasse diagram to describe the collection of models considered, with dots representing models and lines representing nesting relationships between models. If the true model is a sub-model of the assumed model, the bias in estimating treatment effects will be zero. This will be further explained in the following examples. Designs which are optimal for each of these models are labeled CRD, RBD, LND and NBD respectively.

Figure 1: Hasse diagram for the collection of models.

We now compare the properties of each design for each model. In doing so we obtain the optimal function values for the two optimality criteria considering each model. Under the assumption of independent errors that have a common variance , the variance-covariance matrix of the least squares estimator is . Given that interest lies in the comparisons of the designs, the value is not relevant since the value is the same if the model is identical for all proposed designs for a particular experiment.

5.1 Efficiencies of randomized designs

We compare the randomized designs while ignoring the network structure to the optimal designs with network effects, providing evidence that in the majority of cases designs that account for the network structure have higher efficiency than the standard designs. The optimal randomized designs, under the CRM and the RBM, have equal replication of each treatment.

Example 1

The social network in Figure 2 is a co-authorship network, comprising subjects (PhD students, supervisors and co-authors) and edges, which indicate the ties between individuals with common publications, or ties between PhD students and their supervisors at the University of Southampton in the field of design of experiments in 2015. Let us assume that we want to conduct an experiment on this network in order to compare two distinct treatments. The three blocks have been defined following the method we suggest in Section 6, but any other means of deriving these blocks would give the same setup.

Figure 2: Social network with blocks indicated by colors.

We obtain the L-optimal designs (using exhaustive search) for this network which are illustrated in Figure 3. Overall the optimal design for has all treatments equally allocated to the subjects, i.e.  subjects receive each of the two treatments. Subjects allocated within each treatment have similar first and second order degrees (numbers of connections between units of distance one or two). The design for also has equal replication within each block. For , the design is highly dependent on the particular network structure. For instance subjects located at the ends of the network tend to receive the same treatment which is different from their better connected immediate neighbors.

Figure 3: Optimal designs (under NBM), (left) and (right); different colors indicate different blocks and different vertex shapes different treatments.

Tables 1 and 2 illustrate the optimality function values for each design (columns) under the different models (rows). We can obtain the efficiencies of the designs with respect to the optimal designs. The criterion function values of the optimal designs are highlighted in bold. Recall that the smaller the criterion function value the better the design is. A key observation derived from these tables is that a randomized design which ignores the network effects is on average highly inefficient, especially with respect to . Note the values for CRD and RBD are the mean values over all and possible randomizations respectively.

In Table 1, we can observe that when assuming the LNM to be true, the NBD, which has equal replication within blocks, is almost as efficient as the LND (with efficiency ). Under NBM the LND, which has equal replication overall but not within blocks, is efficient. Moreover, under the NBM, a RBD which has equal replication within blocks performs better than LND on average. It is worth noting that the optimal function values, which are (under the LNM) and (under the NBM) are approximately equal to the minimum average variance possible for the unstructured case when having independent units (i.e.  with ).

In Table 2 the CRD and RBD perform poorly under both models with their efficiencies being on average and under the LNM and NBM respectively when ignoring blocks, and and respectively when accounting for blocks. These low efficiencies can be explained by the fact that these standard designs ignore the indirect treatment effects resulting from the network structure among units. We are able to estimate the network effects only under the LNM and the NBM. Moreover, when assuming the NBM to be true, the LND is only efficient.

The general poor design performance of the LND relative to the NBD under the NBM results from having a completely different structure and different allocation to unequal- and relatively small-sized blocks. The optimization process for the LNM drives the design away from having an equal treatment allocation within blocks, resulting in many neighboring vertices receiving the same treatment. This means that, if we account for the blocks, we almost certainly do better.

Models Optimal designs
CRM 0.1818 0.1818 0.1818 0.1818
RBM 0.2034 0.1818 0.2685 0.1818
LNM 0.2126 0.2191 0.1822 0.1827
NBM 0.2500 0.2270 0.3621 0.1828
Table 1: Comparisons of the designs for under different models.
Models Optimal designs
LNM 0.1447 0.1927 0.0237 0.0366
NBM 0.2354 0.2503 0.0998 0.0388
Table 2: Comparisons of the designs for under different models.

The boxplots in Figure 4 show the L-efficiencies of the designs based on for all CRDs and RBDs. For , the CRD and RBD have median efficiencies and respectively, with minimum efficiencies and respectively. On the other hand for in Figure 5, CRD and RBD perform similarly with median efficiencies being and

, the lower quartiles being

and and the upper quartiles being and . This results from ignoring the network effects. The LND is approximately and as efficient as the NBD for and respectively.

Figure 4: Boxplots of efficiencies for (assuming the NBM).
Figure 5: Boxplots of efficiencies for (assuming the NBM).
Example 2

Moving on to a design on a moderate-sized social network (see Figure 6) we obtain optimal designs assuming each of the models. This network is a subset of the Facebook network as obtained from the Stanford network data set collection (snap.stanford.edu/data/egonets-Facebook.html), comprising vertices (Facebook members) and undirected edges (mutual virtual friendships) forming an ego-network. The ‘center’ vertex of an ego-network (the ‘ego’) is not included in graph , which consists of only the ego’s friends (its contacts). We wish to compare two treatments, which could correspond, for instance, to advertisements as discussed earlier.

The comparisons of the (near-)optimal designs under and under are given in Tables 3 and 4 respectively. Note the values for CRD and RBD are the mean values over a large number of possible randomizations (we have evaluated designs). Figures 7 and 8 depict the boxplots of efficiencies under L-optimality of the design based on and respectively for a number of possible designs (chosen at random) for CRD and RBD which do not take into account network effects, although they exist.

Figure 6: A Facebook ego-network with 24 blocks indicated by colors.
Models Optimal designs for )
CRM 1.2346 1.2347 1.2346 1.2346
RBM 1.3298 1.2432 1.3042 1.2432
LNM 1.2481 1.2747 1.2346 1.2348
NBM 1.3749 1.2907 1.3177 1.2432
Table 3: Comparisons of the designs for under different models.
Models Optimal designs for )
LNM 0.1121 0.1474 0.0119 0.0149
NBM 0.1553 0.1647 0.0312 0.0230
Table 4: Comparisons of the designs for under different models.
Figure 7: Boxplots of efficiencies for .
Figure 8: Boxplots of efficiencies for .

5.1.1 Recommendations

A number of design examples led us to make the following general guidelines:

  • On average the randomization process leads to fairly poor designs, i.e. high variance (this is especially true under ). For Example 2, all designs for were less than efficient which indicates that randomized designs are highly inefficient.

  • When ignoring both block and network effects, the design can perform poorly.

  • When taking into account the network effects but not the block effects, the design performs satisfactorily for but can perform poorly for when there is strong evidence of an underlying block structure.

  • By using blocks and ignoring network effects, with approximately a chance (upper quartile) one can do just as well as when taking into account the network effects, but on average one does worse (for ).

  • Evidence suggests that we will be better off by using blocks instead of ignoring them, independently of whether we are taking into account network effects (for ).

The optimal designs under the second optimality criterion are generally unbalanced, i.e. the treatments have unequal replication in the design. Therefore, it is not surprising that the standard designs perform poorly with median efficiencies lower than . Thus in practice, when there is a strong belief that the units are governed by a network structure and/or when blocking structure exists, then it is of importance to account for those in order to obtain an efficient design.

5.2 Bias due to model misspecification

We have shown that randomized designs are not efficient when used to fit the NBM. However, in practice, if a CRD or RBD is used, the CRM or RBM are likely to be used respectively. We now show that when we randomize and fit the corresponding model, the design can result in biased estimators of the quantities of interest. Although the bias has not been taken into account in the design optimality criteria, we can obtain it as a function of the model parameters and show that under the false assumption that there are no network effects the bias can be significant.

We employ Example 1 to investigate the potential impact of model bias. When fitting a model with no network effect (i.e. assuming CRM or RBM) we calculate the expectation of the bias introduced in the parameter estimates for all possible balanced designs (which are optimal when there is no network effect) in terms of the true unknown parameters. We derive the design bias in the estimators of the model parameters , , due to model misspecification in the Appendix, i.e.


and is the identity matrix.

Note that the bias depends on the chosen design. We obtain the biases under the different models with respect to the model parameters in Table 5. We see that if the network effects are substantial compared to the (direct) treatment effects, then ignoring them can potentially lead to over- or under-estimated treatment effects. Thus, by not taking into account network effects in our design, we produce an experiment which can have higher variance than necessary if the correct model is fitted and biased estimators if a simpler model is fitted.

True model Optimal designs
Table 5: Bias under model misspecification.

We can extract the bias of the treatment effects estimates (the second line in each matrix). Assuming that we wrongly ignore network effects, we have the following two cases of the bias for the direct treatment effects estimator, where the expectation is taken over all possible randomizations


If the true values of the network effects are zero (so that

), the balanced design will produce unbiased estimators. We should note that the network effects are not in general of the same size. However, for our investigation we assume them to be equal (

), and without loss of generality we set . Under this assumption the bias in (2) is slightly larger than in (1), i.e. . A conclusion drawn from this example is that if we use a model that ignores network effects there is no obvious benefit in terms of bias from including blocks. Thus having blocks does not insure us against the bias introduced by wrongly excluding network effects.

In a completely randomized design there is a chance of any specific pair of units receiving both treatments and . To investigate if the bias from treatment effects results from network effects passed on from units receiving treatment to those receiving treatment , we focus on the number of links connecting the different treatments, which we call . The plot in Figure 9 shows the bias (when assuming an optimal RBD under the true NBM) in treatment effect estimates stemming from network effects against the proportion of edges, which connect pairs of units receiving different treatments. Thus the expectation of the bias is over a conditional distribution of randomizations given a fixed proportion of edges. The locations of the plotting symbols are related to the coefficients of the network effects in the bias equation under each design and are dependent on the size of the true . As such they correspond to the bias from RBDs for estimating the treatment effects due to the network effects under the assumption that the underlying parameters are equal to one. The expectation of the bias over all designs for the intersection point, which corresponds to approximately half for the proportion of links, is equal to zero. It should be noted that many designs are overlapping and each location of the plotting symbols can represent the bias due to network effects for alternative RBDs (see Table 6).

38 322 1532 4710 9460 12742 11838 7674 3540 952 112
0.37 0.41 0.44 0.48 0.52 0.56 0.59 0.63 0.67 0.7 0.74
Table 6: Number of balanced designs for each proportion of edges.

The boxplots in Figure 10 depict the bias in the estimation of treatment effects over all possible RBDs in the case that we assume that for every proportion of links. This expectation is conditional on the proportion of links being fixed, but averaged over the subset of randomizations which respect that. The limits of the boxplots are relative to the size of the true network effect. This plot suggests that in general the complete randomization while ignoring the network structure does not perform well. In combination with the previous observation, we can conclude that the least expected bias in the treatment effects is achieved when the number of pairs of connected units which receive different treatments equals approximately half the total number of edges of the network (see Figure 10 at the proportion ).

Figure 9: Bias in treatment effects due to network effects.

The experimenter should impose restrictions on the allocation of treatment combinations such that the number of connected pairs of units receiving different treatments roughly equals half of the total number of edges in the network. Restrictions on the randomization in such a way enables us to protect the experimental results against bias in treatment effects stemming from potential network effects. For more information about restricted randomization refer to Mead et al. (2012, Ch. 11).

Figure 10: Bias for treatment effects when .

6 Defining blocks using spectral clustering

In our model, we assume that individuals who belong to closed-form communities exhibit similar response patterns within the same communities and dissimilar ones from units of different communities irrespective of the presence or absence of viral effects of specific treatment. Sometimes these communities are defined by features not directly related to the graph structure, such as age group, sex, nationality, etc. At other times they are defined by the structure of the graph itself. Then we have to deduce the community structure by using a clustering algorithm.

Spectral clustering, as the name implies, makes use of the graph spectrum, i.e. the set of eigenvalues and eigenvectors of a matrix describing the graph structure, to group vertices into communities. The graph spectrum plays a major role in the understanding of the structure and dynamics in the network since it is linked with numerous graph properties

(Chung, 1997; Von Luxburg, 2007). Spectral clustering is a versatile clustering method which can be applied to any network and is considered to give good quality solutions. To identify the number of clusters, we additionally use the modularity of the network, which is a measure which quantifies the strength of the community structure in a network (Newman, 2006).

Apart from the adjacency matrix, a graph can be represented by other connectivity matrices such as the Laplacian matrices, which are the basic tools for spectral clustering. The main differences between spectral clustering techniques lie in the choice of the Laplacian matrix. For an in-depth coverage of the principles of spectral clustering refer to Chung (1997) and Von Luxburg (2007) and references therein. We chose to obtain our results using the normalized Laplacian graph, , which is related to a random walk making it a useful tool for capturing a diffusion process, such as the treatment propagation effects, in a network. It is given as , where is the diagonal degree matrix (where the entries are the degrees of the vertices). In fact, is the transition matrix of a standard random walk on the given graph. Recall that a walk is a sequence of edges connected to a sequence of vertices, where vertices can appear more than once. A random walk is a walk across a network created by taking repeated random steps. The vertices with high degree are more likely to be visited by the random walk because there are more ways of reaching them.

For detecting communities we implement the normalized spectral clustering algorithm, namely the Shi and Malik (SM) algorithm (Shi and Malik, 2000) as described by Von Luxburg (2007). We describe the SM algorithm in the Appendix with appropriate adjustments for the purposes of this work. This clustering algorithm gives no hint about the choice of the number of clusters or ‘goodness’ of each partition. To quantify the quality of partitions and to choose the number of communities based on the ‘best’ partition, we use modularity (Newman and Girvan, 2004), which is defined as the difference between the fraction of the edges that fall within clusters and the fraction of the edges that would be expected to fall within the clusters if the edges were assigned randomly but keeping the degrees of the vertices unchanged. When the communities are not better than the random partition or when the network does not exhibit any community structure, the modularity score is zero or negative (see Appendix for the detailed definition). As such is regarded as the appropriate candidate for indicating the number of intrinsic communities which will define the blocks to be used in the design process, if the th partition corresponds to the highest modularity score over all partitions.

Revisiting Example 2 we provide and implement the step-by-step methodology.

Step one: Blocking Structure. Expressing the topology of the network through the normalized Laplacian matrix , we produce a number of possible network partitions for different values of (dimensionality of the eigenvector space, i.e. the number of eigenvectors) using the spectral clustering algorithm SM. Then we assess all the partitions produced using the quality function of modularity and choose the number of communities (fixed ) to be used for the block design, which maximizes the value of modularity. For the social network at hand the maximum modularity value over all possible partitions is found for clusters. Figure 11 illustrates the modularity scores for different numbers of communities obtained via the graph partition. This screening stage helps us to choose the number of communities, where the modularity score takes its maximum value. The resulting clustering is illustrated in Figure 6 with different colors indicating the different 24 blocks.

Figure 11: Modularity values for partitions of .

Step two: Optimal design with block and network effects (using PEN). The optimal function values resulting from this search (allowing for multiple initial designs) were found to be and . Similar to the first example, is approximately equal to the estimated variance of the difference of the two treatments when the units are independent and the model does not contain network or block effects (that is ). Whilst the optimal design for estimating has and units receiving treatments and respectively, the optimal design for estimating the difference in network effects has and units receiving treatments and respectively.

7 Discussion

Many experiments can be regarded as being on networks by appropriate specification of the connectivity matrix. The suggested methods are easily adaptable to a wide class of networked experiments and to different block definitions for enabling the researcher to make comparisons between different treatments both directly and indirectly.

This paper offers a framework for finding optimal block designs with network effects. It provides evidence that conventional designs such as randomized designs while ignoring the network structure are inefficient compared to block designs with network effects. We saw that a model which wrongly ignores block and/or network effects leads to inefficient designs.

We also suggest a way to define blocks, while taking into account the community structure. We justify our choice of using spectral clustering due to the simplicity of its implementation (standard linear algebra), without requiring any explicit distributional model and without making any kind of assumption on the formation of the clusters. We have three main components: spectral clustering of the given social network to project the vertices of the network onto an eigenvector-space; modularity to choose the ‘best’ partition of the network over all partitions; and optimal block designs, for the clusters of subjects produced.

Based on a wide variety of examples similar to the ones we presented, we can suggest some general guidelines. Heuristically the experimental design for estimating the treatment effects tends to have equal replication within each block on the units having a similar number of connections, while for estimating the network effects the design is greatly influenced by the network’s first and second order connections.


Bias due to model misspecification

We provide the derivation of the bias in the estimators of the model parameters due to model misspecification.

Consider performing an experiment on a network, where the adjacency matrix, , is given. We assume that the true model is the NBM. However, the postulated model for the experiment is the RBM, which ignores network effects (i.e. ). We want to obtain the bias in the parameter estimates due to the model misspecification as a function of the unknown model parameters. This will be accomplished by using generalized inverse matrices (Harville, 1997, Ch.9). Let and be estimators of with and being the extended design matrices under the NBM and RBM respectively. Note that is the best linear unbiased estimator for so that . With the necessary algebraic calculations on generalized inverse matrices, it follows that the bias of the design, , under the assumption that there are network effects but we do not take them into account, is

where is the identity matrix. Let and represent the and matrices respectively

Then for the block-diagonal matrix

we have that is a non-singular matrix (it has full rank) and, defining a matrix

where is of dimension , we obtain

implying that is a generalized inverse of if and only if , or if and only if . Hence, we have

Thus the bias introduced in the estimates of the parameters, , under the false assumption that there are no network effects is given by the quantity . This quantity is the result of ignoring the network effects, which is represented by an adjustment of the intercept and the treatment effect estimates. Observe that , unless which results from or .

The Shi and Malik algorithm

The SM algorithm involves three steps: I. Compute the normalized graph Laplacian and its spectrum (as based on the known adjacency matrix of the network); II. Dimensionality reduction: using the first eigenvectors of the graph Laplacian that correspond to the first eigenvalues sorted in ascending order, let be the matrix containing as columns. (Note that the eigenvector-space varies based on the chosen dimensionality .); III. Clustering step: treating each row of as a data point,

, group them via the (standard) k-means algorithm into

(dimensionality of the eigenvector space) clusters, . Therefore, the vertices of the network are projected into a -dimensional space, where is the number of the first nontrivial eigenvectors of . As a result, each unit is allocated to one cluster.

We perform the clustering step for various numbers of clusters , , the upper bound chosen so that there should be at least two units within a cluster for achieving treatment comparisons. Note that the partition method can rely on a different clustering step rather than k-means if required. However, we used this standard one, which is the most used partitioning method.

This clustering algorithm gives no hint about the choice of the number of clusters or ‘goodness’ of each partition. To quantify the quality of partitions and to choose the number of communities based on the ‘best’ partition, we use modularity,

where and are binary indicators of whether vertices and belong to group or not (membership vectors) and is the total degree of all the vertices. In other words, modularity is defined as the difference between the fraction of the edges that fall within clusters and the fraction of the edges that would be expected to fall within the clusters if the edges were assigned randomly but keeping the degrees of the vertices unchanged. The factor of accounts for the fact that every vertex pair is counted twice. When the communities are not better than the random partition or when the network does not exhibit any community structure, is zero or negative.

As such is regarded as the appropriate candidate for indicating the number of intrinsic communities which will define the blocks to be used in the design process, if the th is the highest modularity score over all produced partitions, i.e. .


The authors gratefully acknowledge the Economic and Social Research Council (ESRC) for funding this research.


  • Atkinson et al. (2007) Atkinson, A. C., A. N. Donev, and R. D. Tobias (2007). Optimum Experimental Designs, with SAS. Oxford: Oxford University Press.
  • Bailey (2008) Bailey, R. A. (2008). Design of Comparative Experiments. Cambridge: Cambridge University Press.
  • Besag and Kempton (1986) Besag, J. and R. A. Kempton (1986). Statistical analysis of field experiments using neighbouring plots. Biometrics 42(2), 231–251.
  • Chung (1997) Chung, F. R. K. (1997). Spectral graph theory. In CBMS Regional Conference Series in Mathematics, Number 92. American Mathematical Society.
  • Cook and Nachtsheim (1980) Cook, R. D. and C. J. Nachtsheim (1980). A comparison of algorithms for constructing exact D-optimal designs. Technometrics 22, 315–324.
  • Druilhet (1999) Druilhet, P. (1999). Optimality of neighbor-balanced designs. Journal of Statistical Planning and Inference 81, 141–152.
  • Fedorov (1972) Fedorov, V. V. (1972). Theory of Optimal Experiments. New York: Academic Press.
  • Harville (1997) Harville, D. A. (1997). Matrix Algebra from a Statistician’s Perspective. New York: Springer.
  • Kunert and Martin (2000) Kunert, J. and R. J. Martin (2000). On the determination of optimal designs for an interference model. The Annals of Statistics 28(6), 1728–1742.
  • Kunert and Mersmann (2011) Kunert, J. and S. Mersmann (2011). Optimal designs for an interference model. Journal of Statistical Planning and Inference 141(4), 1623–1632.
  • Mead et al. (2012) Mead, R., S. G. Gilmour, and A. Mead (2012). Statistical Principles for the Design of Experiments. Cambridge: Cambridge University Press.
  • Newman (2006) Newman, M. E. J. (2006). Modularity and community structure in networks. Proceedings of the National Academy of Sciences of the United States of America 103(23), 8577–8582.
  • Newman and Girvan (2004) Newman, M. E. J. and M. Girvan (2004). Finding and evaluating community structure in networks. Physical Review E 69(2), 026113.
  • Parker et al. (2016) Parker, B., S. G. Gilmour, and J. Schormans (2016). Optimal design of experiments on connected units with application to social networks. Journal of the Royal Statistical Society, Series C 66(3), 455–480.
  • Pearce (1957) Pearce, S. C. (1957). Experimenting with organisms as blocks. Biometrika 44(1-2), 141–149.
  • Shi and Malik (2000) Shi, J. and J. Malik (2000). Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(8), 888–905.
  • Von Luxburg (2007) Von Luxburg, U. (2007). A tutorial on spectral clustering. Statistics and Computing 17(4), 395–416.