Fixed-effects model: the most convincing model for meta-analysis with few studies

02/11/2020 ∙ by Enxuan Lin, et al. ∙ 0

According to Davey et al. (2011) with a total of 22,453 meta-analyses from the January 2008 Issue of the Cochrane Database of Systematic Reviews, the median number of studies included in each meta-analysis is only three. In other words, about a half or more of meta-analyses conducted in the literature include only two or three studies. While the common-effect model (also referred to as the fixed-effect model) may lead to misleading results when the heterogeneity among studies is large, the conclusions based on the random-effects model may also be unreliable when the number of studies is small. Alternatively, the fixed-effects model avoids the restrictive assumption in the common-effect model and the need to estimate the between-study variance in the random-effects model. We note, however, that the fixed-effects model is under appreciated and rarely used in practice until recently. In this paper, we compare all three models and demonstrate the usefulness of the fixed-effects model when the number of studies is small. In addition, we propose a new estimator for the unweighted average effect in the fixed-effects model. Simulations and real examples are also used to illustrate the benefits of the fixed-effects model and the new estimator.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Meta-analysis is a quantitative method for synthesizing results from multiple independent studies. Through combining the randomized trials of the same comparison of interventions, the estimates of correlation between two variables, or the measures of the accuracy of a diagnostic test, meta-analysis can help researchers to understand inter-study differences, to compare multiple treatments simultaneously, and, more generally, to synthesize evidence from different types of studies (Berkey et al., 1995; Hunter and Schmidt, 2004; Lu and Ades, 2004; Ades and Sutton, 2006; Leeflang et al., 2008; Borenstein et al., 2011). According to Web of Science Core Collection as of 12 December 2019, there are more than 186,600 publications in the literature with the keyword of “meta-analyses” in the title or abstract. Without any doubt, meta-analysis has become one of the most important areas in medical research.

There are two commonly used models in meta-analysis: the common-effect model and the random-effects model. The common-effect model, also known as the fixed-effect model, assumes one common effect across all the studies, denoted by the common effect

. In contrast, the random-effects model assumes that the true effect sizes may vary from study to study but they follow a certain distribution, e.g. a normal distribution. The summary effect in the random-effects model is known as the

mean effect, which is the mean of the underlying distribution.

According to Davey et al. (2011) with a total of 22,453 meta-analyses from the January 2008 Issue of the Cochrane Database of Systematic Reviews, the median number of studies included in each meta-analysis is only three. In other words, about a half or more of meta-analyses conducted in the literature include only two or three studies. For meta-analysis with few studies in the presence of heterogeneity, the common-effect model and the random-effects model are likely to yield misleading or unreliable results (Hedges and Vevea, 1998; Poole and Greenland, 1999; Henmi and Copas, 2010; Gonnermann et al., 2015; Röver et al., 2015; Bender et al., 2018

). On one hand, when the observed effect sizes have large differences, the common-effect model often has an inflated type I error rate and provides a poor coverage probability for estimated confidence interval of the

common effect. On the other hand, the estimate of the mean effect in the random-effects model relies heavily on the between-study variance estimate. For meta-analysis with few studies, however, it is unlikely to have a reliable estimate for the between-study variance(Cornell et al., 2014).

In addition to the two common models, there is another model available for meta-analysis termed as the fixed-effects model (Laird and Mosteller, 1990; Hedges and Vevea, 1998). The fixed-effects model assumes that the effect sizes of the studies are deterministic and different. That is, it neither assumes the study-specific effect sizes to be all the same as in the common-effect model, nor requires them to follow a certain distribution as in the random-effects model. For meta-analysis with few studies, the fixed-effects model is capable to provide a good balance between the common-effect model and the random-effects model. It not only avoids the unreasonable assumption on a common effect in the common-effect model when the heterogeneity exists, but also avoids the low accuracy of the between-study variance estimate in the random-effects model when there are only few studies.

For the fixed-effects model, two summary effects have been suggested in the literature: the unweighted average effect (Laird and Mosteller, 1990; Bender et al., 2018) and the weighted average effect (Rice et al., 2018). The summary effects selection should depend on the research goal of the meta-analysis. If all the studies are regraded as equally important, the unweighted average effect by Laird and Mosteller (1990) should be used. We note that the summary effects and conclusions for the fixed-effects models are pertaining to the observed studies only. The estimator of the weighted average effect proposed by Rice et al. (2018) is the same as the estimator of the common effect in the common-effect model. For the unweighted average effect

, the existing unbiased estimator by

Laird and Mosteller (1990) does not fully utilize the information across the studies.

In this paper, we demonstrate that the fixed-effects model can serve as the most convincing model for meta-analysis with few studies through real examples. For the unweighted average effect in the fixed-effects model, in addition to the unweighted average, we propose a new estimator with different weight for each study that fully utilizes the information across the studies. Through theory and simulation, we demonstrate that our optimal estimator performs better than the unweighted average estimator in all settings in the sense of having minimal mean squared error.

The rest of the paper is organized as follows. In Section 2, we review the three models for meta-analysis. In Section 3, we propose an optimal estimator for the unweighted average effect and assess its performance through numerical studies. In Section 4, we illustrate the usefulness of the fixed-effects model using real examples. Finally, we conclude the paper with a brief discussion in Section 5, and provide the technical results in the Appendix.


2 Existing models for meta-analysis

The common-effect model and the random-effects model are two most popular models for meta-analysis. They were first introduced in the 1950s (Bliss, 1952; Cochran, 1954), and have now been widely adopted for synthesizing multiple studies. The fixed-effects model was introduced in Laird and Mosteller (1990) and Hedges and Vevea (1998), but received little attention until recently (Bender et al., 2018; Rice et al., 2018). In this section, we provide a brief review on the three models, including their model assumptions, their effect sizes, and their key limitations in performing a meta-analysis.

Figure 1: Schematics of the three models for meta-analysis: the common-effect model, the fixed-effects model, and the random-effects model. The squares represent the observed effect sizes, the diamonds represent the study-specific effect sizes, and the circles represent the summary effects.

2.1 Common-effect model

The common-effect model assumes that all the studies share a common effect, for which we denote it by . In other words, as shown in Figure 1(a), the differences in the observed effect sizes are all subject to sampling errors. Mathematically, we can express the common-effect model as

(1)

where are the observed effect sizes, is the common effect, are the random errors for studies , and is the total number of studies. In addition, the random errors are assumed to be normally distributed with zero mean and variances , where “ind” represents the random errors are independent of each other. To avoid confusion with the “fixed-effects” model in Figure 1(b) or more specifically in Section 2.3, we refrain from using the fixed-effect model, but refer to model (1) consistently as the common-effect model throughout the paper.

With model (1), the common effect can be unbiasedly estimated as

where are the study-specific weights that are derived by minimizing the mean squared error (MSE) of (Laird and Mosteller, 1990). For ease of reference, we have also provided a detailed derivation of the study-specific weights in Appendix A. A study with a large sample size will more likely yield a small study-specific variance, and consequently, a large weight will be assigned to that study to quantify the level of reliability.

Further by the normality assumption on the random errors, follows a normal distribution with mean and variance ; and accordingly, a confidence interval (CI) of is . Note that the estimates of the within-study variances are treated as true variances to compute the common effect and its CI.


2.2 Random-effects model

In many studies, the differences between the observed effect sizes can be relatively large such that they are not able to be explained fully by sampling errors. Such differences are well known as the heterogeneity of the studies. The heterogeneity may arise from various aspects, including, for example, the different designs in experiments or the different covariants in the model regression (Brockwell and Gordon, 2001). When such a scenario occurs, a common effect for all the studies is unlikely to be true; and consequently, if model (1

) is still applied, it often yields underestimated standard errors of the estimated common effects and confidence intervals with insufficient coverage.

To account for the heterogeneity of the studies, as shown in Figure 1(c), a random-effects model is often considered by assuming the study-specific effect sizes also follow an underlying distribution, e.g., a normal distribution. To be more specific, the random-effects model can be expressed as

(2)

where are the observed effect sizes, is the mean effect, are the deviations of the study-specific effect sizes from the mean effect, and are the random errors. We further assume that the deviations are independent and identically distributed (i.i.d.) from , the random errors are the same as in model (1), and they are independent of each other. In addition, we refer to as the between-study variance and as the within-study variances.

With model (2), if is known, the mean effect can be unbiasedly estimated as

where are the study-specific weights that are derived by minimizing the MSE of (Laird and Mosteller, 1990). For a detailed derivation of the study-specific weights, please refer to Appendix B. For the special case when , the random-effects model reduces to the common-effect model. Further by the normality assumptions on both the effect sizes and the random errors, we have following a normal distribution with mean and variance ; and accordingly, a CI of is . Note that the between-study variance is unknown and needs to be estimated from the data. In particular, there exists many methods for estimating

in the literature including the moments estimator, the likelihood-based estimate, and the empirical Bayes estimate. For a detailed review and comparison of these existing estimators, one may refer to

Sidik and Jonkman (2010), Veroniki et al. (2016), and the references therein. When few studies are included in a meta-analysis, a reliable estimate of the between-study variance is hard to achieve for most existing methods including the Paule and Mandel (1982) method and the DerSimonian and Laird (1986) method. As illustrated in Section 4, the resulting CI may be too wide to be useful in practice.


Figure 2: Schematics of the three models for meta-analysis: the common-effect model, the fixed-effects model, and the random-effects model. Unlike the common-effect model and the random-effects model, the fixed-effects model does not exist a clear position to place the circles for the summary effect.

2.3 Fixed-effects model

In the fixed-effects model, the effect size of each study is assumed to be a fixed but unequal quantity (Ramsey and Schafer, 2012). That is, we neither require the study-specific effect sizes to be all the same as in model (1), nor assume that they follow a certain distribution as in model (2). Under these specifications, we can present the fixed-effects model as

(3)

where are the observed effect sizes, are the study-specific effect sizes, and are independent random errors that are normally distributed with zero mean and variance . Model (3) is referred to as a fixed-effects model since

are deterministic rather than random variables as in model (2).

A major limitation of the fixed-effects model is that it does not provide a clear summary effect as the common effect in model (1) or the mean effect in model (2). In other words, there does not exist a clear position to place the circles for the summary effect, as shown in Figure 2. Two summary effects have been proposed in the literature. One is the unweighted average effect as defined in Laird and Mosteller (1990) and Bender et al. (2018), and the other is the weighted average effect as defined in Rice et al. (2018).

Unweighted Average Effect: For the fixed-effects model, by treating each study as equally important, one natural summary effect is

(4)

We refer to this summary effect as the unweighted average effect. Following this definition, Laird and Mosteller (1990) proposed to estimate by

(5)

Further by the normality assumption in model (3), follows a normal distribution with mean and variance , and a CI of is .

Weighted Average Effect: Allowing the weights to depend on the within-study variances and the sample sizes, Rice et al. (2018) proposed the weighted average effect as

where are the sample sizes, are the proportions associated with each individual study, and reflect an average rate of information over the observations from the th study. They further proposed to estimate the weighted average effect by

(6)

Note that is the same as the unbiased estimator of the common effect in the common-effect model. Under the normality assumption in model (3), follows a normal distribution with mean and variance , and a CI of is .


3 Optimal estimation for the fixed-effects model

For better application of the fixed-effects model in meta-analysis with few studies, we propose an optimal estimator for the unweighted average effect in this section. We also assess the estimation accuracy of the proposed estimator through numerical studies.


3.1 Optimal estimator of the unweighted average effect

Note that the common effect estimator and the mean effect estimator are both weighted averages of the observed effect sizes, in which the weights are assigned by the inverse-variance method. Following the same logic, we propose to estimate the unweighted average effect in (4) by

(7)

where are the weights assigned to each individual study. In the special case when for all , our new estimator will be the same as the unweighted estimator in (5). It is worth nothing that, even though is an unbiased estimator, it does not guarantee to be the optimal estimator for .

To derive the optimal weights in (7), we choose to minimize the MSE of as follows:

We have shown in Appendix C that, when

(8)

for , there exists a unique positive solution for the optimal weights as

(9)

Note that assumption (8) is to guarantee the weights to be positive. When the assumption does not hold, there is no explicit solution for the optimal weights. In such settings, the computational methods such as interior point and conjugate gradient can be used to find the numerical solutions. When , assumption (8) always holds and we have

When , if assumption (8) holds, the optimal weights can also be expressed as

Compared to the unbiased estimator in (5), our new estimator fully utilizes the information across the studies, and hence provides a smaller MSE. Specifically, the bias of our new estimator is

When for all , it follows that and which are exactly the same as in Section 2.1. That is, when the fixed-effects model reduces to the common-effect model, our proposed estimator will be exactly the same as the common effect estimator in the common-effect model. Note that is derived with minimizing the MSE, which means This leads to

where the equality holds only if , that is, so that is the same as . This shows that our new estimator always has a smaller variance than the existing estimator. Taking as an example, the difference between the two variances can be expressed as which is non-negative with the equality holding only when .

From another perspective, we note that where . Therefore, the optimal estimator is a weighted average of the unweighted average effect estimator by Laird and Mosteller (1990) and the weighted average effect estimator by Rice et al. (2018). When all are the same, it reduces to in (6). Finally, under the assumptions of the fixed-effects model, follows a normal distribution with mean and variance , and a CI of is .


3.2 Numerical comparison

Since a half or more of meta-analyses include only two or three studies, we therefore conduct numerical studies to evaluate the performance of our new estimator for and 3. The unbiased estimator is also considered for comparison to explore how much improvement that our new estimator can achieve. To visualize the numerical results, we compute the MSE, the squared bias, and the variance of the two estimators, and plot them in Figure 3 for and in Figure 4 for .

In the first study with , we consider the fixed-effects model in (3) with several different settings of and . Specifically for the top three plots in Figure 3, we consider , , , and with the difference ranging from 0 to 10; while for the bottom three plots, we consider , , , and with the ratio ranging from 1 to 10. Our second study is with . For the top three plots in Figure 4, we consider , , , , , and with the difference ranging from 0 to 10; while for the bottom three plots, we consider , , , , , and with the ratio ranging from 1 to 10.

From the numerical results, it is evident that our optimal estimator has a smaller MSE than the unbiased estimator in all settings for and . It is also evident that the MSE as the sum of the squared bias and the variance is mainly contributed by the variance of the estimator. The improvement increases as the difference between means decreases or the ratio between variances increases. Note that we have also conducted numerical studies for many other settings (not shown to save space), and the comparison results remain similar as those in Figures 3 and 4. To conclude, in comparison with the unbiased estimator, our new estimator sacrifices a small bias for a significant reduction in variance of the estimator so that the MSE decreases.

Figure 3: Comparison of the two estimators for the unweighted average effect with . The red lines with empty circles represent the results of the unbiased estimator , and the blue lines with solid circles represent the results of our new estimator . The top three plots represent the results along with the range of , and the bottom three plots represent the results along with the range of .
Figure 4: Comparison of the two estimators for the unweighted average effect with . The red lines with empty circles represent the results of the unbiased estimator , and the blue lines with solid circles represent the results of our new estimator . The top three plots represent the results along with the range of , and the bottom three plots represent the results along with the range of .

4 Applications to meta-analysis with few studies

In this section, we apply the fixed-effects model to four real examples in clinical trails, including studies with both continuous and binary outcomes, and compare them with the common-effect model and the random-effects model. We compute both the optimal estimator and the Laird and Mosteller (L-M) estimator for the unweighted average effect in the fixed-effects model.

4.1 Meta-analysis with two studies

The first example is with Ding et al. (2018)

, who reported a systematic review about the association between three genetic variants (rs1058205, rs2735839, and rs266882) in kallikrein 3 (KLK3) and prostate cancer risk, with the observed effect sizes being the odds ratios (OR). The genotypes reflected the disease include CC, CT, and TT. In a subgroup meta-analysis, only two studies are included as shown in Figure

5(a). The first study indicates a significant association between the rs1058205 TT and CC (OR = 0.29, CI = 0.19 to 0.43), whereas the second study does not give a significant result (OR = 0.93, CI = 0.78 to 1.12). Combining the two studies, the common-effect model gives a significant result for the common effect (OR = 0.77, CI = 0.66 to 0.89). Nevertheless, with the large heterogeneity between the two studies (, ), the assumption of a common effect may not hold so that the meta-analytic results from the common-effect model may be misleading. On the other hand, the random-effects model does not provide a significant result for the mean effect with a wide CI (OR = 0.53, CI = 0.17 to 1.67). In contrast, different from the above two models, the fixed-effects model does not involve an unstable estimate of the between-study variance, and meanwhile it does not impose the unrealistic assumption of a common effect. Specifically, it provides significant results for the unweighted average effect (the L-M estimator: OR = 0.52, CI = 0.42 to 0.63; the optimal estimator: OR = 0.53, CI = 0.43 to 0.64), which is very close to the mean effect in the random-effects model (OR = 0.53); and more importantly, it yields a narrower CI for the unweighted average effect which is about the same length as that for the the common effect in the common-effect model.

The second example investigated the effectiveness of interventions for reducing nonoccupational sedentary behavior in adults and older adults (Shrestha et al., 2019). As shown in Figure 5(b), the observed effect sizes are the mean differences (MD). Neither study showed a significant difference between the two groups (study 1: MD = -75.2, CI = -153.4 to 3; study 2: MD = -7.5, CI = -32.17 to 17.17). Combining the two studies, the common-effect model gives a non-significant result for the common effect (MD = -13.63, CI = -37.15 to 9.9). The random-effects model also does not have a significant result for the mean effect (MD = -30.76, CI = -93.78 to 33.25). To summarize, neither the common-effect model nor the random-effects model reports a significant result. Nevertheless, we note that the fixed-effects model with the L-M estimator reports a significant result (MD = -41.35, CI = -82.35 to -0.35), which may not be very reasonable. Fortunately, the fixed-effects model with our optimal estimator leads to the same non-significant result as the common-effect model and the random-effects model (MD = -33.69, CI = -67.51 to 0.13), which is close to the estimate of the mean effect in the random-effects model.

Figure 5: Meta-analyses with exactly two studies, where (a) represents the association between the genetic variant rs1058205 in kallikrein 3 (KLK3) and prostate cancer risk, and (b) represents the association between asthma and periodontal disease in adults.

4.2 Meta-analysis with three studies

The first example is with Armitage et al. (2019). The authors reported a systematic review about the efficacy and safety of statin therapy in older people, with the observed effect sizes being the risk ratios (RR). In a subgroup of effects on major vascular events per mmol/L reduction in LDL cholesterol, subdivided by the age larger than 75, only three studies are included for meta-analysis as shown in Figure 6(a). The first two studies both report non-significant results (study 1: RR = 0.95, CI = 0.82 to 1.11; study 2: RR = 0.95, CI = 0.81 to 1.12), and third study shows a significant result (RR = 0.77, CI = 0.75 to 0.80). Combining the three studies, the common-effect model gives a significant result for the common effect (RR = 0.79, CI = 0.77 to 0.81). Nevertheless, with the large heterogeneity among the three studies (, ), the meta-analytic results based on a common effect assumption may be misleading in practice. On the other hand, the random-effects model does not show a significant result for the mean effect with a wide CI (RR = 0.88, CI = 0.74 to 1.04). Compared to the two models, the fixed-effects model provides significant results for the unweighted average effect (the L-M estimator: RR = 0.89, CI = 0.83 to 0.95; the optimal estimator: RR = 0.88, CI = 0.83 to 0.94), which is very close to the estimate of the mean effect in the random-effects model; and both estimators yield narrower CIs for the unweighted average effect than that for the mean effect in the random-effects model.

Figure 6: Meta-analyses with exactly three studies, where (a) represents the efficacy and safety of statin therapy in older people, and (b) represents the effectiveness of interventions for reducing nonoccupational sedentary behavior in adults and older adults.

The second example is extracted from the systematic review to investigate the effectiveness of interventions for reducing nonoccupational sedentary behavior in adults and older adults (Shrestha et al., 2019, Figure 6(b)). We address the outcomes in the subgroup of the continuous outcomes, which includes only three studies. The observed effect sizes in this meta-analysis are the mean differences (MD). Specifically, the first study reports a non-significant result (MD = 5.8, CI = -7.77 to 19.37), the second study also reports a significant result (MD = -14, CI = -20.36 to -7.64), and the third study reports a non-significant result (MD = 6, CI = -10.64 to 22.64). Combining the three studies, the common-effect model gives a significant result for the common effect (MD = -8.68, CI = -14.12 to -3.23). Nevertheless, with the large heterogeneity among the three studies (, ), the meta-analytic results from the common-effect model may be misleading. In contrast, the random-effects model does not offer a significant result for the mean effect (MD = -1.99, CI = -17.4 to 13.42). In this example, we note that the fixed-effects model also show non-significant results by both estimators (the L-M estimator: MD = -0.73, CI = -8.2 to 6.73; the optimal estimator: MD = -1.45, CI = -8.49 to 5.59). Specifically, it provides a point estimate of the unweighted average effect that is nearly the same as that for the mean effect in the random-effects model (SMD = -1.99), and meanwhile it produces a much narrower CI for the unweighted average effect than that for the mean effect in the random-effects model. Note also that the optimal estimator has, once again, a slightly narrower CI than the L-M estimator.

Based on the four real examples, we find that, based on different assumptions, the three different models may have different summary effect estimates and conclusions. The common-effect model should be avoided in the presence of heterogeneity. On the other hand, the CI for the mean effect of the random-effects model is usually too wide to be useful when the number of studies is small. The fixed-effects model tends to have estimates of unweighted average effect close to the estimate of the mean effect in the random-effects model with much narrower CIs. Therefore, they provide a more meaningful summary for the effect sizes when the number of studies is small and the heterogeneity is non-negligible.


5 Conclusion

Two commonly used models for meta-analysis, the common-effect model and the random-effects model, both have their own limitations. The common-effect model may lead to misleading results when the heterogeneity among studies cannot be ignored. The CI for the mean effect based on the random-effects model may be too wide to be useful when the number of studies is small. These limitations are major obstacles in practice since a large portion of meta-analyses have small numbers of studies with non-negligible heterogeneity. On the other hand, the fixed-effects model not only avoids the unreasonable assumption on a common effect in the common-effect model when the heterogeneity exists, but also avoids the low accuracy of the between-study variance estimate in the random-effects model when there are only few studies.

In this paper, we demonstrated that the fixed-effects model can serve as an alternative approach for meta-analysis under these situations with conclusions limited to the observed studies. We also proposed an optimal estimator for the unweighted average effect in the fixed-effects model that fully utilizes the information across the studies. With a small increase in bias but a large reduction in variance, the optimal estimator always has a smaller MSE than the existing estimator in the literature. Also as shown in Figure 2, which quantity should be used as a summary effect in the fixed-effects model is not immediately clear. Both the unweighted and weighted average effects were proposed in the literature. The unweighted average effect treats each study as equally important and is easy to interpret (Laird and Mosteller, 1990; Bender et al., 2018). In contrast, the weighted average effect allows researchers to assign different weights to different studies (Domínguez Islas and Rice, 2018; Rice et al., 2018). Which average effect to use for the fixed-effects model may depend on different factors such as the goal of the meta-analysis and the sample sizes of different studies, and may require further research.


References

  • Ades and Sutton (2006) Ades, A. E. and Sutton, A. J. (2006). Multiparameter evidence synthesis in epidemiology and medical decision-making: current approaches. Journal of the Royal Statistical Society: Series A, 169(1), 5–35.
  • Armitage et al. (2019) Armitage, J., Baigent, C., Barnes, E., Betteridge, D. J., Blackwell, L., Blazing, M., Bowman, L., Braunwald, E., Byington, R., Cannon, C., et al. (2019). Efficacy and safety of statin therapy in older people: a meta-analysis of individual participant data from 28 randomised controlled trials. The Lancet, 393(10170), 407–415.
  • Bender et al. (2018) Bender, R., Friede, T., Koch, A., Kuss, O., Schlattmann, P., Schwarzer, G., and Skipka, G. (2018). Methods for evidence synthesis in the case of very few studies. Research Synthesis Methods, 9(3), 382–392.
  • Berkey et al. (1995) Berkey, C. S., Hoaglin, D. C., Mosteller, F., and Colditz, G. A. (1995). A random-effects regression model for meta-analysis. Statistics in Medicine, 14(4), 395–411.
  • Bliss (1952) Bliss, C. I. (1952). The statistics of bioassay, with special reference to the vitamins. Journal of the American Statistical Association, 50(272), 1368–1371.
  • Borenstein et al. (2011) Borenstein, M., Hedges, L. V., Higgins, J. P., and Rothstein, H. R. (2011). Introduction to Meta-Analysis. Hoboken: John Wiley and Sons.
  • Brockwell and Gordon (2001) Brockwell, S. E. and Gordon, I. R. (2001). A comparison of statistical methods for meta-analysis. Statistics in Medicine, 20(6), 825–840.
  • Cochran (1954) Cochran, W. G. (1954). The combination of estimates from different experiments. Biometrics, 10(1), 101–129.
  • Cornell et al. (2014) Cornell, J. E., Mulrow, C. D., Localio, R., Stack, C. B., Meibohm, A. R., Guallar, E., and Goodman, S. N. (2014). Random-effects meta-analysis of inconsistent effects: a time for change. Annals of Internal Medicine, 160(4), 267–270.
  • Davey et al. (2011) Davey, J., Turner, R. M., Clarke, M. J., and Higgins, J. P. (2011). Characteristics of meta-analyses and their component studies in the Cochrane Database of Systematic Reviews: a cross-sectional, descriptive analysis. BMC Medical Research Methodology, 11(1), 160.
  • DerSimonian and Laird (1986) DerSimonian, R. and Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7(3), 177–188.
  • Ding et al. (2018) Ding, W., Ren, K., Yue, C., Zou, J., Zuo, L., Zhang, L., Bai, Y., Okada, A., Yasui, T., and Mi, Y. (2018). Association between three genetic variants in kallikrein 3 and prostate cancer risk. Bioscience Reports, 38(6), BSR20181151.
  • Domínguez Islas and Rice (2018) Domínguez Islas, C. and Rice, K. M. (2018). Addressing the estimation of standard errors in fixed effects meta-analysis. Statistics in Medicine, 37(11), 1788–1809.
  • Gonnermann et al. (2015) Gonnermann, A., Framke, T., Großhennig, A., and Koch, A. (2015). No solution yet for combining two independent studies in the presence of heterogeneity. Statistics in Medicine, 34(16), 2476–2480.
  • Hedges and Vevea (1998) Hedges, L. V. and Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3(4), 486–504.
  • Henmi and Copas (2010) Henmi, M. and Copas, J. B. (2010). Confidence intervals for random effects meta-analysis and robustness to publication bias. Statistics in Medicine, 29(29), 2969–2983.
  • Hunter and Schmidt (2004) Hunter, J. E. and Schmidt, F. L. (2004). Methods of Meta-analysis: Correcting Error and Bias in Research Findings. New York: Sage Publications.
  • Laird and Mosteller (1990) Laird, N. M. and Mosteller, F. (1990). Some statistical methods for combining experimental results. International Journal of Technology Assessment in Health Care, 6(1), 5–30.
  • Leeflang et al. (2008) Leeflang, M. M., Deeks, J. J., Gatsonis, C., and Bossuyt, P. M. (2008). Systematic reviews of diagnostic test accuracy. Annals of Internal Medicine, 149(12), 889–897.
  • Lu and Ades (2004) Lu, G. and Ades, A. E. (2004). Combination of direct and indirect evidence in mixed treatment comparisons. Statistics in Medicine, 23(20), 3105–3124.
  • Paule and Mandel (1982) Paule, R. C. and Mandel, J. (1982). Consensus values and weighting factors. Journal of Research of the National Bureau of Standards, 87(5), 377–385.
  • Poole and Greenland (1999) Poole, C. and Greenland, S. (1999). Random-effects meta-analyses are not always conservative. American Journal of Epidemiology, 150(5), 469–475.
  • Ramsey and Schafer (2012) Ramsey, F. and Schafer, D. (2012). The Statistical Sleuth: A Course in Methods of Data Analysis. Boston: Cengage Learning.
  • Rice et al. (2018) Rice, K., Higgins, J. P., and Lumley, T. (2018). A re-evaluation of fixed effect(s) meta-analysis. Journal of the Royal Statistical Society: Series A, 181(1), 205–227.
  • Röver et al. (2015) Röver, C., Knapp, G., and Friede, T. (2015). Hartung-Knapp-Sidik-Jonkman approach and its modification for random-effects meta-analysis with few studies. BMC Medical Research Methodology, 15(1), 99.
  • Shrestha et al. (2019) Shrestha, N., Grgic, J., Wiesner, G., Parker, A., Podnar, H., Bennie, J. A., Biddle, S. J., and Pedisic, Z. (2019). Effectiveness of interventions for reducing non-occupational sedentary behaviour in adults and older adults: a systematic review and meta-analysis. British Journal of Sports Medicine, 53(19), 1206–1213.
  • Sidik and Jonkman (2010) Sidik, K. and Jonkman, J. N. (2010). A comparison of heterogeneity variance estimators in combining results of studies. Statistics in Medicine, 26(9), 1964–1981.
  • Veroniki et al. (2016) Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G., Kuss, O., Higgins, J. P., Langan, D., and Salanti, G. (2016). Methods to estimate the between-study variance and its uncertainty in meta-analysis. Research Synthesis Methods, 7(1), 55–79.