1.1 A brief history of forecasting
The ability to foretell, or divine, future events for millennia has been seen as a valued skill. While there are records of Babylonians attempting to predict weather patterns as early as 4000 BCE based on climatological observations , early attempts at divination were just as likely to be driven by unscientific observation. However, in the last 150 years, rapid technological advancements have made data-driven forecasting a reality across a number of scientific and mathematical fields. .
The science of forecasting was pushed forward especially in the second half of the 20th century by the fields of meteorology and economics, but more recently other fields have started to build on this research. Examples include world population projections [4, 5], political elections,[6, 7, 8], seismology[9, 10, 11], as well as infectious disease epidemiology [12, 13, 14, 15, 16].
Forecasting has been an active and growing area of research for over a century (Figure 1), with particular acceleration observed since 1980. While research focused on forecasting infectious diseases started in earnest in the 1990s, since 2005 the number of articles on infectious disease forecasting has increased seven-fold, at a faster pace than research on general forecasting during that time, which increased by a factor of 3. In 1991, forecasting was the topic of one of every thousand published academic papers, based on counts from the Science Citation Index and the Social Science Citation Index, obtained from the Web of Science. In 2017, over four of every thousand indexed publications were about forecasting.
1.2 What is a forecast?
In common parlance, there is not a strong distinction between the terms ‘prediction’ and ‘forecast.’ Nor does there exist a strong consensus in the biomedical, ecological, or public health literature on the distinction. Nate Silver has suggested that etymologically, the term forecast “implied planning under conditions of uncertainty” in contrast to prediction, which was a more ancient idea associated with superstition and divination . In the modern scientific world, some fields, such as seismology, use the term forecast to refer to a probabilistic statement in contrast to a prediction which is a “definitive and specific” statement about a future event. In other fields, the difference in meaning is even less clearly defined, with forecasting often connoting the prediction of a future value or feature of a time-series . We note that conventionally in biomedical research the term ‘prediction’ refers to an individual-level clinical outcome (e.g
. risk-scores give individualized predictions of heart attack or stroke risk), but both ‘prediction’ and ‘forecast’ are used interchangeably to refer to events or outcomes that may impact more than one person. The term ‘forecast’ often refers broadly to a quantitative estimate about a future trend or event observable or experienced by many individuals as opposed to than a local, individualized outcome. Our use of the term forecasting takes aspects from several of these definitions. Specifically, we define a forecast asa quantitative statement about an event, outcome, or trend that has not yet been observed, conditional on data that has been observed.
Probabilistic forecasts represent an important class of forecasts. They are forecasts that have a statement about the uncertainty surrounding an event, outcome, or trend, and are not just providing a single “best guess” estimate about the event. For example, a “point” forecast for the number of dengue hemorrhagic fever cases in Singpore in the 14th calendar week of 2024 could be simply the number 25. However, a probabilistic forecast for the same target might say that while 25 cases is the most likely specific outcome, the probability of seeing exactly 25 cases is only 0.10 or 10%.
Note that a forecasted event or outcome need not necessarily be in the future, as events in the past that have not yet been quantitatively measured may also be forecasted. For example, on May 1 we may have data for a particular time-series available through April 1. We could make a “forecast” of the time-series for April 15 even though this event is in the past. This type of forecast has been referred to as a “nowcast”,[19, 20] although more generally, this is a special case of a forecast.
1.3 Forecasting challenges that are specific to infectious disease
There are operational and statistical challenges in forecasting that are specific to the setting of infectious disease. These challenges in and of themselves may not be unique to the field, but taken together, they describe obstacles that forecasters face when taking on a problem in infectious disease.
Challenge 1: System Complexity
When attempting to forecast the transmission of an infectious disease, and in particular a vector-borne disease, researchers need to account for processes on micro and macro scales. Behaviors of and interactions between viruses, vectors, hosts, and the environment each play a part in determining the transmission of a disease. For example, it has been hypothesized that rapid changes in climate may lead to unforeseen shifts in vector population dynamics, highlighting the fragility of models that attempt to generate forecasts based on existing knowledge about these complex systems.[21, 22, 23]
Researchers have developed mechanistic models based on biological and behavioral principles that encode the processes by which diseases spread (Section 2.1.1). For vector-borne diseases, these models often include complex dynamics (see, e.g.  and ). That said, vector dynamics are often omitted from models of vector-borne disease for simplicity and tractability. While numerous mechanistic models have been fit to data to provide inference about disease transmission parameters for vector-borne diseases [25, 24], far fewer studies have examined prospective forecasting performance of such models [26, 22].
The need for models to mirror the ecological complexity of the system stands in conflict with a fundamental principle of forecasting which is to use as simple a model as necessary. Providing a good ‘fit’ to in-sample data will not guarantee that such a model will generate accurate or even reasonable forecasts. Due to the dearth of high-resolution data on vector populations and host infections, understanding whether and how such detailed biological data can improve the accuracy of forecasts of population-level transmission largely remains to be seen (Section 2.3).
Challenge 2: Data Sparsity
A central challenge in forecasting vector-borne disease, and infectious diseases more generally, is how to balance the complexity of the biological and social models used with the coarseness of available data . For forecasting weather, scientists rely on tens of thousands of sensors across the world, collecting continuous real-time data. There are no analogues to these rich and highly accurate data streams for infectious disease researchers. The gold-standard data in epidemiological surveillance arise from systems that typically capture only a fraction of all cases and often are reported with substantial delays and/or revisions to existing data. Technology shows some promise to provide more richly detailed data in a timely fashion about humans, climate, and vectors alike. However, new methods and data streams will be required to develop and implement increasingly refined forecasting models for vector-borne diseases.
Challenge 3: The Forecasting Feedback Loop
Forecasts of disease incidence can encourage governments and public health organizations to intervene to slow transmission. If forecasts of infectious disease are used to inform targeted interventions or risk communication strategies and the interventions change the course of the epidemic, then the forecast itself becomes enmeshed in the causal pathway of an outbreak. This feedback loop has been identified as the single most important challenge separating infectious disease forecasting from forecasting natural phenomena such as weather .
In settings where forecasts will be used to inform interventions, this feedback loop of infectious disease forecasting should be taken into account in the forecasts. Without such accounting, if a forecast predicts an outbreak and triggers an intervention that prevents the epidemic from occurring, then the forecast itself would be seen as wrong, despite this being a public health victory. This implies that forecasting models should, when in these settings, create multiple forecasts under different intervention scenarios. Mechanistic forecasting models, that use explicit disease transmission parameters, may be best suited for these types of forecasts, since intervention effects could be incorporated directly as impacting these parameters. However, any forecasts from such a model should be subjected to intense scrutiny, since it will necessarily be based on very strong assumptions about the intervention and transmission patterns.
Methodological development is needed in this area to address open scientific questions. What model frameworks can best balance forecast accuracy with the ability to incorporate multiple potential future scenarios? Can forecast models be used to assess intervention effectiveness?
1.4 Definitions and basic notation
Here, we introduce some basic mathematical notation for time-series forecasting that we use throughout this chapter. In many forecasting applications, the available data are often a time series of observed values for a particular location or setting. For infectious disease applications, these data are often a measure of incidence, such as case counts or the percentage of all doctor visits with primary complaint about a particular disease. In the text that follows, we use language specific to that of spatio-temporal disease incidence data, although much of what we describe can be applied more generally as well.
We start with a simple example and later extend the notation to more realistic scenarios. In our example, we have a complete (i.e. no missing data) time series of infectious disease case counts from a single location, such as a school or hospital. We define as an observed value of this incidence in time interval from our time series
. We assume that these observations are draws from random variables
, where the probability distribution ofmay be dependent on , prior values of represented as , and a matrix of other covariates . (Often, the analyst may wish to include multiple different lagged values of a single covariate. In this notation, for simplicity, these would all be considered to be part of .) We use throughout to refer to the total number of time points in the time-series and to refer to a specific time point relative to which a forecast is generated.
Two important features of our observed data are frequency and scale. In our example, incidence is recorded at regular time intervals. Furthermore, many infectious disease time series have a cyclical element. We define the frequency of a time series as the number of observations within a single cycle. For example, if we have monthly incidence data and know that there are annual weather patterns that influence incidence in our observed data, the frequency of our time series would be 12 months or observations.
Targets are the as-yet-unknown features of the data that are the subject of forecasting. In our example, we may want to forecast incidence at a certain future time—but targets can be a variety of endpoints extrapolated from the observed data. For forecasts of the time-series values itself, i.e. when a target is defined to be a past or future value of the time-series , we use a special nomenclature, referring to them as ‘k-step-ahead‘ forecasts. In general, we define as a random variable for target positioned relative to time . For example, in the infectious disease context, could be:
incidence at time , or ,
incidence at time either in the future or past relative to time , or , where is a positive or negative integer,
peak incidence within some period of time or season, or where are defined to be within season ,
the time(s) at which a peak occurs within some season, or
a binary indicator of whether incidence at time is above a specified threshold , or .
A forecast, as defined in Section 1.2, must provide quantitative information about an outcome. In the context of this notation, a probabilistic forecast can be represented as a predictive density function for a target, or . The form of this density function depends on the type of variable that is, and it could be derived from a known parametric distribution or specified directly. For example, if the target is a binary outcome (e.g
. whether in week 4, the observed incidence will be above 10 cases) the density could be specified as a Bernoulli distribution with a parameter associated with the probability of the outcome occurring. It could also be specified directly as a probability that the incidence is >10 and the probability that the incidence is. For an integer-valued target (e.g
. the number of new cases occurring in February), the predictive density could be represented by a Poisson distribution with a given mean or as a vector of probabilities associated with all possible integer values of cases.
To enable clear definitions for forecasting in real-time, forecasts must be associated with a specific time . This time represents the point relative to which targets are defined. For example, if a forecast is associated with week 45 in 2013, then a ‘-1-step-ahead’ (read ‘minus-one-step-ahead’) forecast would be associated with incidence in week 44 of 2013 and a ‘3-step-ahead’ forecast would be associated with week 48.
Another consideration for infectious disease forecasting is the forecast horizon, the temporal range that the forecast predicts [29, 30]. Regardless of the model type, many recent infectious disease forecasting efforts have focused on short time scales (weeks or months) [31, 32, 33, 34, 35, 36, 37, 16, 38, 39, 40, 41]. These studies demonstrated the importance of recent case counts and seasonality on the immediate trajectory of infectious disease incidence. In 2015, the National Oceanic and Atmospheric Administration (NOAA) and the Centers for Disease Control and Prevention (CDC) hosted a competition to make within-season forecasts for longer forecast horizons, such as annual dengue incidence, epidemic peak, and peak height, for San Juan, Puerto Rico and Iquitos, Peru . Prior to these competitions, long-term forecasts were more commonly used for chronic disease prevalence than for non-chronic infectious disease incidence .
2 Models used for forecasting infectious diseases
2.1 Mechanistic vs. Statistical: a taxonomy of forecasting models
According to Myers et al., forecasting models for infectious diseases take either a ‘biological approach’ or a ‘statistical approach’ . Others have phrased this distinction as one of mechanistic (i.e. biological) and phenomenological (i.e. statistical) models. A model based on disease biology can account for previously unforeseen scenarios that are possible due to transmission dynamics, however these models often require specification of a large number of parameters and covariates in order to make forecasts. On the other hand, statistical forecasting models are restricted by the assumption that future incidence will follow the patterns of incidence observed in the past, but can be specified without full knowledge of the disease process or interactions between members of the population. In this section, we discuss the major modeling methods across the biological-statistical spectrum (Table 1).
|Method||SIR model||Vector modelled?||Disease|
|EAKF||SI||Yes||West Nile |
|Ordinary diff. eqs.||SEIR||Yes||dengue , malaria |
|Ross-MacDonald NN||SIR||Yes||dengue |
|Ross-MacDonald ODE||SIR||Yes||dengue [45, 46]|
|Stochastic diff. eqs.||SI||Yes||Theileria orientalis |
|ARIMA||-||No||dengue , malaria [48, 49]|
||Cellular automaton||SIR||No||dengue |
|GLM/Regression||-||No||dengue [51, 34, 13, 16], malaria , other [53, 54, 55]|
|Neural networks||-||No||malaria , Zika |
|Markov chain||SEIR||-||Ebola |
|Stochastic diff. eqs.||SEIR||-||Ebola [61, 62]|
|Agent-based||SEIR||-||influenza [63, 64]|
|Chain binomial||SEIR||-||influenza |
|Dynamic Bayesian||SIR||-||influenza [65, 66]|
|EAKF||SIRS||-||influenza [67, 68]|
|Expert opinion||-||-||influenza |
|GLM/Regression||-||-||influenza [70, 71]|
|Random forest||-||-||influenza |
|ARIMA||-||-||influenza , other [41, 36]|
|Neural network||-||-||influenza , hepatitis A |
2.1.1 Mechanistic models
Compartmental models are the standard biological, or mechanistic, approach for modeling infectious disease [77, 78, 79]. Kermack and McKendrick proposed one such model, now known as the susceptible-infectious-recovered (SIR) model, in which members of a population transition through each compartment (susceptible to infectious to recovered) over the course of an epidemic . While this process mimics the behavior of an outbreak, the simplest model assumes that the population is “well mixed”, such that each individual is equally likely to encounter any other individual. Since this is unlikely, researchers can add more compartments (e.g. adults and children), along with contact rates between compartments, or individually model each member of the population (i.e. agent-based modeling) . Though greater complexity requires more modeling assumptions, compartmental models have been effective at estimating underlying infectious disease processes and the potential impact of interventions [79, 82, 77, 83].
Compartmental models have been extended for use in forecasting infectious disease incidence. Variations on the SIR model have been used to develop forecast models for influenza.[31, 37, 65] Some of these approaches incorporate humidity into a SIRS compartmental model.[67, 38] The Ross-MacDonald model, a compartmental model originally developed for malaria that accounts for interactions between mosquitoes and humans, has been used to make forecasts of dengue fever.[44, 45, 46] However, other mechanistic approaches to forecast dengue outbreaks have not modeled mosquitoes specifically.
2.1.2 Classical statistical models
On the statistical side of the modeling spectrum, many regression-style methods have been used for forecasting. Perhaps the most well-known statistical method for time series is the auto-regressive integrated moving average, or ARIMA.
ARIMA models use a linear, regression-type equation in which the predictors are lags of the dependent variable and/or lags of the forecast errors. ARIMA and seasonal ARIMA (SARIMA) models are frequently applied to infectious disease time series[47, 78, 30, 86, 57]. Lu et al. combined a SARIMA model with a Markov switching model (a type of compartmental model) to account for anomalies in the surveillance process .
Also under the subheading of trend and seasonal estimation are simple exponential smoothing strategies, known as Holt-Winters models [87, 88]. Exponential smoothing techniques involve taking weighted averages of past observations with exponentially decreasing weights further from the present. Holt-Winters in particular is known for its efficient and accurate predictive ability [89, 90]. These approaches have been used successfully in forecasting dengue fever  and leprosy .
Some researchers have used generalized linear regression models to develop infectious disease forecasts. In some cases, researchers used lagged covariates (e.g. temperature, rainfall, or prior incidence) to predict future incidence [53, 34, 35, 55, 16]. Held and Paul also combined statistical and biological theory by building a regression model that consisted of three components of disease incidence: endemic, epidemic, and spatio-temporal epidemic (to account for spread of disease across locations) . This has become a well-established framework for forecasting infectious disease surveillance data [92, 15, 57]
, and is accompanied by open-source software implementing the methods.
2.1.3 Modern statistical methods
Modern statistical methods, i.e
. not the classical time-series and regression-based approaches, are an increasingly popular way to forecast infectious disease incidence. These methods include non-parametric approaches as well as more black-box machine-learning style algorithms. We focus in this section on stand-alone forecasting methods, for a discussion on ensemble methods, see Section2.4.
Statistical or machine learning approaches have been in existence for decades. While machine-learning and statistical methods are sometimes classified separately, we group them together as “statistical”, as both terms encapsulate approaches that use patterns from past incidence in order to forecast future incidence.
These approaches can be used for ‘data mining’, by which large amounts of data are extracted from various online sources for pattern-recognition tasks, or for modeling, using empirical methods such as random forests, neural networks, or or support vector machines that do not make any parametric model assumptions. These techniques came about in the computer science and artificial intelligence communities (see, e.g.), but can also be expressed statistically .
Several papers have found that machine-learning modeling methods can outperform standard statistical models for infectious disease forecasting: random forests outperformed ARIMA forecasting avian influenza 
, a maximum entropy model outperformed logistic regression forecasting hemorrhagic fever with renal syndrome, and fuzzy association rule mining outperformed logistic regression forecasting dengue outbreaks . Additionally, kernel conditional density estimation, a semi-parametric method, was shown to have more well-calibrated probabilistic forecasts than SARIMA and other regression-based approaches for forecasting highly variable dengue outbreaks in San Juan, Puerto Rico . Heteroskedastic Gaussian processes showed better forecast accuracy than generalized linear models in forecasting dengue fever. Neural networks have also been used for forecasting influenza [74, 97], Zika , and Hepatitis A .
2.1.4 Comparisons between mechanistic and statistical models
From an epidemiological perspective, mechanistic models have several clear advantages over statistical models. They are biologically motivated, and therefore have parameters that relate to well-established theory and can be interpreted by experts in the field. They have been adapted in previous work to include vector dynamics, which could be important for forecasting outbreaks of vector-borne diseases.[25, 24, 22] Mechanistic models can flexibly incorporate features such as interventions or behavioral changes, which can be critical, especially if forecasts are desired for different intervention scenarios (see Section 1.3). While mechanistic models can be built to rely heavily on previously observed data, they also can be instantiated with very little prior data, such as in emerging outbreaks (see Section 2.2). Additionally, while forecasts from statistical models are typically bounded by trends that have been previously observed, mechanistic models can forecast outside of previously observed trends if the underlying states of the model call for such dynamics.
Despite these advantages, in forecasting settings where substantial historical data is available, statistical models may prove more effective at using past observed trends to forecast the future. Many statistical models were designed to be either more flexibly or parsimoniously parameterized, meaning that they may be able to more easily capture dynamics common to infectious disease time-series such as auto-regressive correlation and seasonality. Additionally, they can be built to rely less heavily on specific assumptions about a particular biological disease transmission model, giving them flexibility to adapt when the data does not follow a particular pattern. In other words, since any specified mechanistic model is necessarily a simplification of the true underlying disease process, the question is how much will forecast accuracy suffer as a result of the inevitable model misspecification. In many cases, heavily parameterized mechanistic models may be more sensitive to model misspecification than a more flexible statistical model.
In practice, however, the distinction between statistical and mechanistic models is not always sharply defined. Due to complexities of real-time surveillance systems, forecasting targets of public health interest often represent a mixture of different signals that would be challenging in practice to be forecasted accurately by a single mechanistic model. For example, surveillance data of case counts may include distorted signals due to figments of partially automated reporting systems. This might be manifested in decreased case counts during holidays, or an increase in case counts at the end of the season when final reports are being prepared. In other settings, the actual target of interest may be a composite measure, such as with influenza-like illness (ILI) in the US. In both of these settings, the signal which is being predicted may be driven by factors that are not directly relevant to trends in disease transmission (e.g. clinical visitation or reporting patterns, changes in the processes used for diagnosis or case reporting). In these settings, statistical models that can have a more flexible understanding of the trends they are using to fit data may be at an advantage over mechanistic models. Research has demonstrated the value of coupling flexible statistical formulations with mechanistic curve-fitting to produce accurate forecasts.[66, 68, 61, 62]
Despite many unanswered questions about when and in what settings one type of model will generally do better than the other, studies that make explicit, data-driven comparisons are fairly uncommon. Multi-team infectious disease forecasting challenges provide some of the best data available on this important question. For forecasting both seasonal influenza and dengue outbreaks, what limited data there are suggest that mechanistic and statistical approaches show fairly similar performance, with statistical models showing a slight advantage.[20, 98, 99] A collaborative effort during the 2014 west Africa Ebola outbreak to forecast synthetic data showed fairly comparable results from mechanistic and statistical models and did not make an explicit comparison between the two. Summary analyses from other challenges have not been published, but we note that a very simple quasi-mechanistic model was the best performing model in forecasting the pattern of emergence of Chikunguya in the Americas [100, 101]. In summary, more research is needed to improve our understanding about whether different types of forecasting methods can be shown to be more reliable than others, especially as data availability and resolution improves over time.
2.2 Forecasting in emergent settings
In emerging outbreak scenarios, where limited data is available, mechanistic models may be able to take advantage of assumptions about the underlying transmission process, enabling rudimentary forecasts even with minimal data. On the other hand, many statistical models without assuming a mechanistic structure rely on past data to be able to make forecasts. That said, any forecasts made in settings with limited data must be subjected to rigorous sensitivity analyses, as such forecasts will necessarily be heavily reliant on model assumptions.
A wide range of different mechanistic models have been used in settings where infectious disease forecasts are desired for an emerging threat. A simple non-linear growth model performed the best in a prospective challenge for forecasting Chikungunya in the Americas . This growth model made the assumption that the rate of change in the cumulative number of cases through the course of an outbreak follows a parabolic rise and fall, by which it can quickly approximate the parameters of an SIR model and estimate the peak incidence, duration, and total number of cases during an epidemic. Unlike many other mechanistic forecasting approaches, this model has a small number of parameters, is easy to fit, and makes only a few assumptions about the underlying disease process. A deterministic SEIR model was used to forecast synthetic Ebola epidemic data, showing comparable performance to other methods on the same data [60, 14]. A stochastic SEIR model also forecasted synthetic Ebola data, and showed somewhat less reliable performance compared to other methods . Data-driven agent-based models have also been shown to be a viable forecasting tool for emerging infectious diseases . An SIR model, similar to one used to forecast seasonal dengue fever and influenza, was used with a more complex compartmental structure to forecast the spread of Ebola during the outbreak in West Africa in 2014 . A set of quasi-mechanistic models were used to forecast Zika virus transmission during 2017 to help plan for vaccine trials .
2.3 Using external data sources to inform forecasts
Traditional approaches to infectious disease forecasting often have relied on a single time-series, or multiple similar time-series (e.g. incidence from multiple locations). However, other types of epidemiological data may provide important information about current transmission patterns.
The use of data on prevalence and abundance of disease vectors in forecasting models has not been extensively explored in the literature. This is likely due in part to climate data being more widely available and the belief that such data could serve as a good proxy for actual data on vectors. That said, perhaps the most well-developed area for incorporating vector data into forecasts is the use of prevalence data in forecasting mosquito-borne diseases such as West Nile virus[104, 105, 26], and dengue fever. However, none of these studies have explicitly quantified the added value of vector data on forecast accuracy. While the hypothetical benefits of good vector surveillance data have been clearly quantified, the benefit of these data in practice (especially when other reasonable proxy data may be available; see “Climate data” below) is still unclear.
Leveraging laboratory data, collected either through passive or active surveillance strategies, may provide crucial data about what specific pathogens are currently being transmitted and could inform forecasting efforts. This is an area that warrants more research, as few efforts have tackled the challenge of having laboratory test data inform forecasts at the population level. One model uses an aggregate measure of genetic distance of circulating influenza strains from the strains in the vaccine as a variable to help forecast peak timing and intensity of seasonal outbreaks in the US [108, 109]. Some efforts have also been made to make strain-specific forecasts for influenza . Other efforts have focused on longer-term forecasts of what strains will predominate in a given season, with an eye towards providing information to influenza vaccine manufacturers . These efforts have moved beyond influenza, and forecasting pathogen evolution is being worked on for a variety of different pathogens .
Another, and very different, kind of epidemiological data for forecasting is expert opinion. Long seen as a useful indicator in business applications , expert opinion has recently begun to be used in infectious disease applications [69, 76]. While not traditional clinical data, expert opinion surveys leverage powerful computers, i.e. human brains, that can synthesize historical experience with real-time data. Intuitive interfaces can facilitate the specification of quantitative and digitally entered forecasts from experts who need not be technically savvy, lowering the barriers to participation and subsequent analysis . In the 2016/2017 influenza season in the US, a forecast model based on expert opinion was a top-performer in a CDC-led forecasting competition.[98, 20] Human judgment and expert opinion surveys are a promising area for further forecasting research, especially in contexts with limited data availability.
Digital epidemiology has been defined as the use of digital data for epidemiology when the data were “not generated with the primary purpose of doing epidemiology” . Broadly speaking, this might refer to online search query data, social media data, satellite imagery, or climate data, to name a few. These resources may hold promise for forecasters who want to incorporate “Big Data” streams into their models. In the past 10 years, much research has explored the potential for leveraging multiple data streams to improve forecasting efforts, but this practice is still in its nascent stages. So far, the utility of digital epidemiological data for forecasting has been somewhat limited, perhaps due to challenges in our understanding of how digital data generated by human behavior and interactions with the digital world relate to epidemiological targets [27, 116, 115].
Perhaps the most famous and controversial example of using digital data streams to support infectious disease prediction surround the early promising performance[117, 118] and later dismal failure  of Google Flu trends to predict the influenza-like-illness in the US. Google Flu trends was based on tracking influenza-related search terms entered into the search engine. Although Google eventually discontinued the public face of the project due to poor performance, criticism of the Google Flu trends approach centered around how data was included or excluded, interpreted, and handled rather than the algorithm that produced the actual forecasts [120, 121]. Ongoing research on using search engine data in forecasting has continued despite the failure of Google Flu trends, producing incremental but consistent improvements to forecast accuracy [122, 123, 124, 125, 126]. More focused search query data, such as data from clinician queries, has also been shown promise for assisting real-time forecasting efforts.[127, 128]
The use of climate data for epidemic forecasting serves as another clear example of re-purposing data for epidemiology. While climate factors are known biological drivers of infection risk (e.g. the impact of absolute humidity on influenza virus fitness , or temperature and humidity providing optimal conditions for mosquito breeding), the evidence supporting the use of climate data in forecasting models is mixed. Climatological factors such as temperature, rainfall, and relative humidity were used to forecast annual counts of dengue hemorrhagic fever in provinces of Thailand . However, only temperature and rainfall were included after a rigorous covariate selection process and neither were included in the final model, although subanalyses showed variation in these associations across different geographic regions of Thailand. Climate factors were shown to improve forecasts of dengue outbreak timing in Brazil, but played a less influential role in dengue forecasts in Mexico . Aggregated measures of absolute humidity have been incorporated into influenza forecasts in the US [38, 122]. However, without clear standardization across these studies, these mixed results may reflect heterogeneity in the spatial and temporal scales at which forecasts are made, climate factors are measured and aggregated, and disease transmission actually occurs.
2.4 Forecasting with ensembles
Ensemble forecasting models, or models that combine multiple forecasts into a single forecast, have been the industry standard in weather forecasting and a wide array of other prediction-focused fields for decades. By fusing together different modeling frameworks, ensembles that have a diverse library of models to choose from end up incorporating information from multiple different perspectives and sources. When appropriate methods are used to combine either point or probabilistic forecasts, the resulting ensemble should in theory always have better long-run performance than any single model.[131, 132, 133] However, researchers have suggested that adjustments are necessary to correct for the introduction of bias and miscalibration in the process of building ensemble models.
Ensembles have been increasingly used in infectious disease applications and have shown promising results. For forecasting influenza, several model averaging approaches have shown improved performance over individual models [136, 137, 138]. Similar approaches have yielded similar results for dengue fever , lymphatic filariasis , and Ebola .
In many of these examples, however, the number and diversity of distinct modeling approaches was fairly small. To unlock the full potential value of ensemble forecasting, as well as understanding the added value of contributions from new and different data sources or modeling strategies, more scalable frameworks for building forecast models are required. There is a need to develop infrastructure and frameworks that can facilitate the building of ensemble forecast models. This will require clear technical definitions of modeling and forecasting standards. Given the history of improved forecast performance due to improvements in ensemble methodology in other fields that focus on non-temporal data, the prospects for continued ensemble development in the area of temporal and time-series data, and especially infectious disease settings, are bright.
3 Components of a Forecasting System
Due to the complex biological, social, and environmental mechanisms underlying infectious disease transmission, the true underlying processes that give rise to the observed data is unknown. This makes choosing one model, or even multiple, from the selection of possible choices quite challenging. How can one decide which model is the best for forecasting future targets?
Deciding on the structure of the forecasting exercise, including the targets and the evaluation metrics, provides critical information to help decide upon appropriate methods. Because models perform differently depending on the forecast target, type of forecast, model training technique, and evaluation metric, it is important to specify the forecasting system prior to fitting the models. This ensures that the system is tailored to the particular design of the exercise.
3.1 Forecast type
When building a forecasting system, the first steps are to choose the forecast target (as described in Section 1.4) and type. Forecast targets are often dictated by the goals of a public health initiative. Researchers and public health officials collaborate to find a forecast target that is most useful for allocating resources and implementing interventions to reduce the severity of an infectious disease outbreak.
The forecast target helps inform the selection of the forecast type. Forecasts can typically be classified as either point forecasts or probabilistic forecasts. Some authors classify “interval” forecasts as a separate entity, however, these can be seen as a simplified version of a fully probabilistic forecast.
A point forecast is a forecast of a single value that attempts to minimize a function of the error between that value and the eventually observed value. The mean, median, or mode of a predictive distribution is often used as the point forecast for a specified target. This choice may depend on the specific metric used for evaluation, as the mean is the theoretically optimal choice if point forecasts are being evaluated with mean squared error (MSE) and the median is optimal if being evaluated by mean absolute error (MAE). While point forecasts are simpler to produce and interpret, they may make simplifying assumptions about the underlying probability distribution, leading to low-quality forecasts. For example, a point forecast based on the mean may represent a value for which there is actually a small likelihood of occurring if, for example, it lies between the peaks of a multi-modal distribution. This could mislead officials and researchers into forecasting a medium-sized outbreak when the full distribution actually shows that the most likely future scenarios are for either low incidence or an epidemic outbreak.
Interval forecasts supplement point forecasts with a “prediction interval”, or a range of likely values. The nominal level of a prediction interval indicates the percentage of eventually-observed outcomes that should fall within that interval. If a model makes 100 forecasts, about 95 should fall within the 95% prediction interval. More generally, a ()% prediction interval can be thought of as the interval that has a significance level of . Interval forecasts are typically derived from some form of a probabilistic model or assessment of in-sample forecast error or uncertainty.
A fully probabilistic forecast must specify a probability distribution function The goal of a probabilistic forecast is to assign the maximum probability to the true future value. Probabilistic forecasts can specify a closed-form parametric density function (e.g., a Gaussian distribution with a mean and variance) or an empirical distribution, either with an empirical cumulative density function, a set of samples from the predictive density, or a binned density function, with probabilities assigned to a discrete set of possible outcomes. Density estimation often requires simulation-generating methodology, which can be more time-consuming and computationally-intensive than other techniques. Ongoing advances in computing continue to make density forecasting methods more feasible for researchers. Density forecasts contain the most nuanced information of all of the forecasting methods, but are often the most difficult to interpret and communicate to non-expert collaborators.
3.2 Evaluation and scoring
There is a rich literature on scoring and evaluating all types of forecasts. Of course appropriate metrics will depend on the forecasting setting and the scoring criteria for a particular exercise. In general, models should be fit with a loss function or “goodness of fit” criteria that that is similar or identical to the method that will be used to evaluate forecasts.
General principles for scoring forecasts
Research suggests that metrics should be scale-independent.[140, 142] For example, within a single infectious disease time series, larger incidence values are both more difficult to forecast and often have larger errors on an absolute scale than smaller incidence values simply because they are larger numbers. Thus, incidence values near the seasonal peak are both larger and more variable than incidence near the seasonal nadir and, consequently, forecasting model error will depend on the size of the value it is forecasting. In these situations, something closer to scale-independence can be achieved either by using logged metrics can weight errors more equally across different scales or by using relative measures of accuracy (see, e.g., equation 1 below).
Metrics should be defined and finite in reasonable scenarios. This principle ensures that scores from single forecasts may be combined together, for example with an average. If single forecasts could be infite in reasonable scenarios, one individual forecast could eclipse all other scores in a summary measure such as an average. However, even non-experts can agree that a model that forecasts negative values of disease incidence should be considered invalid and it could be appropriate to have an infinite scoring value for such a forecast.
Forecasts should be evaluated using proper scoring rules. A proper scoring rule incentivizes the forecaster to report exactly what the model predicts. An improper scoring rule could incentivize adjusting the forecast reported by a model to “game” the system only to get a better score. While using an improper scoring rule is unlikely to change the relative performance of a set of models dramatically, it can lead to settings where the best scoring forecast is one that has been modified in undesirable ways. For example, when probabilistic forecasts for season peak week are scored by evaluating the probability assigned to the true peak week plus or minus one week (an improper score used by the US CDC in influenza forecasting competitions[98, 20]), even high-scoring forecast models can be adjusted to have better scores by adjusting the probabilities assigned to different weeks in a systematic way that is ‘dishonest’ to the original forecast.
Forecast accuracy should always be evaluated with out-of-sample observations and with as large of a sample of observations as is feasible. Since forecasts are by definition predictions of as-yet-unobserved data, it is critical to evaluate forecast accuracy on out-of-sample data. This means that forecasts should only be evaluated on observations that were not used to fit or train the model. Ideally, the out-of-sample data would be “prospective” in the sense that all of evaluated observations would be from a point in time after the training data. However, in cases with limited data availability, this may not be feasible. Data quantity can be a limiting factor for many real-world datasets, especially for infectious disease surveillance. The analyst must balance competing needs of having sufficient data for training a realistic model with holding out data (ideally prospectively) for cross-validation and testing.
3.2.1 Evaluating point forecasts
Point forecasts are typically evaluated on their own using metrics such as mean squared error (MSE) or the mean absolute error (MAE). For comparative evaluation of point forecasts in practice, many researchers recommend using a metric that scales the forecasting error against that of a reference model [142, 141, 143].
One example is the relative mean absolute error (rMAE), which divides the mean absolute error of one forecasting model (model A) by the mean absolute error of a second model (model B):
Recall that is the target of interest forecasted at time and represents the forecast from model A at time . In principle rMAE may be calculated between any two models, however, it is common for rMAE to be calculated for a set of models against a common ‘reference’ or basline model in the denominator.
An additional desirable feature of rMAE is that it is interpretable for public health officials. When this means that the forecasting model has less error than the reference model on the scale of the original data and means that the forecasting model has more error than the reference model. For example, if model A has an rMAE of 0.9 compared to a reference seasonal model of case counts for a particular disease that means that the predictions from model A were 10% closer to the observed value than predictions from the reference model were.
3.2.2 Evaluating interval forecasts
Interval forecasts can be evaluated by their coverage rate and their width. Prediction intervals should be as narrow as possible while covering a proportion of forecasts approximately equal to that expected by its level.
Perhaps the most commonly used interval metric is the coverage rate (CR) which is simply the fraction of all % prediction intervals that cover the true value. Therefore
where is the indicator function (equalling 0 if the expression inside is FALSE and 1 if TRUE), and and are the lower and upper bounds of a % prediction interval for observation . The observed can be evaluated for its proximity to .
A interval evaluation metric recommended for it’s being a ‘proper’ scoring metric is
where it is desirable to minimize the score . Forecasting models are penalized for having wider intervals and for having observed values that fall far outside of the intervals. Observations that fall outside of large prediction intervals (small ) are penalized more than those that fall outside of small prediction intervals (large ).
3.2.3 Evaluating probabilistic forecasts
In the long run, probabilistic forecasts should have distributions that are consistent with the distribution of the observed values. Models that assign more weight to the eventually observed values should be scored better than those that do not. A commonly used proper scoring rule for probabilistic forecasts is the log score. Aggregated across many predictions, this metric is defined as
where is the estimated probability of observing the target
. However, this metric is sensitive to outliers, as any observation with a forecasted probability of zero causes the metric to go to negative infinity (though adjustments can be made to avoid this). As an alternative, Funket al. recommend using multiple metrics to evaluate the unbiasedness, calibration, and sharpness of infectious disease forecasts .
The continuous ranked probability score (CRPS) is a proper scoring rule that measures the difference between the forecasted and observed cumulative distributions . This metric measures both the bias and the uncertainty of the forecasted density and thus rewards forecasts that assign weight closer to the observed value, even if it doesn’t assign much weight exactly on the observed value. A point forecast with no uncertainty will have a CRPS equal to the absolute error of the forecast. Unbiased forecasts with more uncertainty will have a higher CRPS than for unbiased forecasts with less uncertainty, however biased forecasts with more uncertainty can have a smaller CRPS than biased forecasts with less uncertainty. While CRPS is scale-dependent, dividing the CRPS of a forecasting model by the MAE of a benchmark model (as in the relative mean absolute error) yields a scale-independent continuous ranked probability skill score [147, 148].
3.2.4 Tests for comparing models
Comparing the relative performance of different models can yield important insights about the benefits of a unique data source or one approach over another. However, in a time-series forecasting context (the most common for infectious disease forecasts) several clear statistical challenges are present when making forecast comparisons. Most importantly, outcomes and forecasts at multiple nearby timepoints will be correlated with each other, reducing and complicating the understanding of the power of these tests to detect “significant” differences. The Diebold-Mariano test is the most well-known test to compare the errors from two forecasting models. This method is implemented in software packages, for example, the forecast package in R.[150, 151] Other permutation based approaches have also been used to compare the significance between the forecast errors for two models.
However, in the infectious disease forecasting literature it has not yet become common practice to run such tests. Instead, authors have tended to rely on simple numeric comparisons of forecast errors between two or more models. Not running a formal test allows for the possibility that the observed differences are due to chance. However, from a practical perspective, as long as the forecasts are truly prospective in nature and the comparisons presented were the only ones made, such a comparison can provide tangible information about which model to choose for decision-making. In situations where a definitive statement about the predominance of one model over another is desired, a formal test will likely be the best evidence available.
3.3 Model training and testing
In order for a forecasting model to be useful for researchers or officials it needs to be generalizable to data beyond the observations that were used for fitting. For instance, a model that perfectly forecasts monthly dengue incidence over the past ten years, but performs worse than a reasonable guess—e.g. the average monthly incidence—over the next five years is not very useful. We would be better off using the reasonable guess instead of the forecasting model. Though we can never be certain that our best model will perform well outside of our dataset, we can get a better idea of its out-of-sample performance using cross-validation with a training and testing set. We illustrate this concept with an example from Lauer et al., in which we forecasted annual dengue hemorrhagic fever (DHF) incidence in Thailand for 76 provinces.
A central challenge for forecasting in general is to train a model in such a way that we minimize the error on the unobserved data, i.e., the test data. For real-time forecasts, the test data will be unobserved at the time a model is specified. When forecasting is performed retrospectively for data that has already been collected, the test data will already have been observed. Strictly speaking, such an experiment is not forecasting at all, as it does not involve making predictions about data that have not yet been observed. Nonetheless, it can be an important part of forecasting research to understand model performance in these settings. To ensure the validity and generalizability of such findings, it is critical to only use the test data once the models have been specified from the training phase.
Typically, forecasters use cross-validation methods to evaluate and estimate error on not-yet-seen observations. There is a rich literature on cross-validation methods, including some techniques specific to time-series applications. These methods tend to reward slightly more complex models that may have more error on the testing data than a smaller or simpler model would. Thus, in addition to selecting the model that performs best in the training phase by a pre-specified information criterion or cross-validation metric, forecasters should also choose a more parsimonious model that has more error in the training phase as a check against overfitting.
Prior to fitting any model, we split our data into a ‘training’ sample (for initial model selection) and a ‘testing’ sample (for final model evaluation) [155, 95]. These steps are standard practice in the field, and similar to formal recommendations for modeling disease surveillance data. In this example, data from years 2000 through 2009 (760 observations) served as the training phase data and years 2010 through 2014 (380 observations) served as the test phase data (Figure 2a). The training sample is used for model experimentation and parameter tuning.
There is no one right answer for how to split data into training and testing sets, however the choice may be informed by prior knowledge about the modeling setting. We chose to model the training phase data using leave-one-year-out cross-validation, so that each year’s training forecast would be conditional on the remaining 9 years of data. While this does not preserve strict ordering of data (e.g., the data from 2000 is predicted based on a model fit to data from 2001 through 2009), it ensures that each of the training period forecasts is based on the same amount of data. The alternative would have been to implement a training regimen that would have predicted 2001 based only on 2000 data, 2002 based only on 2000-2001 data. Due to the limited length of this dataset, this would mean that early forecasts would be based on substantially less data. Using leave-one-year-out cross-validation ensures that each of the 10 years of training forecasts will have the same amount of data, and a roughly similar amount of data that we expect to have in the test phase. However, if substantially more data were available prior to 2000 (say, more than 5 years of data) then it might have been desirable to implement prospective cross-validation in the training phase as well.
The training period is complete once all candidate models have generated out-of-sample forecasts for each of the training years. Typically, a small number of models are selected to pass into the test phase. In our example, we ran leave-one-year-out cross validation on the training phase data to select our model. In this procedure, we fit a model on 9 of the 10 years to predict the final year — e.g. fitting on 2001-2009 to predict 2000. We repeated this to predict the province-level DHF incidence in each of the 10 years, recorded the error for each prediction, and then took the mean absolute error across all predictions and called it the “cross-validation (CV) error” for a given model (Figure 2b). We performed cross-validation for 202 models with different specifications and covariate combinations. The model that minimized the CV error had 5 covariates, while the model that minimized the in-sample residual error across the entire training phase had 14 covariates (Figure 3
). In addition to the 5-covariate model, we also selected the smallest model within one standard deviation of the smallest CV error — in this case it was a univariate model — to forecast the test phase.
In the test phase, we implemented a prospective testing schema to more realistically simulate real-time forecasts (Figure 2c). This rolling-origin-recalibration window, as it has been called, is implemented by first fitting the model to the training data to forecast the first test phase observation. Then the first observation from the test phase is moved into the training data, the model is re-fit and the second test phase observation is forecasted. We used this method in the testing phase of our example as it is good for evaluating how a model might perform in real-time, as more data is collected and assimilated into the model fitting process. In our example, we evaluated the testing phase forecasts using relative mean absolute error (rMAE) of our model over baseline forecasts based on the ten-year median incidence rate for each province (Figure 2d). The univariate model had less testing phase error than the best cross-validation model (Figure 3). Additionally, the univariate model had about 20% less error than a baseline forecast, a rolling 10-year-median for each province (data not shown).
When a forecaster is interpreting the results, she should recommend for future use the model that performed best in the test phase. The goal at the outset of this exercise was to find the model that makes the best forecasts that are generalizable outside of our data, which is defined by the performance on the testing phase. Specific times or places where a different model showed good results could be areas for future forecasting activities, however analysts should cautious about over-interpreting small-sample size results in training or test phase results.
Prior to splitting the data into training and testing, it is critical to think about how many observations are needed for each sample. There need to be enough training observations to properly fit the model and there need to be enough testing observations to properly evaluate the model. With a short time-series, there may be too few data to split and thus only cross validation can be conducted on all of the data; in this scenario, interpretations about the model performance will be weaker than those with a separate testing phase.
4 Operationalizing forecasts for public health
Making forecasts on infectious disease data is difficult due to the culmination of a variety of factors, from the microscopic to the population level, which are difficult to do in any circumstances, but are compounded by logistical issues when trying to make forecasts in real time. Logistical challenges include assimilating newly-collected data into the forecasting framework, accounting for delays in case reporting, and effectively communicating the model results to public health officials.
4.1 Reporting delays
Making forecasts in real time introduces the dimension of reporting delays into our forecasting models. From a disease surveillance perspective, reporting delays are a timeliness issue which varies by features of the disease (ease of diagnosis, incubation time), surveillance entity (local, state, or national government), transmission type (electronic or not), and case load, as well as variability in reporting between surveillance systems [158, 159]. From a data perspective, this means that observed values in the recent past are subject to change.
Since most forecasting models make the assumption that values used for fitting are fixed, we need to adapt our forecasting process by “nowcasting” observed values in the recent past. One method of now-casting is to only include “sufficiently complete” data up such that a forecasting model can make stable forecasts. For instance, 75% of dengue hemorrhagic fever cases in Thailand were reported to the Thai Ministry of Public Health within 10 weeks of infection . To account for this, we ignored the last 12 weeks (actually 6 biweeks, to be exact) before forecasting forward. In our notation, we fit our model to data to make a k-step forecast, , where . Another method of nowcasting is to use past reporting delays to model recent incomplete counts. Several frameworks have been proposed to nowcast infectious disease incidence based on past reporting rates.[92, 160] Other approaches for nowcasting have incorporated digital surveillance data.[19, 126]
When case counts for prior time periods are subject to change, it is important for researchers to have a collection of data “snapshots”, so that past situations can be investigated retrospectively with the information that was available at the time. Thus, database of should contain records of cases as they are reported, containing the date of illness and incidence that is timestamped upon deposit into the database.
4.2 Communication of results
Public health authorities have shown increasing interest in working with infectious disease forecasters in the light of recent important public health crises. Starting in 2009 with the pandemic influenza A outbreak, public health officials turned to forecasters for estimates of burden and burden averted due to vaccines and antivirals. During the Ebola outbreak in 2014, public health officials again turned to prediction for specific information regarding the potential outbreak size and intervention impacts. These efforts highlight how infectious disease forecasting can support public health practice now and in the future.
What makes a good forecast?
Previous work in meteorology has outlined 3 distinct forecast attributes of a forecast that contribute to its usefulness, or “goodness” . If we apply these guidelines to infectious disease forecasting, we can surmise that a forecast is good if it is (a) consistent: reflecting the forecaster’s best judgment, (b) quality: forecasts conditions that are actually observed during the time being forecasted, and (c) valuable: informs policy or other decision-making that results in increased benefits to individuals or society.
For a forecast to reflect the forecaster’s “best judgment” means that the forecast is reasonable based on the forecaster’s expert knowledge base, prior experience, and best and current methodology. The forecaster’s internal judgments are not usually available for evaluation or quantification, but could say that a forecast is not a reflection of best judgment if we discover that a forecasting model contains an error or under some conditions produces values outside the range of possible values.
To meet the conditions for high quality, forecasted values must correspond closely to observed values. The field of forecast verification is so vast and specialized that we could not possibly give it a comprehensive treatment here. Suffice it to say that reducing error is central goal of the field of forecasting. Examples of quality measurement approaches include the mean absolute error and the mean-squared error, which reflect forecast accuracy. Other examples include measures of bias, skill (often a comparison to reference models), and uncertainty .
Infectious disease forecasts are valuable if they are used to influence decisions. Sometimes value can sometimes be accessed in quantitative units (e.g. lives or money saved or lost). Forecast quality influences value to a large extent, but so do other more qualitative features of how the forecast is communicated. For example, a forecast will have a larger impact on decision-making if it is timely, presented clearly, and uses meaningful units in addition to being accurate or improving on a previous system.
5 Conclusion and Future Directions
There has been a great deal of progress made in infectious disease forecasting, however the field is very much still in its infancy. Forecasts of epidemics can inform public health response and decision-making, including risk communication to the general public, and timing and spatial targeting of interventions (e.g. vaccination campaigns or vector control measures). However, to maximize the impact that forecasts can have on the practice of public health, interdisciplinary teams must come together to tackle a variety of challenges, from the technological and statistical, to the biological and behavioral. To this end, the field of infectious disease forecasting should emphasize the development and integration of new theoretical frameworks that can be directly linked to tangible public health strategies.
To facilitate the development of scalable forecasting infrastructure and continued research on improving forecasting, the field should focus on developing data standards for both surveillance data and forecasts themselves. This will foster continued methodological development and facilitate scientific inquiry by enabling standard comparisons across forecasting efforts. One key barrier to entry to this field is that the problems are operationally complex: a model may be asked to forecast multiple targets at multiple different times, using only available data at a given time. Converging on standard language and terminology to describe these challenges is key to growing the field and will accelerate discovery and innovation for years to come.
-  Donald N McCloskey. The art of forecasting: From ancient to modern times. Cato J., 12:23, 1992.
-  Nate Silver. The signal and the noise: why so many predictions fail but some don’t. The Penguin Group, New York, New York, USA, 2012.
-  Willis Isbister Milham. Meteorology: a text-book on the weather, the causes of its changes, and weather forecasting for the student and general reader. Norwood The Macmillan Company, New York, 1918.
-  Patrick Gerland, Adrian E Raftery, Hana Ševčíková, Nan Li, Danan Gu, Thomas Spoorenberg, Leontine Alkema, Bailey K Fosdick, Jennifer Chunn, Nevena Lalic, et al. World population stabilization unlikely this century. Science, 346(6206):234–237, 2014.
-  Adrian E Raftery, Nan Li, Hana Ševčíková, Patrick Gerland, and Gerhard K Heilig. Bayesian probabilistic population projections for all countries. Proceedings of the National Academy of Sciences, 109(35):13915–13921, 2012.
-  James E Campbell. Polls and votes: the trial-heat presidential election forecasting model, certainty, and political campaigns. American Politics Quarterly, 24(4):408–433, 1996.
-  Andreas Graefe. German election forecasting: Comparing and combining methods for 2013. German Politics, 24(2):195–204, 2015.
-  Michael S Lewis-Beck and Mary Stegmaier. Us presidential election forecasting. PS: Political Science & Politics, 47(2):284–288, 2014.
-  Edward H Field, Timothy E Dawson, Karen R Felzer, Arthur D Frankel, Vipin Gupta, Thomas H Jordan, Tom Parsons, Mark D Petersen, Ross S Stein, RJ Weldon, et al. Uniform california earthquake rupture forecast, version 2 (ucerf 2). Bulletin of the Seismological Society of America, 99(4):2053–2107, 2009.
-  Andrew Bray and Frederic Paik Schoenberg. Assessment of point process models for earthquake forecasting. Statistical Science, pages 510–520, 2013.
Daniel W. Chambers, Jenny A. Baglivo, John E. Ebel, and Alan L. Kafka.
Earthquake Forecasting Using Hidden Markov Models.Pure and Applied Geophysics, 169(4):625–639, apr 2012.
-  Matthew Biggerstaff, David Alper, Mark Dredze, Spencer Fox, Isaac Chun-Hai Fung, Kyle S Hickmann, Bryan Lewis, Roni Rosenfeld, Jeffrey Shaman, Ming-Hsiang Tsou, et al. Results from the centers for disease control and prevention?s predict the 2013–2014 influenza season challenge. BMC infectious diseases, 16(1):357, 2016.
-  Rachel Lowe, Christovam Barcellos, Caio AS Coelho, Trevor C Bailey, Giovanini Evelim Coelho, Richard Graham, Tim Jupp, Walter Massa Ramalho, Marilia Sá Carvalho, David B Stephenson, et al. Dengue outlook for the world cup in brazil: an early warning model framework driven by real-time seasonal climate forecasts. The Lancet infectious diseases, 14(7):619–626, 2014.
-  Cécile Viboud, Kaiyuan Sun, Robert Gaffey, Marco Ajelli, Laura Fumanelli, Stefano Merler, Qian Zhang, Gerardo Chowell, Lone Simonsen, Alessandro Vespignani, et al. The rapidd ebola forecasting challenge: Synthesis and lessons learnt. Epidemics, 2017.
-  Leonhard Held, Sebastian Meyer, and Johannes Bracher. Probabilistic forecasting in infectious disease epidemiology: the 13th armitage lecture. Statistics in medicine, 36(22):3443–3460, 2017.
-  Nicholas G. Reich, Stephen A. Lauer, Krzysztof Sakrejda, Sopon Iamsirithaworn, Soawapak Hinjoy, Paphanij Suangtho, Suthanun Suthachana, Hannah E. Clapham, Henrik Salje, Derek A. T. Cummings, and Justin Lessler. Challenges in real-time prediction of infectious disease: A case study of dengue in thailand. PLOS Neglected Tropical Diseases, 10(6):e0004761, June 2016.
-  World Health Organization. Fact sheets: Infectious diseases.
-  Francis X. Diebold. Elements of forecasting. Department of Economics, University of Pennsylvania, 4th edition, 2001.
-  Logan C Brooks, David C Farrow, Sangwon Hyun, Ryan J Tibshirani, and Roni Rosenfeld. Nonmechanistic forecasts of seasonal influenza with iterative one-week-ahead distributions. PLoS computational biology, 14(6):e1006134, 2018.
-  Nicholas G Reich, Logan C Brooks, Spencer J Fox, Sasikiran Kandula, Craig J McGowan, Evan Moore, Dave Osthus, Evan L Ray, Abhinav Tushar, Teresa K Yamana, et al. A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the united states. Proceedings of the National Academy of Sciences, 116(8):3146–3154, 2019.
-  Sandip Mandal, Ram Rup Sarkar, and Somdatta Sinha. Mathematical models of malaria-a review. Malaria journal, 10(1):202, 2011.
-  Chris P Jewell and Richard G Brown. Bayesian data assimilation provides rapid decision support for vector-borne diseases. Journal of the Royal Society Interface, 12(108):20150367, 2015.
-  Richard S Ostfeld and Jesse L Brunner. Climate change and ixodes tick-borne diseases of humans. Philosophical Transactions of the Royal Society B: Biological Sciences, 370(1665):20140051, 2015.
-  Robert C Reiner Jr, T Alex Perkins, Christopher M Barker, Tianchan Niu, Luis Fernando Chaves, Alicia M Ellis, Dylan B George, Arnaud Le Menach, Juliet RC Pulliam, Donal Bisanzio, et al. A systematic review of mathematical models of mosquito-borne pathogen transmission: 1970–2010. Journal of The Royal Society Interface, 10(81):20120921, 2013.
-  José Lourenço and Mario Recker. The 2012 madeira dengue outbreak: epidemiological determinants and future epidemic potential. PLoS neglected tropical diseases, 8(8):e3083, 2014.
-  Nicholas B. Defelice, Eliza Little, Scott R. Campbell, and Jeffrey Shaman. Ensemble forecast of human West Nile virus cases and mosquito infection rates. Nature Communications, 8:14592, 2017.
-  Kelly R Moran, Geoffrey Fairchild, Nicholas Generous, Kyle Hickmann, Dave Osthus, Reid Priedhorsky, James Hyman, and Sara Y Del Valle. Epidemic forecasting is messier than weather forecasting: The role of human behavior and internet data streams in epidemic forecast. The Journal of infectious diseases, 214(suppl_4):S404–S408, 2016.
-  Dylan B. George, Wendy Taylor, Jeffrey Shaman, Caitlin Rivers, Brooke Paul, Tara O?Toole, Michael A. Johansson, Lynette Hirschman, Matthew Biggerstaff, Jason Asher, and Nicholas G. Reich. Technology to advance infectious disease forecasting for outbreak management. Nature Communications, 2019.
-  M F Myers, D J Rogers, J Cox, A Flahault, and S I Hay. Forecasting disease risk for increased epidemic preparedness in public health. Advances in parasitology, 47:309–30, 2000.
-  Ireneous N. Soyiri and Daniel D. Reidpath. An overview of health forecasting. Environmental Health and Preventive Medicine, 18(1):1–9, January 2013.
-  P J Birrell, G Ketsetzis, N J Gay, B S Cooper, A M Presanis, R J Harris, A Charlett, X S Zhang, P J White, R G Pebody, and D De Angelis. Bayesian modeling to unmask and predict influenza a/h1n1pdm dynamics in london. Proc Natl Acad Sci U S A, 108(45):18238–18243, 2011.
-  Anna L Buczak, Phillip T Koshute, Steven M Babin, Brian H Feighner, and Sheryl H Lewis. A data-driven epidemiological prediction method for dengue outbreaks using local and remote sensing data. BMC medical informatics and decision making, 12:124, 2012.
-  D. O. Gerardi and L. H. A. Monteiro. System identification and prediction of dengue fever incidence in rio de janeiro. Mathematical Problems in Engineering, 2011, 2011.
-  Yien Ling Hii, Huaiping Zhu, Nawi Ng, Lee Ching Ng, and Joacim Rocklöv. Forecast of dengue incidence using temperature and rainfall. PLOS Neglected Tropical Diseases, 6(11):e1908, November 2012.
-  Rachel Lowe, Trevor C. Bailey, David B. Stephenson, Richard J. Graham, Caio A. S. Coelho, Marilia Sá Carvalho, and Christovam Barcellos. Spatio-temporal modelling of climate-sensitive disease risk: Towards an early warning system for dengue in brazil. Computers and Geosciences, 37(3):371–381, 2011.
-  Hsin-Min Lu, Daniel Zeng, and Hsinchun Chen. Prospective infectious disease outbreak detection using markov switching models. IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, 22(4):565–577, 2010.
-  Hiroshi Nishiura. Real-time forecasting of an epidemic using a discrete time stochastic model: a case study of pandemic influenza (h1n1-2009). BioMedical Engineering OnLine, 10(1):15, 2011.
-  Jeffrey Shaman, Alicia Karspeck, Wan Yang, James Tamerius, and Marc Lipsitch. Real-time influenza forecasts during the 2012?2013 season. Nature Communications, 4:2837, December 2013.
-  Jeffrey Shaman, Wan Yang, and Sasikiran Kandula. Inference and forecast of the current west african ebola outbreak in guinea, sierra leone and liberia. PLoS Currents, 6:ecurrents.outbreaks.3408774290b1a0f2dd7cae877c8b8f, 2014.
-  Ayako Sumi and Ken-ichi Kamo. Mem spectral analysis for predicting influenza epidemics in japan. Environmental Health and Preventive Medicine, 17(2):98–108, March 2012.
-  Weirong Yan, Yong Xu, Xiaobing Yang, and Yikai Zhou. A hybrid model for short-term bacillary dysentery prediction in yichang city, china. Japanese Journal of Infectious Diseases, 63(4):264–270, 2010.
-  National Oceanographic and Atmospheric Administration. Dengue forecasting project website, June 2015. Access date: May 29, 2015.
-  Adrian M Tompkins, Felipe J Colón-González, Francesca Di Giuseppe, and Didacus B Namanya. Dynamical malaria forecasts are skillful at regional and local scales in uganda up to 4 months ahead. GeoHealth, 3(3):58–66, 2019.
-  Tuan Q. Dinh, Hiep V. Le, Tru H. Cao, Quang C. Luong, and Hai T. Diep. Forecasting the magnitude of dengue in Southern Vietnam. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 9621, pages 554–563, 2016.
-  M. Amaku, F. Azevedo, M. N. Burattini, G. E. Coelho, F. A.B. Coutinho, D. Greenhalgh, L. F. Lopez, R. S. Motitsuki, A. Wilder-Smith, and E. Massad. Magnitude and frequency variations of vector-borne infection outbreaks using the Ross-MacDonald model: explaining and predicting outbreaks of dengue fever. Epidemiology and Infection, 144(16):3435–3450, 2016.
-  Guanghu Zhu, Jianpeng Xiao, Bing Zhang, Tao Liu, Hualiang Lin, Xing Li, Tie Song, Yonghui Zhang, Wenjun Ma, and Yuantao Hao. The spatiotemporal transmission of dengue and its driving mechanism: A case study on the 2014 dengue outbreak in Guangdong, China. Science of the Total Environment, 622-623:252–259, 2018.
-  Michael A. Johansson, Nicholas G. Reich, Aditi Hota, John S. Brownstein, and Mauricio Santillana. Evaluating the performance of infectious disease forecasts: A comparison of climate-driven and seasonal dengue forecasts for mexico. Scientific Reports, 6(August):33707, September 2016.
-  Mohammad Y. Anwar, Joseph A. Lewnard, Sunil Parikh, and Virginia E. Pitzer. Time series analysis of malaria in Afghanistan: using ARIMA models to predict future trends in incidence. Malaria Journal, 15(1):566, 2016.
-  Kate Zinszer, Ruth Kigozi, Katia Charland, Grant Dorsey, Timothy F. Brewer, John S. Brownstein, Moses R. Kamya, and David L. Buckeridge. Forecasting malaria in a highly endemic country using environmental and clinical predictors. Malaria Journal, 14(1):245, 2015.
-  Teresa K Yamana, Sasikiran Kandula, and Jeffrey Shaman. Superensemble forecasts of dengue outbreaks. Journal of The Royal Society Interface, 13(123):20160410, 2016.
-  Stephen A Lauer, Krzysztof Sakrejda, Evan L Ray, Lindsay T Keegan, Qifang Bi, Paphanij Suangtho, Soawapak Hinjoy, Sopon Iamsirithaworn, Suthanun Suthachana, Yongjua Laosiritaworn, et al. Prospective forecasts of annual dengue hemorrhagic fever incidence in thailand, 2010–2014. Proceedings of the National Academy of Sciences, 115(10):E2175–E2182, 2018.
-  Maquins Odhiambo Sewe, Yesim Tozan, Clas Ahlm, and Joacim Rocklöv. Using remote sensing environmental data to forecast malaria incidence at a rural district hospital in Western Kenya. Scientific Reports, 7(1):2589, 2017.
-  Paul D Haemig, S Sjöstedt de Luna, A Grafström, Stefan Lithner, Åke Lundkvist, Jonas Waldenström, Jonas Kindberg, Johan Stedt, and Björn Olsén. Forecasting risk of tick-borne encephalitis (tbe): using data from wildlife and climate to predict next year’s number of human victims. Scandinavian journal of infectious diseases, 43(5):366–72, 2011.
-  Hai-Ning Liu, Li-Dong Gao, Gerardo Chowell, Shi-Xiong Hu, Xiao-Ling Lin, Xiu-Jun Li, Gui-Hua Ma, Ru Huang, Hui-Suo Yang, Huaiyu Tian, and Hong Xiao. Time-specific ecologic niche models forecast the risk of hemorrhagic fever with renal syndrome in dongting lake district, china, 2005?2010. PLoS ONE, 9(9):e106839, September 2014.
-  Sean M. Moore, Andrew Monaghan, Kevin S. Griffith, Titus Apangu, Paul S. Mead, and Rebecca J. Eisen. Improvement of disease prediction and modeling through the use of meteorological ensembles: Human plague in uganda. PLoS ONE, 7(9):e44431, September 2012.
-  Anna L Buczak, Benjamin Baugher, Linda J Moniz, Thomas Bagley, Steven M Babin, and Erhan Guven. Ensemble method for dengue prediction. PloS one, 13(1):e0189988, 2018.
-  Evan L. Ray, Krzysztof Sakrejda, Stephen A. Lauer, Michael A. Johansson, and Nicholas G. Reich. Infectious disease prediction with kernel conditional density estimation. Statistics in Medicine, 36(30):4908–4929, December 2017.
-  Santosh Thakur and Ramesh Dharavath. Artificial neural network based prediction of malaria abundances using big data: A knowledge capturing approach. Clinical Epidemiology and Global Health, 7(1):121–126, 2019.
-  Mahmood Akhtar, Moritz UG Kraemer, and Lauren Gardner. A dynamic neural network model for real-time prediction of the zika epidemic in the americas. bioRxiv, page 466581, 2018.
-  Robert H Gaffey and Cécile Viboud. Application of the cdc ebolaresponse modeling tool to disease predictions. Epidemics, 2017.
-  Sebastian Funk, Anton Camacho, Adam J Kucharski, Rosalind M Eggo, and W John Edmunds. Real-time forecasting of infectious disease dynamics with a stochastic semi-mechanistic model. Epidemics, 2016.
-  Jason Asher. Forecasting ebola with a regression transmission model. Epidemics, 22:50–55, 2018.
-  Ayaz Hyder, David L. Buckeridge, and Brian Leung. Predictive Validation of an Influenza Spread Model. PLoS ONE, 8(6), 2013.
-  Elaine Nsoesie, Madhav Mararthe, and John Brownstein. Forecasting Peaks of Seasonal Influenza Epidemics. PLoS Currents, 2013.
-  Dave Osthus, Kyle S Hickmann, Petruţa C Caragea, Dave Higdon, and Sara Y Del Valle. Forecasting seasonal influenza with a state-space sir model. The annals of applied statistics, 11(1):202, 2017.
-  Dave Osthus, James Gattiker, Reid Priedhorsky, and Sara Y. Del Valle. Dynamic Bayesian Influenza Forecasting in the United States with Hierarchical Discrepancy (with Discussion). Bayesian Analysis, 14(1):261–312, mar 2019.
-  Jeffrey Shaman and Alicia Karspeck. Forecasting seasonal outbreaks of influenza. Proceedings of the National Academy of Sciences of the United States of America, 109(3):20425–30, 2012.
-  Sen Pei and Jeffrey Shaman. Counteracting structural errors in ensemble forecast of influenza outbreaks. Nature communications, 8(1):925, 2017.
-  David C Farrow, Logan C Brooks, Sangwon Hyun, Ryan J Tibshirani, Donald S Burke, and Roni Rosenfeld. A human judgment approach to epidemiological forecasting. PLoS computational biology, 13(3):e1005248, 2017.
-  Leonhard Held and Michaela Paul. Modeling seasonality in space-time infectious disease surveillance data. Biometrical Journal, 54(6):824–843, November 2012.
-  Edward Goldstein, Sarah Cobey, Saki Takahashi, Joel C. Miller, and Marc Lipsitch. Predicting the epidemic sizes of influenza A/H1N1, A/H3N2, and B: A statistical method. PLoS Medicine, 8(7), 2011.
-  Michael J Kane, Natalie Price, Matthew Scotch, and Peter Rabinowitz. Comparison of arima and random forest time series models for prediction of avian influenza h5n1 outbreaks. BMC Bioinformatics, 15(1):276, 2014.
-  Andrea Freyer Dugas, Mehdi Jalalpour, Yulia Gel, Scott Levin, Fred Torcaso, Takeru Igusa, and Richard E. Rothman. Influenza Forecasting with Google Flu Trends. PLoS ONE, 8(2):e56176, feb 2013.
-  Qinneng Xu, Yulia R Gel, L Leticia Ramirez Ramirez, Kusha Nezafati, Qingpeng Zhang, and Kwok-Leung Tsui. Forecasting influenza in hong kong with google search queries and statistical model fusion. PloS one, 12(5):e0176690, 2017.
-  Peng Guan, De-Sheng Huang, and Bao-Sen Zhou. Forecasting model for the incidence of hepatitis a based on artificial neural network. World journal of gastroenterology: WJG, 10(24):3579, 2004.
-  Michael S Deiner, Lee Worden, Alex Rittel, Sarah F Ackley, Fengchen Liu, Laura Blum, James C Scott, Thomas M Lietman, and Travis C Porco. Short-term leprosy forecasting from an expert opinion survey. PloS one, 12(8):e0182245, 2017.
-  Matthew James Keeling and Pejman Rohani. Modeling Infectious Diseases in Humans and Animals, volume 47. Princeton University Press, Princeton, New Jersey, 2007.
-  Constantinos I Siettos and Lucia Russo. Mathematical modeling of infectious disease dynamics. Virulence, 4(4):295–306, May 2013.
-  Justin Lessler and Derek A. T. Cummings. Mechanistic models of infectious disease and their impact on public health. American Journal of Epidemiology, 183(5):415–422, March 2016.
-  W. O. Kermack and A. G. McKendrick. A contribution to the mathematical theory of epidemics. Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 115(772):700–721, 1927.
-  S Eubank, H Guclu, V S Kumar, M V Marathe, A Srinivasan, Z Toroczkai, and N Wang. Modelling disease outbreaks in realistic urban social networks. Nature, 429(6988):180–184, May 2004.
-  Neil M Ferguson, Derek A T Cummings, Christophe Fraser, James C Cajka, Philip C Cooley, and Donald S Burke. Strategies for mitigating an influenza pandemic. Nature, 442(7101):448–452, July 2006.
-  Nicholas G. Reich, Sourya Shrestha, Aaron A. King, Pejman Rohani, Justin Lessler, Siripen Kalayanarooj, In-Kyu Yoon, Robert V. Gibbons, Donald S. Burke, and Derek A. T. Cummings. Interactions between serotypes of dengue highlight epidemiological impact of cross-immunity. Journal of the Royal Society, Interface, 10(86):20130414, 2013.
-  David L. Smith, Katherine E. Battle, Simon I. Hay, Christopher M. Barker, Thomas W. Scott, and F. Ellis McKenzie. Ross, Macdonald, and a theory for the dynamics and control of mosquito-transmitted pathogens. PLoS Pathogens, 8(4), 2012.
-  GEP Box and GM Jenkins. Some statistical aspects of adaptive optimization and control. Journal of the Royal Statistical Society. Series B ( …, 24(2):297–343, 1962.
-  Steffen Unkel, C. Paddy Farrington, Paul H. Garthwaite, Chris Robertson, and Nick Andrews. Statistical methods for the prospective detection of infectious disease outbreaks: a review. Journal of the Royal Statistical Society: Series A (Statistics in Society), 175(1):49–82, January 2012.
-  Charles C Holt. Forecasting seasonals and trends by exponentially weighted moving averages. International journal of forecasting, 20(1):5–10, 2004.
-  Peter R Winters. Forecasting sales by exponentially weighted moving averages. Management science, 6(3):324–342, 1960.
-  Sarah Gelper, Roland Fried, and Christophe Croux. Robust forecasting with exponential and holt–winters smoothing. Journal of forecasting, 29(3):285–300, 2010.
-  Paul Goodwin et al. The holt-winters approach to exponential smoothing: 50 years old and going strong. Foresight, 19:30–33, 2010.
-  Leonhard Held, Michael Höhle, and Mathias Hofmann. A statistical framework for the analysis of multivariate infectious disease surveillance counts. Statistical modelling, 5(3):187–199, 2005.
-  Michael Höhle and Matthias an der Heiden. Bayesian nowcasting during the STEC O104:H4 outbreak in Germany, 2011. Biometrics, 70(4):993–1002, dec 2014.
-  Sebastian Meyer, Leonhard Held, and Michael Höhle. Spatio-temporal analysis of epidemic phenomena using the R package surveillance. Journal of Statistical Software, 77(11):1–55, 2017.
-  Tin Kam Ho. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition, volume 1, pages 278–282. IEEE, 1995.
-  Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Number 2. Springer, 2009.
-  Leah R Johnson, Robert B Gramacy, Jeremy Cohen, Erin Mordecai, Courtney Murdock, Jason Rohr, Sadie J Ryan, Anna M Stewart-Ibarra, Daniel Weikel, et al. Phenomenological forecasting of disease incidence using heteroskedastic gaussian processes: A dengue case study. The Annals of Applied Statistics, 12(1):27–66, 2018.
-  Yuexin Wu, Yiming Yang, Hiroshi Nishiura, and Masaya Saitoh. Deep learning for epidemiological predictions. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1085–1088. ACM, 2018.
-  Craig J McGowan, Matthew Biggerstaff, Michael Johansson, Karyn M Apfeldorf, Michal Ben-Nun, Logan Brooks, Matteo Convertino, Madhav Erraguntla, David C Farrow, John Freeze, et al. Collaborative efforts to forecast seasonal influenza in the united states, 2015–2016. Scientific reports, 9(1):683, 2019.
-  Michael A Johansson et al. Advancing probabilistic epidemic forecasting through an open challenge: The dengue forecasting project. under review, 2019.
-  Joceline Lega and Heidi E Brown. Data-driven outbreak forecasting with a simple nonlinear growth model. Epidemics, 17:19–26, 2016.
-  DARPA. CHIKV Challenge Announces Winners, Progress toward Forecasting the Spread of Infectious Diseases, 2015.
-  Srinivasan Venkatramanan, Bryan Lewis, Jiangzhuo Chen, Dave Higdon, Anil Vullikanti, and Madhav Marathe. Using data-driven agent-based models for forecasting emerging infectious diseases. Epidemics, 2017.
-  Jason Asher, Christopher Barker, Grace Chen, Derek Cummings, Matteo Chinazzi, Shelby Daniel-Wayman, Marc Fischer, Neil Ferguson, Dean Follman, M. Elizabeth Halloran, Michael Johansson, Kiersten Kugeler, Jennifer Kwan, Justin Lessler, Ira M. Longini, Stefano Merler, Andrew Monaghan, Ana Pastore y Piontti, Alex Perkins, D. Rebecca Prevots, Robert Reiner, Luca Rossi, Isabel Rodriguez-Barraquer, Amir S. Siraj, Kaiyuan Sun, Alessandro Vespignani, and Qian Zhang. Preliminary results of models to predict areas in the Americas with increased likelihood of Zika virus transmission in 2017. bioRxiv, page 187591, sep 2017.
-  A Marm Kilpatrick and W John Pape. Predicting human west nile virus infections with mosquito surveillance data. American journal of epidemiology, 178(5):829–835, 2013.
-  Justin K Davis, Geoffrey Vincent, Michael B Hildreth, Lon Kightlinger, Christopher Carlson, and Michael C Wimberly. Integrating environmental monitoring and mosquito surveillance to predict vector-borne disease: prospective forecasts of a west nile virus outbreak. PLoS currents, 9, 2017.
-  Yuan Shi, Xu Liu, Suet-Yheng Kok, Jayanthi Rajarethinam, Shaohong Liang, Grace Yap, Chee-Seng Chong, Kim-Sung Lee, Sharon SY Tan, Christopher Kuan Yew Chin, et al. Three-month real-time dengue forecast models: an early warning system for outbreak alerts and policy decision support in singapore. Environmental health perspectives, 124(9):1369–1375, 2015.
-  Teresa K Yamana and Jeffrey Shaman. A framework for evaluating the effects of observational type and quality on vector-borne disease forecast. Epidemics, page 100359, 2019.
-  Xiangjun Du, Aaron A King, Robert J Woods, and Mercedes Pascual. Evolution-informed forecasting of seasonal influenza A (H3N2). Science translational medicine, 9(413):eaan5325, oct 2017.
-  Xiangjun Du and Mercedes Pascual. Incidence Prediction for the 2017-2018 Influenza Season in the United States with an Evolution-informed Model. PLoS Currents, 2018.
-  Sasikiran Kandula, Wan Yang, and Jeffrey Shaman. Type- and Subtype-Specific Influenza Forecast. American Journal of Epidemiology, 185(5):395–402, mar 2017.
-  Dylan H Morris, Katelyn M Gostic, Simone Pompei, Trevor Bedford, Marta Łuksza, Richard A Neher, Bryan T Grenfell, Michael Lässig, and John W Mccauley. Predictive Modeling of Influenza Shows the Promise of Applied Evolutionary Biology. Trends in Microbiology, 2017.
-  James Hadfield, Colin Megill, Sidney M Bell, John Huddleston, Barney Potter, Charlton Callender, Pavel Sagulenko, Trevor Bedford, and Richard A Neher. Nextstrain: real-time tracking of pathogen evolution. bioRxiv, page 224048, nov 2017.
-  James Surowiecki. The wisdom of crowds : why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations. Doubleday, 2004.
-  David V Budescu and Eva Chen. Identifying expertise to extract the wisdom of crowds. Management Science, 61(2):267–280, 2014.
-  Marcel Salathé. Digital epidemiology: what is it, and where is it going? Life Sciences, Society and Policy, 14(1):1, dec 2018.
-  Reid Priedhorsky, Dave Osthus, Ashlynn R. Daughton, Kelly R. Moran, and Aron Culotta. Deceptiveness of internet data for disease surveillance. arXiv, nov 2017.
-  Jeremy Ginsberg, Matthew H. Mohebbi, Rajan S. Patel, Lynnette Brammer, Mark S. Smolinski, and Larry Brilliant. Detecting influenza epidemics using search engine query data. Nature, 457(7232):1012–1014, feb 2009.
-  A. F. Dugas, Y.-H. Hsieh, S. R. Levin, J. M. Pines, D. P. Mareiniss, A. Mohareb, C. A. Gaydos, T. M. Perl, and R. E. Rothman. Google Flu Trends: Correlation With Emergency Department Influenza Rates and Crowding Metrics. Clinical Infectious Diseases, 54(4):463–469, feb 2012.
-  D. Lazer, R. Kennedy, G. King, and A. Vespignani. The Parable of Google Flu: Traps in Big Data Analysis. Science, 343(6176):1203–1205, mar 2014.
-  Mauricio Santillana, D Wendong Zhang, Benjamin M Althouse, and John W Ayers. What can digital disease detection learn from (an external revision to) Google Flu Trends? American journal of preventive medicine, 47(3):341–7, sep 2014.
-  Donald R. Olson, Kevin J. Konty, Marc Paladini, Cecile Viboud, and Lone Simonsen. Reassessing Google Flu Trends Data for Detection of Seasonal and Pandemic Influenza: A Comparative Epidemiological Study at Three Geographic Scales. PLoS Computational Biology, 9(10):e1003256, oct 2013.
-  Shihao Yang, Samuel C. Kou, Fred Lu, John S. Brownstein, Nicholas Brooke, and Mauricio Santillana. Advances in using Internet searches to track dengue. PLOS Computational Biology, 13(7):e1005607, jul 2017.
-  Shihao Yang, Mauricio Santillana, and S C Kou. Accurate estimation of influenza epidemics using Google search data via ARGO. Proceedings of the National Academy of Sciences of the United States of America, 112(47):14473–8, nov 2015.
-  Sarah F. McGough, John S. Brownstein, Jared B. Hawkins, and Mauricio Santillana. Forecasting Zika Incidence in the 2016 Latin America Outbreak Combining Traditional Disease Surveillance with Search, Social Media, and News Report Data. PLOS Neglected Tropical Diseases, 11(1):e0005295, jan 2017.
-  Fred Sun Lu, Suqin Hou, Kristin Baltrusaitis, Manan Shah, Jure Leskovec, Rok Sosic, Jared Hawkins, John Brownstein, Giuseppe Conidi, Julia Gunn, Josh Gray, Anna Zink, and Mauricio Santillana. Accurate Influenza Monitoring and Forecasting Using Novel Internet Data Streams: A Case Study in the Boston Metropolis. JMIR public health and surveillance, 4(1):e4, jan 2018.
-  Dave Osthus, Ashlynn R Daughton, and Reid Priedhorsky. Even a good influenza forecasting model can benefit from internet-based nowcasts, but those benefits are limited. PLoS computational biology, 15(2):e1006599, 2019.
-  Mauricio Santillana, Elaine O Nsoesie, Sumiko R Mekaru, David Scales, and John S Brownstein. Using clinicians? search query data to monitor influenza epidemics. Clinical infectious diseases: an official publication of the Infectious Diseases Society of America, 59(10):1446, 2014.
-  Anna R Thorner, Bin Cao, Terrence Jiang, Amy J Warner, and Peter A Bonis. Correlation between uptodate searches and reported cases of middle east respiratory syndrome during outbreaks in saudi arabia. Open forum infectious diseases, 3(1):ofw043, 2016.
-  Jeffrey Shaman and Melvin Kohn. Absolute humidity modulates influenza survival, transmission, and seasonality. Proceedings of the National Academy of Sciences, 106(9):3243–3248, 2009.
-  Rachel Lowe, Anna M Stewart-Ibarra, Desislava Petrova, Markel García-Díez, Mercy J Borbor-Cordova, Raúl Mejía, Mary Regato, and Xavier Rodó. Climate services for health: predicting the evolution of the 2016 dengue season in Machala, Ecuador. The Lancet Planetary Health, 1(4):e142–e151, jul 2017.
-  John M Bates and Clive WJ Granger. The combination of forecasts. Journal of the Operational Research Society, 20(4):451–468, 1969.
-  Spyros Makridakis and Robert L Winkler. Averages of forecasts: Some empirical results. Management Science, 29(9):987–996, 1983.
-  Robert T Clemen and Robert L Winkler. Aggregating probability distributions. Advances in decision analysis: From foundations to applications, page 154, 2007.
-  Clive WJ Granger and Ramu Ramanathan. Improved methods of combining forecasts. Journal of forecasting, 3(2):197–204, 1984.
-  Tilmann Gneiting, Roopesh Ranjan, et al. Combining predictive distributions. Electronic Journal of Statistics, 7:1747–1782, 2013.
-  Teresa K. Yamana, Sasikiran Kandula, and Jeffrey Shaman. Individual versus superensemble forecasts of seasonal influenza outbreaks in the United States. PLOS Computational Biology, 13(11):e1005801, nov 2017.
-  Evan L Ray and Nicholas G Reich. Prediction of infectious disease epidemics via weighted density ensembles. PLoS computational biology, 14(2):e1005910, 2018.
-  Nicholas G Reich, Craig J McGowan, Teresa K Yamana, Abhinav Tushar, Evan L Ray, Dave Osthus, Sasikiran Kandula, Logan C Brooks, Willow Crawford-Crudell, Graham Casey Gibson, et al. A collaborative multi-model ensemble for real-time influenza season forecasting in the us. bioRxiv, page 566604, 2019.
-  Morgan E Smith, Brajendra K Singh, Michael A Irvine, Wilma A Stolk, Swaminathan Subramanian, T Déirdre Hollingsworth, and Edwin Michael. Predicting lymphatic filariasis transmission and elimination dynamics using a multi-model ensemble framework. Epidemics, 18:16–28, mar 2017.
-  J Scott Armstrong. Evaluating forecasting methods. International Journal of Forecasting, (1990):443–472, 2001.
-  Nicholas G. Reich, Justin Lessler, Krzysztof Sakrejda, Stephen A. Lauer, Sopon Iamsirithaworn, and Derek A. T. Cummings. Case Study in Evaluating Time Series Prediction Models Using the Relative Mean Absolute Error. The American Statistician, 70(3):285–292, feb 2016.
-  Rob J. Hyndman and Anne B. Koehler. Another look at measures of forecast accuracy. International Journal of Forecasting, 22(4):679–688, 2006.
-  Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007.
-  Johannes Bracher. On the multibin logarithmic score used in the flusight competitions. Proceedings of the National Academy of Sciences, 116(42):20809–20810, 2019.
-  Sebastian Funk, Anton Camacho, Adam J. Kucharski, Rachel Lowe, Rosalind M. Eggo, and W. John Edmunds. Assessing the performance of real-time epidemic forecasts. bioRxiv, page 177451, aug 2017.
-  Hans Hersbach. Decomposition of the Continuous Ranked Probability Score for Ensemble Prediction Systems. Weather and Forecasting, 15(5):559–570, oct 2000.
-  A. Allen Bradley and Stuart S. Schwartz. Summary Verification Measures and Their Interpretation for Ensemble Forecasts. Monthly Weather Review, 139(9):3075–3089, sep 2011.
-  Konrad Bogner, Katharina Liechti, Luzi Bernhard, Samuel Monhart, and Massimiliano Zappa. Skill of Hydrological Extended Range Forecasts for Water Resources Management in Switzerland. Water Resources Management, 32(3):969–984, feb 2018.
-  Francis X Diebold and Robert S Mariano. Comparing predictive accuracy. Journal of Business & economic statistics, 20(1):134–144, 2002.
-  Rob Hyndman, George Athanasopoulos, Christoph Bergmeir, Gabriel Caceres, Leanne Chhay, Mitchell O’Hara-Wild, Fotios Petropoulos, Slava Razbash, Earo Wang, and Farah Yasmeen. forecast: Forecasting functions for time series and linear models, 2019. R package version 8.7.
-  Rob J Hyndman and Yeasmin Khandakar. Automatic time series forecasting: the forecast package for R. Journal of Statistical Software, 26(3):1–22, 2008.
-  Christoph Bergmeir, Rob J. Hyndman, and Bonsoo Koo. A note on the validity of cross-validation for evaluating autoregressive time series prediction. Computational Statistics & Data Analysis, 120:70–83, April 2018.
-  Jun Shao. Linear model selection by cross-validation. Journal of the American Statistical Association, 88(422):486–494, 1993.
-  Andrew Y. Ng. Preventing “overfitting” of cross-validation data. In In Proceedings of the Fourteenth International Conference on Machine Learning, pages 245–253. Morgan Kaufmann, 1997.
-  M Stone. Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society Series B, 36(2):111–147, 1974.
Benjamin M Althouse, Samuel V Scarpino, Lauren Ancel Meyers, John W Ayers,
Marisa Bargsten, Joan Baumbach, John S Brownstein, Lauren Castro, Hannah
Clapham, Derek AT Cummings, et al.
Enhancing disease surveillance with novel data streams: challenges
EPJ Data Science, 4(1):17, 2015.
-  Christoph Bergmeir and José M. Benítez. On the use of cross-validation for time series predictor evaluation. Information Sciences, 191:192–213, may 2012.
-  Ruth Ann Jajosky and Samuel L Groseclose. Evaluation of reporting timeliness of public health surveillance systems for infectious diseases. BMC Public Health, 4(1):29, dec 2004.
-  Axel Bonačić Marinović, Corien Swaan, Jim van Steenbergen, and Mirjam Kretzschmar. Quantifying reporting timeliness to improve outbreak control. Emerging infectious diseases, 21(2):209–16, feb 2015.
-  Leonardo S Bastos, Theodoros Economou, Marcelo FC Gomes, Daniel AM Villela, Flavio C Coelho, Oswaldo G Cruz, Oliver Stoner, Trevor Bailey, and Claudia T Codeço. A modelling approach for correcting reporting delays in disease surveillance data. Statistics in medicine, 2019.
-  Allan H. Murphy. What Is a Good Forecast? An Essay on the Nature of Goodness in Weather Forecasting. Weather and Forecasting, 8(2):281–293, jun 1993.
-  Ian T. Jolliffe and David B. Stephenson. Introduction. In Ian T. Jolliffe and David B. Stephenson, editors, Forecast verification : a practitioner’s guide in atmospheric science, pages 1–12. John Wiley & Sons Ltd, Chichester, West Sussex, England, 2003.