Standardization in clinical case definitions is a significant challenge. This is becoming more pertinent as the number and types of places, modes of data collection and populations generating data are expanding (from clinical data to healthworker-facilitated data wherein healthworkers visit individuals’ houses, record symptoms and take specimens, to citizen-science studies in which participants report symptoms from home and mail in or submit specimens goff2015surveillance ; fragaszy2016cohort ) making infection prediction based on a specific syndromic case definition (set of symptoms) challenging. Moreover, it’s extremely rare for data from different studies to be collected in the exact same mode, context and from the same type of population. Therefore symptoms (features) can mean different things; “fever” may mean something different reported to a doctor than at home through a smartphone app ray2017predicting ; rehman2018domain
. Furthermore, how young people report may be different from how older people report symptoms. These differences in the data collection as well as the variance in the demographic distributions of the different datasets make the important problem of predicting infection based on syndromic case definitions challenging.
Early work has shown that public health collection methods can be conceptualized as domains, and domain adaptation can be useful for prediction from symptom data sets obtained via these different modes rehman2018domain . Beyond this, to the best of our knowledge no work has addressed the issue of domain differences in health data while also accounting for population attributes (work has also only focused on improving prediction in a target data set via the use of a single source, whereas the work here uses multiple sources from different domains). Incorporation of population structure has not been explored extensively, though in health practice and research attributes of the people contributing the data (here we consider population demographics like age, gender) are commonly available, and there are shared characteristics within these groups saria2010learning . While increasing representation granularity by increasing the number of classes can help, ad hoc discretization into fixed sets can limit ability to model instance-specific variability. Therefore hierarchical approaches have been used (but not yet for domain adaptation); for example Dirichlet processes have been used to allow sharing of mixture components in time-series data, generating global and individual topic parameters saria2010learning .
Hierarchical approaches have primarily been developed in natural language processing, and use Bayesian priors to tie parameters across multiple tasksevgeniou2005learning
. In such methods, each domain has its own domain-specific parameter for each feature which the model links via a hierarchical Bayesian global prior instead of a constant prior. This prior encourages features to have similar weights across domains, unless there is good contrary evidence. Hierarchical Bayesian frameworks are a more principled approach for transfer learning, compared to approaches which learn parameters of each task/distribution independently and smooth parameters of tasks with more information towards coarser-grained onescarlin2010bayes ; mccallum1998improving . An undirected Bayesian transfer hierarchy has been used to jointly model the shapes of different mammals elidan2012convex . While we build on this idea of hierarchical modeling for domain adaptation, here we go further to explicitly model population attributes via hierarchical structure. Also, given that health-related data sets can be collected in many different ways and from varied population samples, we explicitly consider a multi-source situation using empirical information about the included population in multiple studies to contribute to learning the model posterior and improve transfer of information to a new population and domain, with limited infection labels.
Each dataset includes symptoms from individuals, laboratory confirmation of type of respiratory infection virus they had (if any), as well as the age and gender of the person as example basic population attributes. We group these attributes into categories (gender as male/female, age as 0-4 years, 5-15 years, 16-44 years, 45-64 years and 65+ years) rehman2018domain . GoViral data was collected from volunteers who self-reported symptoms online and also mailed in bio-specimens for laboratory confirmation of illness in New York City. It consists of 520 observations out of which 291 had positive laboratory results goff2015surveillance . FluWatch consists of 915 observations (567 positive cases of flu) of volunteers in the United Kingdom. These two datasets belong to the “citizen science” domain fragaszy2016cohort ; rehman2018domain . Hong Kong consists of 4954 observations (1471 positive cases of flu) collected by healthworkers in Hong Kong cowling2010comparative . The Hutterite data is composed of 1281 observations (787 positive cases of flu) of colonies in Alberta, Canada sampled by nurses loeb2010effect . The high variability in attribute distributions in these real-world datasets is illustrated in Figure 1(i).
Overall Undirected Hierarchical Multi-source Bayesian Approach In this framework, the lowest level of the hierarchy represents the datasets (within each domain (in our case, collection mode), , for each of which we have data as shown in Figure 1. As in all Bayesian problems, the dataset parameters should represent the data well. Here, are influenced by the domain-specific parameters (); are generated according to , where is the domain. In the undirected population-aware hierarchical model we allow the domain specific parameters to have multiple parents and learn all parameters simultaneously. Accordingly, the domain parameters are generated according to the distribution . Here, we explicitly include to represent the population parameters (here for the different age group categories, and similarly for genders where . The population parameters and have the root parameter as the parent, which represents invariant information across all of the datasets, classes and population attributes,
. Then, the joint distribution accounting for all of these data and parameters is:.
Hierarchy Priors For all parameters we use independent priors that are computed based on symptom predictivity for each age group and gender. The inclusion of data dependent priors in Bayesian learning has been explored to incorporate domain knowledge into the posterior distribution of parameters darnieder2011bayesian
. For population-aware modeling, data-informed prior distributions are important because the distributions from each dataset are particular to the study, and thus capturing this information adds more information to the analysis than improper or vague priors (e.g. for a sample wherin one demographic group is under-represented), also motivates the multiple parents in the hierarchy. In contrast, using just the root prior for estimating the posterior ignores the demographic information available. Therefore, we use an empirical Bayes approach to specify weakly informative priors, centered around the estimates of the model parametersvan2017prior . Root parameters are centered on the cumulative data since the root parameter captures domain invariant information.
Model Steps First, we use a probabilistic framework to jointly learn each parameter based on all levels of the hierarchy. We use a maximum a-posteriori parameter estimate instead of the full posterior for the joint distribution, which would be computationally intractable. We use a formulation, proposed in elidan2012convex that is amenable to standard optimization techniques, resulting in the objective:
|FEDA (Only symptoms)||0.675||0.521||0.900||0.726|
|FEDA+p (With demographics)||0.693||0.612||0.914||0.824|
|Hierarchical (Only symptoms)||0.685||0.486||0.889||0.719|
|Hierarchical+p (With demographics)||0.710||0.627||0.918||0.827|
For datasets , denotes the parameter for symptom . From a specific dataset’s parameter space, denotes each symptom. is a statistical measure of the symptom in the dataset. In this case the statistical measure is the proportion of the particular symptom resulting in a positive cold/flu test (i.e. the positive predictive value). is the set of all nodes in the hierarchy (here, ). We consider the case of the parameters belonging to a multinomial distributions and consider the log representation. Regularizing parameter was chosen to be 1 to allow Laplacian smoothing mccallum1998improving . The function is a divergence (L2 norm used) over the child and the parent parameters that encourages child parameters to be influenced by parent parameters, and allows a child parameter to be closely linked to more than one parent. The weight
represents the influence balance between node parameters and node parent parameters. Based on hyperparameter tuning, a value of 0.2 forwas used in all experiments. For optimization of the objective function, we use Powell’s method fletcher1963rapidly .
Second, we learn the influence of the each hierarchical levels for a particular dataset; this is done to enable the model to give more weight to one level of the hierarchy when needed. In other words, how much demographic-invariant or domain-invariant information is needed depends upon how much information is in a given dataset. The reason for learning the weights for the different levels for each dataset independently is that each dataset would require different amounts of information from the demographic-specific and the domain-specific parameters, depending upon the demographic distribution of the sample in that dataset as well as the collection mode. Different weights for the domains stems from the fact that each symptom could mean something different across collection mode; ‘fever’ when individually reported through citizen efforts has a different predictive power as when it is collected in a standardized way collected via a healthworker. We use a simple logistic regression for this optimization.
As motivated, we consider the case of transferring information from multiple source data sets from different domains to a largely unlabelled target dataset. In each experiment three source datasets are used in entirety along with a small amount of labelled target data. Area under the ROC curve (AUC) metric is used to assess performance. While each of the datasets have a varied composition in terms of total number of observations and population demographics, we choose to use them all without any pre-processing, as these demonstrate real data set differences and will indicate model performance in such real-world situations. We compare results to five methods to specifically examine the benefit of the (1) hierarchical structure and (2) incorporation of population attributes: Target only (Target), Logistic Regression (LR), Frustratingly Easy Domain Adaptation, which is noted for extreme simplicity and was used previously on symptom data daume2009frustratingly ; rehman2018domain , just with symptoms (FEDA), and with both symptoms and population attributes (FEDA+p), Undirected Hierarchical Bayesian Domain adaptation without population attributes (Hier) and with population attributes (Hier+p).
5 Results and Discussion
Of the methods compared, Target and LR have the poorest performance (Table1). This makes sense, as a target-only model doesn’t incorporate any information from other domains or populations. And, LR doesn’t account for any population attributes. These methods also perform worse than the domain adaptation methods (FEDA and Hier). This indicates that there is domain-specific structure to the data. Finally, the methods that do account for population attributes perform the best. Generally the Hier+p method performs the best; this was studied more based on amount of labelled training data available (below). We also examined the learned parameters, finding that they are intuitive and generally interpretable. If a particular demographic attribute has information about the symptom predictivity, then the weight for it’s influence is high. For example, in the case of the Goviral data, we find that the weights of the gender parameters are close to each other, while the proportion of males to females who were positive for flu/colds is also close to 1. In situations of low amount of training data (e.g. in GoViral, for which there are no observations in the age group of 5-15), correspondingly the influence weight of those categories is low.
The amount of Target data used in Table 1 was chosen based on how performance varies by proportion labels available (Table 2). Results show that Hier+p generally worked better with under 25% labels indicating this approach is particularly suitable in cases of very limited training data. This makes sense, as when more labels are available, more information about features is available and less reliance on population-invariant information is needed. The model harnesses population invariant information from the other data sets (multi-source learning) and domain adaptation to improve prediction on a target when very little feature-specific information is available. Given these findings, we are interested in developing a generalizable framework for understanding how domain and population distribution differences affect results (e.g. Fluwatch poorer results at 10-15% target labels).
This work was supported in part by National Science Foundation grants 1643576 and 1551036.
Bradley P Carlin and Thomas A Louis.
Bayes and empirical Bayes methods for data analysis. Chapman and Hall/CRC, 2010.
-  Benjamin J Cowling, Kwok Hung Chan, Vicky J Fang, Lincoln LH Lau, Hau Chi So, Rita OP Fung, Edward SK Ma, Alfred SK Kwong, Chi-Wai Chan, Wendy WS Tsui, et al. Comparative epidemiology of pandemic and seasonal influenza a in households. New England journal of medicine, 362(23):2175–2184, 2010.
-  William Francis Darnieder. Bayesian methods for data-dependent priors. PhD thesis, The Ohio State University, 2011.
-  Hal Daume III. Frustratingly easy domain adaptation. arXiv preprint arXiv:0907.1815, 2009.
-  Gal Elidan, Ben Packer, Geremy Heitz, and Daphne Koller. Convex point estimation using undirected bayesian transfer hierarchies. arXiv preprint arXiv:1206.3252, 2012.
Theodoros Evgeniou, Charles A Micchelli, and Massimiliano Pontil.
Learning multiple tasks with kernel methods.
Journal of Machine Learning Research, 6(Apr):615–637, 2005.
-  Roger Fletcher and Michael JD Powell. A rapidly convergent descent method for minimization. The computer journal, 6(2):163–168, 1963.
-  Ellen B Fragaszy, Charlotte Warren-Gash, Lili Wang, Andrew Copas, Oliver Dukes, W John Edmunds, Nilu Goonetilleke, Gabrielle Harvey, Anne M Johnson, Jana Kovar, et al. Cohort profile: The flu watch study. International journal of epidemiology, 46(2):e18–e18, 2016.
-  Jennifer Goff, Aaron Rowe, John S Brownstein, and Rumi Chunara. Surveillance of acute respiratory infections using community-submitted symptoms and specimens for molecular diagnostic testing. PLoS currents, 7, 2015.
-  Mark Loeb, Margaret L Russell, Lorraine Moss, Kevin Fonseca, Julie Fox, David JD Earn, Fred Aoki, Gregory Horsman, Paul Van Caeseele, Khami Chokani, et al. Effect of influenza vaccination of children on infection rates in hutterite communities: a randomized trial. Jama, 303(10):943–950, 2010.
-  Andrew McCallum, Ronald Rosenfeld, Tom M Mitchell, and Andrew Y Ng. Improving text classification by shrinkage in a hierarchy of classes. In ICML, volume 98, pages 359–367, 1998.
-  Bisakha Ray and Rumi Chunara. Predicting acute respiratory infections from participatory data. Online journal of public health informatics, 9(1), 2017.
-  Nabeel Abdur Rehman, Maxwell Matthaios Aliapoulios, Disha Umarwani, and Rumi Chunara. Domain adaptation for infection prediction from symptoms based on data from different study designs and contexts. arXiv preprint arXiv:1806.08835, 2018.
-  Suchi Saria, Daphne Koller, and Anna Penn. Learning individual and population level traits from clinical temporal data. In Proceedings of Neural Information Processing Systems, pages 1–9. Citeseer, 2010.
-  Sara van Erp, Joris Mulder, and Daniel L Oberski. Prior sensitivity analysis in default bayesian structural equation modeling. American Psychological Association, 2017.