Automatic structure estimation of predictive models for symptom development

09/12/2018
by   Dennis Becker, et al.
Leuphana Universität Lüneburg
0

Online mental health treatment has the premise to meet the increasing demand for mental health treatment at a lower cost than traditional treatment. However, online treatment suffers from high drop-out rates, which might negate their cost effectiveness. Predictive models might aid in early identification of deviating clients which allows to target them directly to prevent drop-out and improve treatment outcomes. We propose a two-staged multi-objective optimization process to automatically infer model structures based on ecological momentary assessment for prediction of future symptom development. The proposed multi-objective optimization approach results in a temporal-causal network model with the best prediction performance for each concept. This allows for a selection of a disorder-specific model structure based on the envisioned field of application.

READ FULL TEXT VIEW PDF

Authors

page 1

page 2

page 3

page 4

01/14/2022

Mental Health Assessment for the Chatbots

Previous researches on dialogue system assessment usually focus on the q...
09/10/2019

Multi-objective Evolutionary Approach to Grey-Box Identification of Buck Converter

The present study proposes a simple grey-box identification approach to ...
08/14/2019

Towards automated symptoms assessment in mental health

Activity and motion analysis has the potential to be used as a diagnosti...
10/29/2021

MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare

Mental health is a critical issue in modern society, and mental disorder...
12/30/2019

Model-Agnostic Approaches to Multi-Objective Simultaneous Hyperparameter Tuning and Feature Selection

Highly non-linear machine learning algorithms have the capacity to handl...
10/30/2020

Representation Learning for Integrating Multi-domain Outcomes to Optimize Individualized Treatments

For mental disorders, patients' underlying mental states are non-observe...
09/16/2019

Automatic Detection and Classification of Cognitive Distortions in Mental Health Text

In cognitive psychology, automatic and self-reinforcing irrational thoug...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Mental health disorders are an increasing problem in our society. In Europe, one in four people have reported that they suffered from at least one mental disorder during their lifetime [1] and it is estimated that in the USA approximately 26% of the population suffers from at least one mental disorder [18]. These numbers are alarming and create increasing costs for our society [24]. The costs arise from the direct treatment but also due to the indirect costs of the loss of productivity or the workplace [19]. The lost earnings in America are estimated to be $193.2 billion per year [16]. The global costs of mental health were estimated at $2.4 trillion, in 2010, and is predicted to increase to $8.5 trillion by 2030 [3].

As a result, online treatment gains an increasing importance for the delivery of mental health interventions [7, 9]. Online treatments utilize computerized cognitive behavioral therapy in form of brief therapy. These treatments are typically shorter than traditional therapy and specifically target a symptom or behavior. Besides providing more people with the opportunity for treatment, online treatment promises to be more cost-effective than traditional face-to-face therapy [32]. A therapist can supervise more clients in an online setting compared to a traditional face-to-face setting [10, 27], which lowers the intervention costs [14]. However, studies regarding the actual cost-effectiveness report mixed results [38, 12]. One reason might be the considerably high drop-out rates in online treatment [37]. A review study suggests that the drop-out rate of Internet-based treatment programs for psychological disorders ranges from 2 to 83% and with a weighted average of 31% [25]. An early identification of clients prone to drop-out might be crucial because dropped out clients are unlikely to recover and it negatively affects their attitude towards subsequent therapies [40].

Medical decision support systems could help in increasing the adherence and improving the outcome of online treatments [39]. In clinical psychology, the use of decision support systems has been evaluated and successfully used [35]. The application of the evaluated systems ranges from automated screening using health records [28], diagnosis support and treatment recommendations [33, 22], to medication planning and scheduled screening reminders for physicians [6]. Online treatment has the advantage that a multitude of different data sources is already digitally available. Several studies have shown that decision support systems can be useful for improving the quality of provided care, preventing errors, reducing finical costs, and saving human resources [17, 31]. Integration of decision support systems into online treatment can provide similar advantages. Clients that are prone to deteriorate or do not respond to the treatment can receive an early intervention by the supervising therapist. The therapist could provide the deviating clients with additional or a different set of interventions that are more suited to their needs. This would further strengthen the personal bond between the therapist and the client. An overall better treatment experience could also result in improved outcomes, which might reduce relapse risk and further treatment costs.

A regularly assessed measure that is suited for predictive model development is ecological momentary assessment (EMA) [41]. EMA can consist of ratings of moods, thoughts, symptoms, or behavioral patterns and is conducted while clients are engaged in their typical daily routine [41, 29]. Typically, smart-phones are used to record these measures. One particular type of model that allows to describe the interaction among EMA concepts is temporal-causal modelling [34]. Such models capture the knowledge from psychological theories and describe the model as a set of differential equations that represent the interaction among the psychological concepts. Based on this modeling approach, models for the emergence of depression due to stress factors or experience of extreme situations [4], the influence of different therapy strategies [5], and social integration [2] have been described.

Although these models provide the ability to capture psychological knowledge many choices are left to the person who designs them. Furthermore, such models have many free parameters that allow them to provide a close fit to a variety of observed data. This makes them prone to overfitting the data which might negatively affect the prediction of future EMA development. In this paper, we propose to use a two-staged multi-objective genetic algorithm to explore the space of models based on the predictive performance. Multi-objective optimization provides the possibility to analyze the predictive performance of each individual concept. Because for different mental disorders the prediction performance of a particular concept might be more relevant than others. For example, in the case of depression the course of the concept mood might be the most reliable but for anxiety, a different concept could be more relevant.

In the following section, we discuss commonly assessed EMA measures, the design of temporal-causal network models, and a method for multi-objective optimization. Following this, we present a description of our proposed two-stage optimization process for model structure estimation, as well as a plan for analyses and validation of the models estimated by our proposed approach. Finally, we illustrate our ideas for future research and list a number of potential limitations and difficulties with our current study.

2 Research Setting and Approach

In contrast to a model design that captures psychological knowledge and explains observed data, we aim to explore a model design that predicts future concept development. In this section, we describe typically assessed EMA data and the methods that we intend to use to explore predictive models in the application of future symptom development.

2.1 Ecological Momentary Assessment

EMA is part of a clinical and online treatment of mental disorders. Especially, in online treatment that is delivered using a smart-phone, it is easily implemented and inquired. EMA concepts are inquired by presenting the client a series of questions that are rated by the client on a numeric scale from 1 to 10. An overview of commonly assessed EMA concepts and the inquiry question is shown in Table 1.

EMA concept Assessment question
Mood How is your mood right now?
Worry

How much do you worry about things at the moment?

Self-Esteem How good do you feel about yourself right now?
Sleep How did you sleep tonight?
Activities done To what extent have you carried out enjoyable activities today?
Enjoyed activities How much have you enjoyed the days activities?
Social contact How much have you been involved in social interactions today?
Table 1: Example of commonly used EMA concepts

The amount of assessed EMA concepts and the number of inquiries varies among studies. One reason is that assessment can be done multiple times throughout the day for example in the morning and the evening, or only once each day. A reason to reduce the number of EMA inquirements is because a too high workload of EMA assessment might be considered as intrusive and clients might become reluctant to enter their EMA measures. Therefore, some online interventions do not assess all the concepts every day. Another point to consider is that in contrast to clients that use a standalone PC to process the interventions, smart-phones can provide a reminding function. This increases the likelihood of clients responding to these questions. Additionally, online treatments vary in length of active treatment, which can range from 4 to 15 weeks. This shows that there might be variability in the granularity and length of the available EMA data, also with respect to clients that drop-out early. Therefore, it is expected that the amount of available data varies among clients and contains many missing values.

2.2 Temporal-causal Network Models

Temporal-causal network models provide the possibility to represent biological, physical, or social networks, which inherently contain cycles. These models allow to capture the available knowledge in form of a network representation and provide simulations for each represented concept. The individual concepts, or nodes in this network, are connected by differential equations that describe the change of the concept in relation to the connected concepts. For example, the change of the concept Y can be described as the following equation:

(1)

Where the rate of change is defined by three components:

  • A weight that defines the strength of the causal relation to the concept.

  • The factor that defines the speed of change of the concept based on the causal impact.

  • A combining function that combines the causal impacts of connected concepts.

For the choice of combining function, a variety of standard combining functions exist. These standard combining functions are building blocks and can be used during the design of a model. These functions are based, for example, on the sum, product, max, min or a simple logistic function. However, some of these standard combining functions such as the simple logistic function require additional parameters. An example network representation for the EMA data is illustrated in Figure 1.

Figure 1: Possible network model for EMA interaction

Although humans tend to understand a network representation more intuitively, for a computer this representation is more difficult. But the model can also easily be represented as a matrix, as shown in Table 2, which makes it easier to automatically generate different models.

to Mood Worry Sleep Self-Esteem Social Enjoyed Activities
from contact activities done
Mood
Worry
Sleep
Self-Esteem
Social contact
Enjoyed activities
Activities done
Table 2: Matrix representation of the model shown in Figure 1

The matrix representation of the model shows clearly what parameters need to be estimated, and one can always switch between both representations. The network representation allows researchers to intuitively evaluate the created model, whereas the matrix representation is more suited for computational purposes.

2.3 Multi-objective Optimisation

For the two staged model structure optimization, we utilize the NSGA2 (nondominated sorting genetic algorithm II) [11]

optimization algorithm. It will be used for the estimation of the model structure and client specific model parameters. NSGA2 is a genetic algorithm for multi-objective optimization that estimates the parameters with respect to multiple objectives and can be used for any type of optimization. Evolutionary algorithms, in particular, have been proven to provide acceptable solutions and are well suited for discontinuous and multi-modal optimization problems 

[13, 8]. They have already been applied to a variety of real-world optimization tasks, such as design optimization [21, 23], circuit design [26, 30], and routing [20].

An evolutionary algorithm is a population-based search and optimization method that mimics the process of natural evolution [15]. Such algorithms start with an initial population, where each individual of the population is a solution to the problem. Generations are created successively based on the best solutions so far. In each generation of the algorithm, the parameters of the best individuals are merged to generate new solutions. This merging process of parameters is called crossover. The quality of these solutions is then estimated by evaluating the prediction error. The solutions with a higher prediction error are replaced by new solutions that provide a lower prediction error. To create additional diversity in this mechanism, mutation is introduced. Mutation applies small random changes to the parameters of a newly created solution during the crossover process. Typically, the search behavior of the algorithm can be summarized as exploration and exploitation. During exploration, large changes in the search space allow identifying new regions with a lower error on the cost function. Over the course of generations, the diversity of the population will be reduced and smaller changes allow to refine the already estimated solutions to further reduce the error.

However, NSGA2 has a one-dimensional representation of the parameters that are to be optimized. This representation might not perform well for our two-dimensional model matrix. Models that are similar might not be represented in terms of continuity of the parameter space. This could negatively affect the model structure search. Therefore, we consider integrating a two-dimensional parameter representation into NSGA2. This two-dimensional parameter encoding representation requires to replace the crossover and mutation operations and could provide more efficient exploration and optimization in the model space [36].

3 Model Estimation and Validation

In order to estimate a model that predicts future data well, we first have to define what time granularity is to be predicted. Since study data is likely to provide daily measures, we aim at the prediction of a concept value for each day. If multiple measures are available for each day, the average value is assumed as the measure. Next, we have to define for what time frame the predictions are supposed to be accurate.

For the evaluation of the model performance, the one-step-ahead prediction is often considered. This means that the concepts for the next day are predicted and the error is estimated accordingly. Then the measures of this day are used to adapt the model, and a prediction for the next day is given. However, for an online setting, the prediction of the following day might be too short. Typically, a supervising therapist could not check on each client every day. To ensure cost-effectiveness and reduce the effort in supervising many clients, it would be more likely that a therapist checks each or every second week on a client. In addition, it might be unlikely that a deterioration or drop-out happens overnight. It is more likely that a client’s attitude towards the treatment changes over the course of the interventions. Some online interventions advise the clients to complete at least one but preferably two interventions per week. This would make a prediction of a two-week time frame more reasonable to gauge the clients overall development. The length of the training time series required for learning of the model structure might depend on the number of available measures in each week. We assume that at least 3 weeks of data are necessary to learn the underlying process, where a longer time series might be beneficial.

To learn a model structure that is suited to predict future concept development, we separate the data into a training, test, and validation set. This separation allows to fit the model towards the test set, but at the same time remains some data for actual testing of the predictive performance of unseen data that has not been used for learning of the model structure. A model that accurately predicts the next week does not necessarily have to be the best model to predict the following two weeks in advance. It is also questionable if a model that predicts the second week of treatment is as reliable for the prediction after 10 weeks into the treatment. With progression in the treatment, the client might experience a reduction in symptoms and change in concepts might stabilize. However, we are focussing on the model structure because model parameters are likely to vary among clients and also over time. Therefore, based on the model structure individual client parameters need to be estimated.

Figure 2: Illustration of the two-staged model structure estimation approach

This suggests a two-stage approach, which is illustrated in Figure 2. Both stages utilize the NSGA2 multi-objective optimization algorithm, where the first stage estimates a model structure and the second stage client individual model parameters. In the first stage, the NSGA2 optimization algorithm is used to propose a variety of model structures. For these model structures, no particular weights are required, rather only the connections and combination functions are proposed in the first stage. Each model structure has a specific error for each concept. In the illustration of the approach, we only show two dimensions instead of all seven. The error of every concept has to be estimated for each client, which requires individual parameters for each client. These client-specific model parameters account for potentially time-dependent influences and lead to the second stage. In the second stage, parameters for the models are estimated for each client and the prediction error for each concept is estimated. The parameters are estimated from the training set, and the prediction error is estimated on the test set. Each parameter setting provides a different error for each concept. Therefore, we take the minimal error of each concept. These errors are from the same model but are results of different parameter settings, as indicated in the illustration of the approach. This leads to an estimation of an error for each client and each concept. The errors for each concept are summed up and reported back to the first stage. With these errors, the first stage algorithm can then continue the search for a model structure that reduces the overall prediction error on the test set.

We aim to learn a model structure that captures the dynamics of the data. This procedure might be beneficial because it aims at learning the underlying task for each concept and not a model structure for each client. An individual model structure for each client might also be misleading since the data quality among clients varies and contains noise. However, this process will not result in a single model, rather than in the number of the chosen population size of the first stage NSGA2 algorithm. To evaluate the resulting models, we consider the models that have the lowest prediction error in each concept. This results in 7 estimated models, where each model provides the lowest prediction error in one concept. The network representation of these models allows to compare them to existing knowledge about the theoretical interaction among these concepts and comparison of the structure of the resulting models might provide insight into the structural composition of these models. Furthermore, we evaluate the resulting prediction performance of these models on the validation set and compare the prediction performance against reference measures such as a regression model, dynamic linear models, and mean value prediction.

4 Concluding Remarks

In this short paper, we propose an approach for automated predictive model estimation based on EMA data. Specifically, we propose to apply multi-parameter optimization, which results in a model for each concept that provides the best prediction performance. This would allow to chose a model based on the intended purpose. For the case of depression, the mood development might be the most important whereas for social anxiety a prediction of social contact might be a more reliable indicator. By utilizing temporal-causal network models, the resulting models consist of a network structure that captures the psychological knowledge and underlying process. In contrast to other machine learning approaches such as neuronal networks or support vector machines, this makes it possible for humans to understand the resulting model and check the connections against existing results from psychological research. This allows to validate if the resulting models coincide with results from the psychological literature.

Regarding predictive model development for use in online treatment, estimation of a model structure is only the first step. After deriving promising model structures, the next step could be to research how these tools can be integrated into an online treatment setting, such that a supervising therapist can use them beneficially on a client individual level. Furthermore, a prediction of low concepts or rapid changes in the concepts have to be detected automatically, and it has to be analyzed how to derive treatment decisions from these predictions.

This research-in-progress study has a number of limitations. First, the proposed approach might be computational very demanding especially with an increasing number of clients. Because for each model that is proposed by the first stage genetic algorithm, the predictive performance for all clients needs to be estimated. This number would further increase if we assume that a client’s time series can be split into multiple chunks of the targeted two weeks time frame. To overcome this problem, one could consider an alternative route by estimating the model structure directly in one step. This approach would be straightforward, however, to avoid fully connected models, one has to add an additional complexity term that enforces sparse models. This complexity term might consist of the sum of the connection weights. Second, the amount of available EMA data depends on the clients’ compliance to the assessment protocol and treatment engagement. Therefore, the measurements for concepts might be sparse, which could complicate the model structure estimation. Finally, the prediction performance of the estimated models has to be superior to than the reference measures and provide reliable predictions. A model that does not provide reliable predictions about the future might provide insides into the interaction of EMA concepts but not be beneficial for deriving treatment decisions.

References

  • [1] J Alonso, M C Angermeyer, S Bernert, R Bruffaerts, T S Brugha, H Bryson, G de Girolamo, R Graaf, K Demyttenaere, I Gasquet, J M Haro, S J Katz, R C Kessler, V Kovess, J P Lépine, J Ormel, G Polidori, L J Russo, G Vilagut, J Almansa, S Arbabzadeh-Bouchez, J Autonell, M Bernal, M a Buist-Bouwman, M Codony, a Domingo-Salvany, M Ferrer, S S Joo, M Martínez-Alonso, H Matschinger, F Mazzi, Z Morgan, P Morosini, C Palacín, B Romera, N Taub, and W a M Vollebergh. Prevalence of mental disorders in Europe: results from the European Study of the Epidemiology of Mental Disorders (ESEMeD) project. Acta psychiatrica Scandinavica. Supplementum, 109(420):21–27, 2004.
  • [2] Michel Klein Altaf Hussain Abro. Validation of a Computational Model for Mood and Social Integration. Lecture Notes in Computer Science, 10046(November 2016):361–375, 2016.
  • [3] David E. Bloom, Elizabeth Cafiero, Eva Jané-Llopis, Shafika Abrahams-Gessel, Lakshmi Reddy Bloom, Sana Fathima, Andrea B. Feigl, Tom Gaziano, Ali Hamandi, Mona Mowafi, Danny O’Farrell, Emre Ozaltin, Ankur Pandya, Klaus Prettner, Larry Rosenberg, Benjamin Seligman, Adam Z. Stein, Cara Weinstein, and Jonathan Weiss. The Global Economic Burden of Noncommunicable Diseases. World Economic Forum, (September):1–46, 2011.
  • [4] Fiemke Both, Mark Hoogendoor, Michel Klein, and Jan Treur. Modeling the Dynamics of Mood and Depression. In M. Ghallab, C.D. Spyropoulos, N. Fakotakis, and N. Avouris, editors,

    Proceedings of the 18th European Conference on Artificial Intelligence, ECAI’08

    , pages 266–270, 2008.
  • [5] Fiemke Both and Mark Hoogendoorn. Utilization of a virtual patient model to enable tailored therapy for depressed patients. Neural Information Processing, pages 700–710, 2011.
  • [6] D. S. Cannon and S. N. Allen. A Comparison of the Effects of Computer and Manual Reminders on Compliance with a Mental Health Clinical Practice Guideline. Journal of the American Medical Informatics Association, 7(2):196–203, mar 2000.
  • [7] Helen Christensen and Kathleen M Griffiths. The prevention of depression using the internet. The Medical journal of Australia, 177 Suppl:S122—5, October 2002.
  • [8] Carlos A Coello Coello. An Updated Survey of Evolutionary Multiobjective Optimization Techniques : State of the Art and Future Trends.

    1999 Congress on Evolutionary Computation

    , 1(4):3–13, 1999.
  • [9] Pim Cuijpers, Straten A Van, Gerhard Andersson, Annemieke van Straten, Gerhard Andersson, Straten A Van, and Gerhard Andersson. Internet-administered cognitive behavior therapy for health problems: a systematic review. J Behav.Med, 31(0160-7715 (Print)):169–177, 2008.
  • [10] L. E. De Graaf, S a H Gerhards, A Arntz, H Riper, J F M Metsemakers, S M a a Evers, J L Severens, G Widdershoven, and M J H Huibers. Clinical effectiveness of online computerised cognitive-behavioural therapy without support for depression in primary care: Randomised trial. British Journal of Psychiatry, 195(1):73–80, jul 2009.
  • [11] Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and T. Meyarivan. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation, 6(2):182–197, 2002.
  • [12] T. Donker, M. Blankers, E. Hedman, B. Ljótsson, K. Petrie, and H. Christensen. Economic evaluations of Internet interventions for mental health: a systematic review. Psychological Medicine, (AUGUST):1–20, 2015.
  • [13] David E Goldberg, Kalyanmoy Deb, and James H Clark. Genetic Algorithms, Noise, and the Sizing of Populations. Complex Systems, 6:333–362, 1991.
  • [14] Erik Hedman, Erik Andersson, Brjánn Ljótsson, Gerhard Andersson, Christian Rück, and Nils Lindefors. Cost-effectiveness of Internet-based cognitive behavior therapy vs. cognitive behavioral group therapy for social anxiety disorder: Results from a randomized controlled trial. Behaviour Research and Therapy, 49(11):729–736, 2011.
  • [15] John H. Holland. Adaptation in Natural and Artificial Systems. University of Michigan Press, Ann Arbor, MI, USA, 1975.
  • [16] Tr Insel. Assessing the economic costs of serious mental illness. Am. J. Psychiat., 165(6):663–665, 2008.
  • [17] K. Kawamoto, C. A. Houlihan, E. A. Balas, and D. F. Lobach. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ (Clinical research ed.), 330(7494):765, 2005.
  • [18] Rc Kessler and Wt Chiu. Prevalence, Severity, and Comorbidity of Twelve-month DSM-IV Disorders in the National Comorbidity Survey Replication (NCS- R). Archives of general psychiatry, 62(6):617–627, 2005.
  • [19] Ronald C Kessler, Steven Heeringa, Matthew D Lakoma, Maria Petukhova, Agnes E Rupp, Michael Schoenbaum, Philip S Wang, and Alan M Zaslavsky. Individual and societal effects of mental disorders on earnings in the United States: results from the national comorbidity survey replication. The American journal of psychiatry, 165(6):703–11, 2008.
  • [20] J D Knowles and D W Corne. Heuristics for Evolutionary Off-line Routing in Telecommunications Networks. In Proceedings of the Genetic and Evolutionary Computation Conference, number January, pages 574–581, 2000.
  • [21] Oliver König and G Fadel. Application of genetic algorithms in the design of multi-material structures manufactured in rapid prototyping. In Solid Freeform Fabrication Symposium, pages 209–218, 1999.
  • [22] Benji T Kurian, Madhukar H Trivedi, Bruce D Grannemann, Cynthia a Claassen, Ella J Daly, and Prabha Sunderajan. A computerized decision support system for depression in primary care. Primary care companion to the Journal of clinical psychiatry, 11(4):140–146, 2009.
  • [23] Jongsoo Lee and Prabhat Hajela. Parallel genetic algorithm implementation in multidisciplinary rotor blade design. Journal of Aircraft, 33(5):962–969, 1996.
  • [24] Jean Pierre Lépine and Mike Briley. The increasing burden of depression. Neuropsychiatric Disease and Treatment, 7(SUPPL.):3–7, 2011.
  • [25] Katherine M Melville, Leanne M Casey, and David J Kavanagh. Dropout from Internet-based treatment for psychological disorders. The British journal of clinical psychology / the British Psychological Society, 49(Pt 4):455–71, nov 2010.
  • [26] JF Miller, P Thomson, and T Fogarty. Designing electronic circuits using evolutionary algorithms. arithmetic circuits: A case study. Genetic algorithms and evolution strategies in engineering and computer science, 8, 1997.
  • [27] J. Proudfoot, D. Goldberg, A. Mann, B. Everitt, I. Marks, and J. A. Gray. Computerized, interactive, multimedia cognitive-behavioural program for anxiety and depression in general practice. Psychological Medicine, 33(2):S0033291702007225, 2003.
  • [28] Bruce L. Rollman, Barbara H. Hanusa, Trae Gilbert, Henry J. Lowe, Wishwa N. Kapoor, and Herbert C. Schulberg. The Electronic Medical Record. Archives of Internal Medicine, 161(2):189, 2001.
  • [29] Saul Shiffman, Arthur A. Stone, and Michael R. Hufford. Ecological momentary assessment. Annual review of clinical psychology, 4(5):1–32, 2008.
  • [30] Parisa Soleimani, Reza Sabbaghi-Nadooshan, Sattar Mirzakuchaki, and Mahdi Bagheri. Using Genetic Algorithm in the Evolutionary Design of Sequential Logic Circuits. IJCSI International Journal of Computer Science Issues, 8(3):1–7, 2011.
  • [31] E.W. Steyerberg. Applications of prediction models. In Springer, volume 36, pages 11–31. 2009.
  • [32] Deborah F. Tate, Eric A. Finkelstein, Olga Khavjou, and Alison Gustafson. Cost effectiveness of internet interventions: Review and recommendations. Annals of Behavioral Medicine, 38(1):40–45, 2009.
  • [33] H V Thomas, G Lewis, M Watson, T Bell, I Lyons, K Lloyd, S Weich, and D Sharp. Computerised patient-specific guidelines for management of common mental disorders in primary care: A randomised controlled trial. British Journal of General Practice, 54(508):832–837, 2004.
  • [34] Jan Treur. Dynamic modeling based on a temporal-causal network modeling approach. Biologically Inspired Cognitive Architectures, 16(April):131–168, 2016.
  • [35] Yolanda Triñanes, Gerardo Atienza, Arturo Louro-González, Elena De-las Heras-Liñero, María Alvarez-Ariza, and Diego J. Palao. Development and impact of computerised decision support systems for clinical management of depression: A systematic review. Revista de Psiquiatría y Salud Mental (English Edition), 8(3):157–166, 2015.
  • [36] Ming Wen Tsai, Tzung Pei Hong, and Woo Tsong Lin. A two-dimensional genetic algorithm and its application to aircraft scheduling problem. Mathematical Problems in Engineering, 2015, 2015.
  • [37] Wouter Van Ballegooijen, Pim Cuijpers, Annemieke Van Straten, Eirini Karyotaki, Gerhard Andersson, Jan H. Smit, and Heleen Riper. Adherence to internet-based and face-to-face cognitive behavioural therapy for depression: A meta-analysis. PLoS ONE, 9(7), 2014.
  • [38] Sylvia Van Beugen, Maaike Ferwerda, Dane Hoeve, Maroeska M. Rovers, Saskia Spillekom-Van Koulil, Henri??t Van Middendorp, and Andrea W M Evers. Internet-based cognitive behavioral therapy for patients with chronic somatic conditions: A meta-analytic review. Journal of Medical Internet Research, 16(3):1–15, 2014.
  • [39] F Randy Vogenberg. Predictive and Prognostic Models: Implications for Healthcare Decision-Making in a Modern Recession. Am Health Drug Benefits, 2(6), 2009.
  • [40] Kamila S White, Laura B Allen, David H Barlow, Jack M Gorman, M Katherine Shear, and Scott W Woods. Attrition in a multicenter clinical trial for panic disorder. The Journal of nervous and mental disease, 198(9):665–671, 2010.
  • [41] M. Wichers, C. J P Simons, I. M A Kramer, J. A. Hartmann, C. Lothmann, I. Myin-Germeys, A. L. van Bemmel, F. Peeters, Ph Delespaul, and J. van Os. Momentary assessment technology as a tool to help patients with depression help themselves. Acta Psychiatrica Scandinavica, 124(4):262–272, 2011.