A promising opportunity in AI is developing systems that can partner with people to accomplish tasks in ways that exceed the capabilities of either individually (Wang et al., 2016; Kamar, 2016; Gaur et al., 2016). We see many motivating examples: a doctor using a medical expert system (Wang et al., 2016), a judge advised by a recidivism predictor, or a driver supervising a semi-autonomous vehicle. Despite rising interest, there is much to learn about creating effective human-AI teams and what capabilities AI systems should employ to be competent partners.
We study human-AI teams in decision-making settings where a user takes action recommendations from an AI partner for solving a complex task. The user considers the recommendation and, based on previous experience with the system, decides to accept the suggested action or take a different action. We call this type of interaction AI-advised human decision making. The motivation for AI-advised human decision making comes from the fact that humans and machines have complementary abilities (Wang et al., 2016; Kamar et al., 2012) and that AI assistance can speed up decision making when humans can correctly identify when the AI can be trusted (Lasecki et al., 2012a, b).
It might be expected that improvements in the performance of AI systems lead to stronger team performance, but, as with human groups, individual ability is only one of many factors that affect team effectiveness (DeChurch & Mesmer-Magnus, 2010; Grosz, 1996). In fact, the success of the team hinges on the human correctly deciding when to follow the recommendation of the AI system and when to override. If the human mistakenly trusts the AI system in regions where it is likely to err, catastrophic failures may occur. Human-AI teams become especially susceptible to such failures because of discrepancies introduced by system updates that do not account for human expectations. The following example illustrates this situation.
Example (Patient readmission).
A doctor uses an AI system that is 95% accurate at predicting whether a patient will be readmitted following their discharge to make decisions about enlisting the patient in a supportive post-discharge program. The special program is costly but promises to reduce the likelihood of readmission. After a year of interacting with the AI, the doctor develops a clear mental model that suggests she can trust the AI-advised actions on elderly patients. In the meantime, the AI’s developer deploys a new 98% accurate classifier, which errs on elderly patients.
While the AI has improved by 3%, the doctor is unaware of the new errors and might take the wrong actions for some elderly patients.
A doctor uses an AI system that is 95% accurate at predicting whether a patient will be readmitted following their discharge to make decisions about enlisting the patient in a supportive post-discharge program. The special program is costly but promises to reduce the likelihood of readmission. After a year of interacting with the AI, the doctor develops a clear mental model that suggests she can trust the AI-advised actions on elderly patients. In the meantime, the AI’s developer deploys a new 98% accurate classifier, which errs on elderly patients. While the AI has improved by 3%, the doctor is unaware of the new errors and might take the wrong actions for some elderly patients.
This example is motivated by real-world applications for reducing patient readmissions and other costly outcomes in healthcare (Bayati et al., 2014; Caruana et al., 2015), and motivates the need for reducing the cost of disruption caused by updates that violate users’ mental models. Similar challenges are observed in other human-AI collaboration scenarios such as during over-the-air updates in the Tesla autopilot (O’Cane, 2018), and are present in a variety of other settings when AI services consumed by third-party applications, are updated. Despite these problems, developers almost exclusively optimize for AI performance. Retraining techniques largely ignore important details about human-AI teaming, and the mental model that humans develop from interacting with the system. The goal of this work is to make the human factor a first-class consideration of AI updates. In summary, we make the following contributions:
We define the notion of compatibility of an AI update with the user’s mental model created from past experience. We then propose a practical adjustment to current ML (re)training algorithms — an additional differentiable term to the logarithmic loss — that improves compatibility during updates, and allows developers to explore the performance/compatibility tradeoff.
We introduce an open-source experimental platform for studying (i) how people model the error boundary of an AI teammate for an AI-advised decision-making task, and (ii) how they adapt to updates.
Using the platform, we perform user studies showing that updating an AI to increase accuracy, at the expense of compatibility, may degrade team performance. Moreover, experiments on three high-stakes classification tasks (recidivism prediction, in-hospital mortality prediction, and credit-risk assessment) demonstrate that: (i) current ML models are not inherently compatible, but (ii) flexible performance/compatibility tradeoffs can be effectively achieved via a reformulated training objective.
2 AI-Advised Human Decision Making
In our studies, we focus on a simple, but common, model of human-AI teamwork that abstracts many real-world settings, in which ML models support a human decision-maker. In this setting, which we call AI-advised human decision making, an AI system provides a recommendation, but the human makes the final decision. The team solves a sequence of tasks, repeating the following cycle for each time, .
The environment provides an input, .
The AI (possibly mistaken) suggests an action, .
Based on this input, the human makes her decision, .
The environment returns a reward, , which is a function of the user’s action, the (hidden) best action, and other costs of the human’s decision (e.g., time taken).
While interacting over multiple tasks, the team receives repeated feedback about performance, which lets the human learn when she can trust the AI’s answers. The cumulative reward over cycles records the team’s performance.
2.1 Trust as a Human’s Mental Model of the AI
Just as for other automated systems (Norman, 1988), humans create a mental model of AI agents (Kulesza et al., 2012). In AI-advised human decision making, valid mental models of the reliability of the AI output improve collaboration by helping the user to know when to trust the AI’s recommendation. A perfect mental model of the AI system’s reliability could be harnessed to achieve the highest team performance. A simple definition for such a model would be , indicating which inputs the human trusted the AI to solve correctly. In reality, mental models are not perfect (Norman, 2014): users develop them through limited interaction with the system, and people have cognitive limitations. Despite this, users learn and evolve a model of an AI system’s competence over the course of many interactions. In the full version of this paper (Bansal et al., 2019), we show that these models can greatly improve team performance. In this study, we focus on the problem of updating an AI system in a way that it remains compatible with users’ expectations.
3 Compatibility of Updates to Classifiers
In software engineering, an update is backward compatible if the updated system can support legacy software. By analogy, we define that an update to an AI component is locally compatible with a user’s mental model if it does not introduce new errors and the user, even after the update, can safely trust the AI’s recommendations.
Definition (Locally-Compatible Update).
Let denote a mental model that dictates the user’s trust of the AI on input . Let denote whether is the appropriate action for input . An update, , to a learned model, , is locally compatible with iff
In other words, an update is compatible only if, for every input where the user trusts the AI and recommends the correct action, the updated model, , also recommends the correct action. In the rest of this paper, we focus on situations where a classifier’s predictions are actions. For instance, in the patient readmission example, if a classifier predicts that the patient will be readmitted in the next 30 days, the suggested action from the classifier would be to include the patient in a special post-discharge program.
3.1 Globally Compatible Updates
When developers are building an AI system that is used by many individuals, it may be too difficult to track individual mental models or to deploy different updated models to different users. In this situation, an alternative to creating locally compatible updates, is a globally compatible update.
Definition (Globally-Compatible Update).
An updated model, , is globally compatible with , iff
Note that a globally compatible update is locally compatible for any mental model. While global compatibility is a nice ideal, satisfying it for all instances is difficult in practice. More realistically, we seek to minimize the number of errors made by that were not made by , since that will hopefully minimize confusion among users. To make this precise, we introduce the notion of a compatibility score.
Definition (Compatibility Score).
The compatibility score of an update to is given by the fraction of examples on which recommends the correct action, also recommends the correct action.
If introduces no new errors, will be 1. Conversely, if all the errors are new, the score will be 0.
3.2 Dissonance and Loss
To train classifiers, ML developers optimize for the predictive performance of by minimizing a classification loss that penalizes low performance. The equation below shows the negative logarithmic loss for binary classification – a commonly used training objective in ML.
Here, the probabilitydenotes the confidence of the classifier that recommendation is true, while is the true label for (i.e.,
). The negative log loss, like many other loss functions in machine learning, depends only on the true label and the confidence in prediction – it ignores the previous versions of the classifier and, hence, has no preference for compatibility. As a result, retraining using different data can lead to very different hypotheses, introduce new errors, and decrease the compatibility score. To alleviate this problem, we define a new loss functionexpressed as the sum of classification loss and dissonance.
The dissonance of to is a function that penalizes a low compatibility score. Furthermore, is differentiable.
Recall that is high when both and are correct (Eqn 1). Dissonance expresses the opposite notion: measuring if is correct ( denotes an indicator function) and penalizing by the degree to which is incorrect. Equation 3 defines the new loss.
Here, encodes the relative weight of dissonance, controlling the additional loss to be assigned to all new errors. We refer to this version as new-error dissonance. Just as with classification loss, there are other ways to realize dissonance. We explored two alternatives, which we refer to as imitation and strict imitation dissonance. Eqn 4 describes the imitation dissonance which measures the log loss between the prediction probabilities of and :
Eqn 4 is used in model distillation (Ba & Caruana, 2014; Hinton et al., 2015), where the aim is to train a shallower, less expensive model by imitating the probabilities of larger, accurate model. Unfortunately, has the effect of nudging to mimic ’s mistakes as well as its successes. Eqn 5 describes the strict imitation dissonance, which follows a similar intuition but it only adds the log loss between and when is correct.
4 Platform for Studying Human-AI Teams
How might we study the impact of AI accuracy, updates, compatibility, and mental models on the performance of AI-advised human decision making teams? Ideally, we would conduct user studies in real-world settings, varying parameters like the length of interaction, task and AI complexity, reward function, and the AI’s behavior. However, testing in real settings reduces or removes our ability to directly control the performance of the AI and it may largely measure experts’ differing experience in the domain, rather than their interactions with the AI. To control for human expertise and the centrality of mental modeling, we developed the Caja platform, which supports parameterized user studies in an assembly line domain that abstracts away the specifics of problem solving and focuses on understanding the effect of mental modeling on team success. Caja is designed such that no human is a task expert (nor can they become one). In fact, the true label of decision problems is randomly generated so that people cannot learn how to solve the task. However, humans can learn when their AI assistant, Marvin, succeeds and when it errs. Alongside, the human has access to a perfect problem-solving mechanism, which she can use (at extra cost) when she does not trust Marvin.
Caja is a web-based game, whose goal is to make classification decisions for a fixed number of box-like objects. For each object, the team follows the steps S1-S4 to decide whether the object is “defective” or not. In S1 a new object appears (e.g., blue square), in S2 the AI recommends a label (e.g., not-defective), in S3 the player chooses an action (e.g., accept or reject the AI recommendation), and in S4 the UI returns a reward and increments the game score. The objects are composed of many features, but only a subset of them are made human-visible. For example, visual properties like shape, color, and size are visible, but the contents are not. In contrast, the AI has access to all the features but may make errors. At the beginning of the game, users have no mental model of the AI’s error boundary. However, to achieve high scores, they must learn a model using feedback from step S4. Figure 1 shows a screenshot of the game at step S3.
Caja allows study designers to vary parameters, such as the number of objects, number human-visible features, reward function, AI accuracy, and complexity of perfect mental model (number of clauses and literals in the error boundary and stochasticity of errors). Further, it enables one to study the effects of updates to AI by allowing changes to these parameters at any time step. In the next section, we use Caja to answer various research questions.
We present experiments and results in two parts. First, using our platform, we conduct user studies to understand the impact of updates on team performance. Second, we simulate updates for three real-world, high-stakes domains and show how the retraining objective enables an explorable tradeoff between compatibility and performance.
5.1 User Studies
In user studies, we hired MTurk workers and directed them to the Caja platform. The task of workers was to form a team with an AI, named Marvin, and label a set of objects as “defective” or “not defective”. Following AI-advised human decision making, to label an object, a worker can either accept Marvin’s recommendation, which is initially correct 80% of the time, or use the “compute” option, which is a surrogate for the human doing the task herself perfectly but incurring an opportunity cost.
Table 1 summarizes the reward function used in our studies. The matrix is designed in a way that it imitates a high-stakes scenario, i.e., the monetary penalty for a wrong decision is much higher than the reward for a correct decision. Note that the expected value of a naive strategy (e.g., always “Compute” or always “Accept,” without considering the likelihood of Marvin’s correctness) is zero. The only way to get a higher score is by learning when to trust Marvin by playing the game. An error boundary, , is expressed as a conjunction of literals. For example, one possible error boundary would be , which means that Marvin errs at all objects that are blue and have a squared shape. Since many features can be used as literals, we chose them randomly to create isomorphic error boundaries. In the full paper (Bansal et al., 2019), we show that it is easier for workers to create a mental model for boundaries that have fewer literals and are not stochastic. Next, we show results from experiments that introduce updates.
Q1: Do more compatible updates lead to higher team performance than incompatible updates?
To study the impact of updates, we set the number of cycles to 150, and at the 75th cycle, update the classifier to a version that is 5% more accurate (80% 85%). Then, we divide the participants into three groups: same error boundary, compatible error boundary, and incompatible error boundary. The same error boundary group receives an update improving accuracy, but the error boundary is unchanged. For the two other groups, the number of literals (features) in the error boundary changes from two to three. The update for the compatible error boundary group introduces no new errors; for example, if before the update the error boundary was , after the update it may change to . For the incompatible error boundary group, the error boundary introduces new errors violating compatibility. Figure 2 summarizes our results. We also show the performance of workers if no update was introduced (dashed line). The graph demonstrates two main findings on the importance of compatibility. First, a more accurate but incompatible classifier results in lower team performance than a less accurate but compatible classifier (no update) because workers have to relearn the new incompatible error boundary. Second, compatible updates improve team performance. Moreover, the figure shows different stages during the interaction: the user learning the original error boundary, team stabilizes, update causes disruption, and performance stabilizes again.
5.2 Experiments with High-Stakes Domains
Datasets. To investigate whether a tradeoff exists between performance and compatibility of an update, we simulate updates to classifiers for three domains: recidivism prediction (Will a convict commit another crime?) (Angwin et al., 2016), in-hospital mortality prediction (Will a patient die in the hospital?) (Johnson et al., 2016; Harutyunyan et al., 2017), and credit risk assessment (Will a borrower fail to pay back?)111https://community.fico.com/s/explainable-machine-learning-challenge. We selected these high-stakes domains to highlight the potential cost of mistakes caused by incompatible updates in human-AI teams.
Q2: Do ML classifiers produce compatible updates?
For this experiment, we first train a classifier on 200 examples and note its performance. Next, we train another classifier on 5000 examples and note its performance and compatibility score. We train both classifiers by minimizing the negative log loss. Table 2
shows the performance (area under ROC) and compatibility averaged over 500 runs for logistic regression (LR) and multi-layer perceptron (MLP) classifiers. We find that trainingby just minimizing log loss does not ensure compatibility. For example, for logistic regression and the in-hospital mortality prediction task, the compatibility score is as low as 40%. That is, 60% of the instances where was correct are now violated.
Q3: Is there a tradeoff between the performance and the compatibility of an update to AI?
For Q3 (and Q4), we learn the second classifier by minimizing . As depends also on the first classifier, we make its prediction available to the learner. We vary and summarize the resulting performance and compatibility scores across different datasets for the logistic regression classifier in Figure 3 and for different definitions of dissonance (the full paper also provides results for MLP). The figure shows that there exists a tradeoff between the performance of and its compatibility to . This tradeoff is generally more flexible (flat) in the first half of the curves. This shows that, at the very least, one can choose to deploy a more compatible update without significant loss in accuracy. Although such updates are not fully compatible, they might still be relevant to be picked by the developer if the update is supported by efficient explanation techniques that can help users to better understand how the model has changed. In these cases, a more compatible update would also reduce the effort of user (re)training. In the second half, the tradeoff becomes more evident. High compatibility can sacrifice predictive performance. Look-up summaries similar to graphs shown in Figure 3 are an insightful tool for ML developers that can guide them select an accurate yet compatible model based on the specific domain requirements.
Q4: What is the relative performance of the different dissonance functions?
Figure 3 compares the performance of the new-error dissonance function () with the imitation-based dissonances ( and ). As anticipated, performs best on all three domains. The definitions inspired by model distillation, and , assume that is calibrated, and more accurate. Therefore, needs to remain faithful to only the correct regions of a less accurate model . If these assumptions are violated, overfits to non-calibrated confidence scores of , which hurts performance.
6 Discussion and Directions
This work showed that backward compatibility is an essential determinant of team performance, and developers should factor it in system design supported by guiding tools exploring the performance/compatibility tradeoff in a similar way as shown in Figure 3. Varying results in numerous models on the performance/compatibility spectrum. The decision to select the appropriate model depends on several factors, including the user ability to create a mental model, the cost of disruption, and the availability of other alternative approaches for minimizing disruption caused by updates. One such approach is to retrain the user, for example, by leveraging mechanisms from interpretable AI to explain the updated model to users or to explain differences between and . However, this may not always be practical: (1) in practice, developers may push updates frequently, and since re-training requires user’s additional time and effort, it may not be practical to subject experts to repeated re-training; (2) updates can arbitrarily change the decision boundary of a classifier, and as a result, require the user to re-learn a large number of changes; (3) re-training requires the developers to create an effective curriculum or generate a “change summary” based on the update. It is often impossible to compute such summaries in a human-interpretable way. Nevertheless, backward compatibility does not preclude retraining; these techniques are complementary to each other.
Another complementary approach is to share the AI’s confidence in the prediction. Well-calibrated confidence scores can help a user to decide when or how much to trust the system. Unfortunately, confidence scores of ML classifiers are often not calibrated (Nguyen et al., 2015) or a meaningful confidence definition may not exist.
7 Related Work
Prior fundamental work explored the importance of mental models for achieving high performance in group work (Grosz & Kraus, 1999), human-system collaboration (Rouse et al., 1992), and interface design (Carroll & Olson, 1988). Our work builds upon these foundations and studies the problem for AI-advised human decision making. Previous work (Zhou et al., 2017) investigated factors that affect user-system trust, e.g., model uncertainty and cognitive load. The platform proposed in this work enables human studies that can analyze the effect of such factors.
The field of software engineering has initially studied the problem of backward compatibility, seeking to design components that remain compatible with a larger software ecosystem after updates (Bosch, 2009; Spring, 2005). Machine learning research has explored related notions. Stability expresses the ability of a model to not significantly change its predictions given small changes in the training set (Bousquet & Elisseeff, 2001). Consistency, which has application in ML fairness, is a property of smooth classifiers, which output similar predictions for similar instances (Zhou et al., 2004). Catastrophic forgetting
is an anomalous behavior of neural network models that occurs when they are sequentially trained to perform multiple tasks and forget to solve earlier tasks over time(Kirkpatrick et al., 2017). While these concepts are fundamental for analyzing changing trends in continuously learned models, they do not consider human-AI team performance nor prior user experience. Related to our proposed retraining objective is the idea of cost-sensitive learning (Elkan, 2001), where different mistakes may cost differently; for example, false positives may be especially costly. In our case, the cost also depends on the behavior of the previous model .
We studied how updates to an AI system can affect human-AI team performance and introduced methods and measures for characterizing and addressing the compatability of updates. We introduced Caja, a platform for measuring the effect of AI performance and the effect of updates on team performance. Since humans have no experience with Caja’s abstract game, the platform controls for human problem-solving skill, distilling the essence of mental models and trust in one’s AI teammate. Using Caja, we presented experiments demonstrating how an update that makes an AI component more accurate can still lead to diminished human-AI team performance. We introduced a practical re-training objective that can improve the compatibility of updates. Experiments across three data sets show that our approach creates updates that are more compatible, while maintaining high accuracy. Therefore, at the very least, a developer can choose to deploy a more compatible model without sacrificing performance.
- Angwin et al. (2016) Angwin, J., Larson, J., Mattu, S., and Kirchner, L. Machine bias: There’s software across the country to predict future criminals and it’s biased against blacks. ProPublica, 2016.
- Ba & Caruana (2014) Ba, J. and Caruana, R. Do deep nets really need to be deep? In NIPS, pp. 2654–2662, 2014.
Bansal et al. (2019)
Bansal, G., Nushi, B., Kamar, E., Weld, D., Lasecki, W. S., and Horvitz, E.
Updates in human-ai teams: Understanding and addressing the
Proceedings of the 33rd AAAI conference on Artificial Intelligence. AAAI, 2019.
- Bayati et al. (2014) Bayati, M., Braverman, M., Gillam, M., Mack, K. M., Ruiz, G., Smith, M. S., and Horvitz, E. Data-driven decisions for reducing readmissions for heart failure: General methodology and case study. PloS one, 9(10):109264, 2014.
- Bosch (2009) Bosch, J. From software product lines to software ecosystems. In SPLC, pp. 111–119, 2009.
- Bousquet & Elisseeff (2001) Bousquet, O. and Elisseeff, A. Algorithmic stability and generalization performance. In NIPS, pp. 196–202, 2001.
- Carroll & Olson (1988) Carroll, J. M. and Olson, J. R. Mental models in human-computer interaction. In Handbook of human-computer interaction, pp. 45–65. Elsevier, 1988.
- Caruana et al. (2015) Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., and Elhadad, N. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In KDD, pp. 1721–1730. ACM, 2015.
- DeChurch & Mesmer-Magnus (2010) DeChurch, L. A. and Mesmer-Magnus, J. R. The cognitive underpinnings of effective teamwork: A meta-analysis. Journal of Applied Psychology, 95(1):32, 2010.
- Elkan (2001) Elkan, C. The foundations of cost-sensitive learning. In IJCAI, volume 17, pp. 973–978. Lawrence Erlbaum Associates Ltd, 2001.
- Gaur et al. (2016) Gaur, Y., Lasecki, W. S., Metze, F., and Bigham, J. P. The effects of automatic speech recognition quality on human transcription latency. In Proceedings of the 13th Web for All Conference (W4A), pp. 23. ACM, 2016.
- Grosz (1996) Grosz, B. J. Collaborative systems (AAAI-94 presidential address). AI magazine, 17(2):67, 1996.
- Grosz & Kraus (1999) Grosz, B. J. and Kraus, S. The evolution of sharedplans. In Foundations of rational agency. Springer, 1999.
- Harutyunyan et al. (2017) Harutyunyan, H., Khachatrian, H., Kale, D. C., and Galstyan, A. Multitask learning and benchmarking with clinical time series data. arXiv preprint arXiv:1703.07771, 2017.
- Hinton et al. (2015) Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
- Johnson et al. (2016) Johnson, A. E., Pollard, T. J., Shen, L., Li-wei, H. L., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Celi, L. A., and Mark, R. G. MIMIC-III, a freely accessible critical care database. Scientific data, 3:160035, 2016.
- Kamar (2016) Kamar, E. Directions in hybrid intelligence: Complementing AI systems with human intelligence. In IJCAI, 2016.
- Kamar et al. (2012) Kamar, E., Hacker, S., and Horvitz, E. Combining human and machine intelligence in large-scale crowdsourcing. In AAMAS, pp. 467–474. International Foundation for Autonomous Agents and Multiagent Systems, 2012.
- Kirkpatrick et al. (2017) Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, pp. 201611835, 2017.
- Kulesza et al. (2012) Kulesza, T., Stumpf, S., Burnett, M., and Kwan, I. Tell me more?: the effects of mental model soundness on personalizing an intelligent agent. In CHI, pp. 1–10. ACM, 2012.
- Lasecki et al. (2012a) Lasecki, W., Miller, C., Sadilek, A., Abumoussa, A., Borrello, D., Kushalnagar, R., and Bigham, J. Real-time captioning by groups of non-experts. In Proceedings of the 25th annual ACM symposium on User interface software and technology, pp. 23–34. ACM, 2012a.
- Lasecki et al. (2012b) Lasecki, W. S., Bigham, J. P., Allen, J. F., and Ferguson, G. Real-time collaborative planning with the crowd. 2012b.
- Nguyen et al. (2015) Nguyen, A., Yosinski, J., and Clune, J. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In CVPR, pp. 427–436, 2015.
- Norman (1988) Norman, D. The psychology of everyday things. Basic Books, 1988.
- Norman (2014) Norman, D. A. Some observations on mental models. In Mental models, pp. 15–22. Psychology Press, 2014.
- O’Cane (2018) O’Cane, S. Tesla can change so much with over-the-air updates that it’s messing with some owners’ heads. www.theverge.com/2018/6/2/17413732/tesla-over-the-air-software-updates-brakes, 2018.
- Rouse et al. (1992) Rouse, W. B., Cannon-Bowers, J. A., and Salas, E. The role of mental models in team performance in complex systems. IEEE Transactions on SMC, 22(6):1296–1308, 1992.
- Spring (2005) Spring, M. J. Techniques for maintaining compatibility of a software core module and an interacting module, November 29 2005. US Patent 6,971,093.
- Wang et al. (2016) Wang, D., Khosla, A., Gargeya, R., Irshad, H., and Beck, A. H. Deep learning for identifying metastatic breast cancer. arXiv preprint arXiv:1606.05718, 2016.
- Zhou et al. (2004) Zhou, D., Bousquet, O., Lal, T. N., Weston, J., and Schölkopf, B. Learning with local and global consistency. In NIPS, pp. 321–328, 2004.
- Zhou et al. (2017) Zhou, J., Arshad, S. Z., Luo, S., and Chen, F. Effects of uncertainty and cognitive load on user trust in predictive decision making. In IFIP Conference on Human-Computer Interaction, pp. 23–39. Springer, 2017.