DeepAI AI Chat
Log In Sign Up

Deep Confidence: A Computationally Efficient Framework for Calculating Reliable Errors for Deep Neural Networks

by   Isidro Cortes-Ciriano, et al.

Deep learning architectures have proved versatile in a number of drug discovery applications, including the modelling of in vitro compound activity. While controlling for prediction confidence is essential to increase the trust, interpretability and usefulness of virtual screening models in drug discovery, techniques to estimate the reliability of the predictions generated with deep learning networks remain largely underexplored. Here, we present Deep Confidence, a framework to compute valid and efficient confidence intervals for individual predictions using the deep learning technique Snapshot Ensembling and conformal prediction. Specifically, Deep Confidence generates an ensemble of deep neural networks by recording the network parameters throughout the local minima visited during the optimization phase of a single neural network. This approach serves to derive a set of base learners (i.e., snapshots) with comparable predictive power on average, that will however generate slightly different predictions for a given instance. The variability across base learners and the validation residuals are in turn harnessed to compute confidence intervals using the conformal prediction framework. Using a set of 24 diverse IC50 data sets from ChEMBL 23, we show that Snapshot Ensembles perform on par with Random Forest (RF) and ensembles of independently trained deep neural networks. In addition, we find that the confidence regions predicted using the Deep Confidence framework span a narrower set of values. Overall, Deep Confidence represents a highly versatile error prediction framework that can be applied to any deep learning-based application at no extra computational cost.


page 17

page 27

page 28


Reliable Prediction Errors for Deep Neural Networks Using Test-Time Dropout

While the use of deep learning in drug discovery is gaining increasing a...

Concepts and Applications of Conformal Prediction in Computational Drug Discovery

Estimating the reliability of individual predictions is key to increase ...

Confident Neural Network Regression with Bootstrapped Deep Ensembles

With the rise of the popularity and usage of neural networks, trustworth...

Likelihood-ratio-based confidence intervals for neural networks

This paper introduces a first implementation of a novel likelihood-ratio...

Locally Valid and Discriminative Confidence Intervals for Deep Learning Models

Crucial for building trust in deep learning models for critical real-wor...

Reliable Probability Intervals For Classification Using Inductive Venn Predictors Based on Distance Learning

Deep neural networks are frequently used by autonomous systems for their...