Why I'm not Answering

by   Sayera Dhaubhadel, et al.

Safe deployment of deep learning systems in critical real world applications requires models to make few mistakes, and only under predictable circumstances. Development of such a model is not yet possible, in general. In this work, we address this problem with an abstaining classifier tuned to have >95% accuracy, and identify the determinants of abstention with LIME (the Local Interpretable Model-agnostic Explanations method). Essentially, we are training our model to learn the attributes of pathology reports that are likely to lead to incorrect classifications, albeit at the cost of reduced sensitivity. We demonstrate our method in a multitask setting to classify cancer pathology reports from the NCI SEER cancer registries on six tasks of greatest importance. For these tasks, we reduce the classification error rate by factors of 2–5 by abstaining on 25–45% of the reports. For the specific case of cancer site, we are able to identify metastasis and reports involving lymph nodes as responsible for many of the classification mistakes, and that the extent and types of mistakes vary systematically with cancer site (eg. breast, lung, and prostate). When combining across three of the tasks, our model classifies 50% of the reports with an accuracy greater than 95% for three of the six tasks and greater than 85% for all six tasks on the retained samples. By using this information, we expect to define work flows that incorporate machine learning only in the areas where it is sufficiently robust and accurate, saving human attention to areas where it is required.


Hierarchical Deep Learning Ensemble to Automate the Classification of Breast Cancer Pathology Reports by ICD-O Topography

Like most global cancer registries, the National Cancer Registry in Sout...

Hierarchical Deep Learning Classification of Unstructured Pathology Reports to Automate ICD-O Morphology Grading

Timely cancer reporting data are required in order to understand the imp...

EMT-NET: Efficient multitask network for computer-aided diagnosis of breast cancer

Deep learning-based computer-aided diagnosis has achieved unprecedented ...

Classification of cancer pathology reports: a large-scale comparative study

We report about the application of state-of-the-art deep learning techni...

Learning Permutation Invariant Representations using Memory Networks

Many real world tasks such as 3D object detection and high-resolution im...

Model Reduction of Shallow CNN Model for Reliable Deployment of Information Extraction from Medical Reports

Shallow Convolution Neural Network (CNN) is a time-tested tool for the i...

Please sign up or login with your details

Forgot password? Click here to reset