Active Learning++: Incorporating Annotator's Rationale using Local Model Explanation

09/06/2020
by   Bhavya Ghai, et al.
12

We propose a new active learning (AL) framework, Active Learning++, which can utilize an annotator's labels as well as its rationale. Annotators can provide their rationale for choosing a label by ranking input features based on their importance for a given query. To incorporate this additional input, we modified the disagreement measure for a bagging-based Query by Committee (QBC) sampling strategy. Instead of weighing all committee models equally to select the next instance, we assign higher weight to the committee model with higher agreement with the annotator's ranking. Specifically, we generated a feature importance-based local explanation for each committee model. The similarity score between feature rankings provided by the annotator and the local model explanation is used to assign a weight to each corresponding committee model. This approach is applicable to any kind of ML model using model-agnostic techniques to generate local explanation such as LIME. With a simulation study, we show that our framework significantly outperforms a QBC based vanilla AL framework.

READ FULL TEXT

page 1

page 2

page 3

07/31/2017

Interpretable Active Learning

Active learning has long been a topic of study in machine learning. Howe...
12/27/2021

Active Learning with Pseudo-Labels for Multi-View 3D Pose Estimation

Pose estimation of the human body/hand is a fundamental problem in compu...
08/06/2014

When does Active Learning Work?

Active Learning (AL) methods seek to improve classifier performance when...
10/27/2019

An Active Approach for Model Interpretation

Model interpretation, or explanation of a machine learning classifier, a...
10/26/2020

Q-FIT: The Quantifiable Feature Importance Technique for Explainable Machine Learning

We introduce a novel framework to quantify the importance of each input ...
01/24/2020

Explainable Active Learning (XAL): An Empirical Study of How Local Explanations Impact Annotator Experience

Active Learning (AL) is a human-in-the-loop Machine Learning paradigm fa...
04/29/2020

Valid Explanations for Learning to Rank Models

Learning-to-rank (LTR) is a class of supervised learning techniques that...