Human-in-the-loop Handling of Knowledge Drift

03/27/2021
by   Andrea Bontempelli, et al.
0

We introduce and study knowledge drift (KD), a complex form of drift that occurs in hierarchical classification. Under KD the vocabulary of concepts, their individual distributions, and the is-a relations between them can all change over time. The main challenge is that, since the ground-truth concept hierarchy is unobserved, it is hard to tell apart different forms of KD. For instance, introducing a new is-a relation between two concepts might be confused with individual changes to those concepts, but it is far from equivalent. Failure to identify the right kind of KD compromises the concept hierarchy used by the classifier, leading to systematic prediction errors. Our key observation is that in many human-in-the-loop applications (like smart personal assistants) the user knows whether and what kind of drift occurred recently. Motivated by this, we introduce TRCKD, a novel approach that combines automated drift detection and adaptation with an interactive stage in which the user is asked to disambiguate between different kinds of KD. In addition, TRCKD implements a simple but effective knowledge-aware adaptation strategy. Our simulations show that often a handful of queries to the user are enough to substantially improve prediction performance on both synthetic and realistic data.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/13/2020

Learning under Concept Drift: A Review

Concept drift describes unforeseeable changes in the underlying distribu...
research
06/01/2022

Federated Learning under Distributed Concept Drift

Federated Learning (FL) under distributed concept drift is a largely une...
research
04/01/2020

Handling Concept Drifts in Regression Problems – the Error Intersection Approach

Machine learning models are omnipresent for predictions on big data. One...
research
07/25/2017

Concept Drift Detection and Adaptation with Hierarchical Hypothesis Testing

In a streaming environment, there is often a need for statistical predic...
research
04/05/2021

Analyzing Flight Delay Prediction Under Concept Drift

Flight delays impose challenges that impact any flight transportation sy...
research
10/24/2021

Detecting model drift using polynomial relations

Machine learning (ML) models serve critical functions, such as classifyi...

Please sign up or login with your details

Forgot password? Click here to reset