Online Semi-Supervised Concept Drift Detection with Density Estimation

09/25/2019
by   Chang How Tan, et al.
0

Concept drift is formally defined as the change in joint distribution of a set of input variables X and a target variable y. The two types of drift that are extensively studied are real drift and virtual drift where the former is the change in posterior probabilities p(y|X) while the latter is the change in distribution of X without affecting the posterior probabilities. Many approaches on concept drift detection either assume full availability of data labels, y or handle only the virtual drift. In a streaming environment, the assumption of full availability of data labels, y is questioned. On the other hand, approaches that deal with virtual drift failed to address real drift. Rather than improving the state-of-the-art methods, this paper presents a semi-supervised framework to deal with the challenges above. The objective of the proposed framework is to learn from streaming environment with limited data labels, y and detect real drift concurrently. This paper proposes a novel concept drift detection method utilizing the densities of posterior probabilities in partially labeled streaming environments. Experimental results on both synthetic and realworld datasets show that our proposed semi-supervised framework enables the detection of concept drift in such environment while achieving comparable prediction performance to the state-of-the-art methods.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/24/2018

Handling Adversarial Concept Drift in Streaming Data

Classifiers operating in a dynamic, real world environment, are vulnerab...
research
10/02/2019

Concept Drift Detection and Adaptation with Weak Supervision on Streaming Unlabeled Data

Concept drift in learning and classification occurs when the statistical...
research
08/16/2021

Task-Sensitive Concept Drift Detector with Constraint Embedding

Detecting drifts in data is essential for machine learning applications,...
research
02/11/2021

Tackling Virtual and Real Concept Drifts: An Adaptive Gaussian Mixture Model

Real-world applications have been dealing with large amounts of data tha...
research
10/07/2018

Reinforcement Evolutionary Learning Method for self-learning

In statistical modelling the biggest threat is concept drift which makes...
research
07/25/2017

Concept Drift Detection and Adaptation with Hierarchical Hypothesis Testing

In a streaming environment, there is often a need for statistical predic...
research
03/13/2020

DriftSurf: A Risk-competitive Learning Algorithm under Concept Drift

When learning from streaming data, a change in the data distribution, al...

Please sign up or login with your details

Forgot password? Click here to reset