Online Semi-Supervised Concept Drift Detection with Density Estimation

by   Chang How Tan, et al.

Concept drift is formally defined as the change in joint distribution of a set of input variables X and a target variable y. The two types of drift that are extensively studied are real drift and virtual drift where the former is the change in posterior probabilities p(y|X) while the latter is the change in distribution of X without affecting the posterior probabilities. Many approaches on concept drift detection either assume full availability of data labels, y or handle only the virtual drift. In a streaming environment, the assumption of full availability of data labels, y is questioned. On the other hand, approaches that deal with virtual drift failed to address real drift. Rather than improving the state-of-the-art methods, this paper presents a semi-supervised framework to deal with the challenges above. The objective of the proposed framework is to learn from streaming environment with limited data labels, y and detect real drift concurrently. This paper proposes a novel concept drift detection method utilizing the densities of posterior probabilities in partially labeled streaming environments. Experimental results on both synthetic and realworld datasets show that our proposed semi-supervised framework enables the detection of concept drift in such environment while achieving comparable prediction performance to the state-of-the-art methods.



There are no comments yet.


page 1

page 2

page 3

page 4


Handling Adversarial Concept Drift in Streaming Data

Classifiers operating in a dynamic, real world environment, are vulnerab...

Concept Drift Detection and Adaptation with Weak Supervision on Streaming Unlabeled Data

Concept drift in learning and classification occurs when the statistical...

Concept Drift and Covariate Shift Detection Ensemble with Lagged Labels

In model serving, having one fixed model during the entire often life-lo...

Concept Drift Detection for Streaming Data

Common statistical prediction models often require and assume stationari...

Tackling Virtual and Real Concept Drifts: An Adaptive Gaussian Mixture Model

Real-world applications have been dealing with large amounts of data tha...

Reinforcement Evolutionary Learning Method for self-learning

In statistical modelling the biggest threat is concept drift which makes...

Adaptive Online Sequential ELM for Concept Drift Tackling

A machine learning method needs to adapt to over time changes in the env...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.