Concept Drift Detection via Equal Intensity k-means Space Partitioning

Data stream poses additional challenges to statistical classification tasks because distributions of the training and target samples may differ as time passes. Such distribution change in streaming data is called concept drift. Numerous histogram-based distribution change detection methods have been proposed to detect drift. Most histograms are developed on grid-based or tree-based space partitioning algorithms which makes the space partitions arbitrary, unexplainable, and may cause drift blind-spots. There is a need to improve the drift detection accuracy for histogram-based methods with the unsupervised setting. To address this problem, we propose a cluster-based histogram, called equal intensity k-means space partitioning (EI-kMeans). In addition, a heuristic method to improve the sensitivity of drift detection is introduced. The fundamental idea of improving the sensitivity is to minimize the risk of creating partitions in distribution offset regions. Pearson's chi-square test is used as the statistical hypothesis test so that the test statistics remain independent of the sample distribution. The number of bins and their shapes, which strongly influence the ability to detect drift, are determined dynamically from the sample based on an asymptotic constraint in the chi-square test. Accordingly, three algorithms are developed to implement concept drift detection, including a greedy centroids initialization algorithm, a cluster amplify-shrink algorithm, and a drift detection algorithm. For drift adaptation, we recommend retraining the learner if a drift is detected. The results of experiments on synthetic and real-world datasets demonstrate the advantages of EI-kMeans and show its efficacy in detecting concept drift.

READ FULL TEXT

page 1

page 11

page 12

page 14

research
10/16/2022

Class Distribution Monitoring for Concept Drift Detection

We introduce Class Distribution Monitoring (CDM), an effective concept-d...
research
05/04/2021

Automatic Learning to Detect Concept Drift

Many methods have been proposed to detect concept drift, i.e., the chang...
research
03/31/2017

On the Reliable Detection of Concept Drift from Streaming Unlabeled Data

Classifiers deployed in the real world operate in a dynamic environment,...
research
03/13/2020

DriftSurf: A Risk-competitive Learning Algorithm under Concept Drift

When learning from streaming data, a change in the data distribution, al...
research
04/03/2022

A Computational Analysis of Pitch Drift in Unaccompanied Solo Singing using DBSCAN Clustering

Unaccompanied vocalists usually change the tuning unintentionally and en...
research
10/04/2021

DenDrift: A Drift-Aware Algorithm for Host Profiling

Detecting and reacting to unauthorized actions is an essential task in s...
research
07/27/2022

Detecting Concept Drift in the Presence of Sparsity – A Case Study of Automated Change Risk Assessment System

Missing values, widely called as sparsity in literature, is a common cha...

Please sign up or login with your details

Forgot password? Click here to reset