DeepAI
Log In Sign Up

Concept Drift Monitoring and Diagnostics of Supervised Learning Models via Score Vectors

12/12/2020
by   Kungang Zhang, et al.
0

Supervised learning models are one of the most fundamental classes of models. Viewing supervised learning from a probabilistic perspective, the set of training data to which the model is fitted is usually assumed to follow a stationary distribution. However, this stationarity assumption is often violated in a phenomenon called concept drift, which refers to changes over time in the predictive relationship between covariates 𝐗 and a response variable Y and can render trained models suboptimal or obsolete. We develop a comprehensive and computationally efficient framework for detecting, monitoring, and diagnosing concept drift. Specifically, we monitor the Fisher score vector, defined as the gradient of the log-likelihood for the fitted model, using a form of multivariate exponentially weighted moving average, which monitors for general changes in the mean of a random vector. In spite of the substantial performance advantages that we demonstrate over popular error-based methods, a score-based approach has not been previously considered for concept drift monitoring. Advantages of the proposed score-based framework include applicability to any parametric model, more powerful detection of changes as shown in theory and experiments, and inherent diagnostic capabilities for helping to identify the nature of the changes.

READ FULL TEXT

page 1

page 2

page 3

page 4

12/25/2012

Exponentially Weighted Moving Average Charts for Detecting Concept Drift

Classifying streaming data requires the development of methods which are...
12/21/2020

Nonstationarity Analysis of Materials Microstructures via Fisher Score Vectors

Microstructures are critical to the physical properties of materials. St...
10/10/2022

A Hybrid Active-Passive Approach to Imbalanced Nonstationary Data Stream Classification

In real-world applications, the process generating the data might suffer...
06/23/2020

Counterfactual Explanations of Concept Drift

The notion of concept drift refers to the phenomenon that the distributi...
03/21/2022

From Concept Drift to Model Degradation: An Overview on Performance-Aware Drift Detectors

The dynamicity of real-world systems poses a significant challenge to de...
10/06/2020

Supervised Seeded Iterated Learning for Interactive Language Learning

Language drift has been one of the major obstacles to train language mod...
02/02/2021

Drift Estimation with Graphical Models

This paper deals with the issue of concept drift in supervised machine l...