Automatic Model Monitoring for Data Streams

by   Fábio Pinto, et al.

Detecting concept drift is a well known problem that affects production systems. However, two important issues that are frequently not addressed in the literature are 1) the detection of drift when the labels are not immediately available; and 2) the automatic generation of explanations to identify possible causes for the drift. For example, a fraud detection model in online payments could show a drift due to a hot sale item (with an increase in false positives) or due to a true fraud attack (with an increase in false negatives) before labels are available. In this paper we propose SAMM, an automatic model monitoring system for data streams. SAMM detects concept drift using a time and space efficient unsupervised streaming algorithm and it generates alarm reports with a summary of the events and features that are important to explain it. SAMM was evaluated in five real world fraud detection datasets, each spanning periods up to eight months and totaling more than 22 million online transactions. We evaluated SAMM using human feedback from domain experts, by sending them 100 reports generated by the system. Our results show that SAMM is able to detect anomalous events in a model life cycle that are considered useful by the domain experts. Given these results, SAMM will be rolled out in a next version of Feedzai's Fraud Detection solution.


page 1

page 2

page 3

page 4


Class Distribution Monitoring for Concept Drift Detection

We introduce Class Distribution Monitoring (CDM), an effective concept-d...

MemStream: Memory-Based Anomaly Detection in Multi-Aspect Streams with Concept Drift

Given a stream of entries over time in a multi-aspect data setting where...

On the Reliable Detection of Concept Drift from Streaming Unlabeled Data

Classifiers deployed in the real world operate in a dynamic environment,...

STUDD: A Student-Teacher Method for Unsupervised Concept Drift Detection

Concept drift detection is a crucial task in data stream evolving enviro...

Feature Relevance Analysis to Explain Concept Drift – A Case Study in Human Activity Recognition

This article studies how to detect and explain concept drift. Human acti...

Unsupervised Detection of Behavioural Drifts with Dynamic Clustering and Trajectory Analysis

Real-time monitoring of human behaviours, especially in e-Health applica...

ARMS: Automated rules management system for fraud detection

Fraud detection is essential in financial services, with the potential o...

Please sign up or login with your details

Forgot password? Click here to reset