Lightweight Automated Feature Monitoring for Data Streams

by   João Conde, et al.

Monitoring the behavior of automated real-time stream processing systems has become one of the most relevant problems in real world applications. Such systems have grown in complexity relying heavily on high dimensional input data, and data hungry Machine Learning (ML) algorithms. We propose a flexible system, Feature Monitoring (FM), that detects data drifts in such data sets, with a small and constant memory footprint and a small computational cost in streaming applications. The method is based on a multi-variate statistical test and is data driven by design (full reference distributions are estimated from the data). It monitors all features that are used by the system, while providing an interpretable features ranking whenever an alarm occurs (to aid in root cause analysis). The computational and memory lightness of the system results from the use of Exponential Moving Histograms. In our experimental study, we analyze the system's behavior with its parameters and, more importantly, show examples where it detects problems that are not directly related to a single feature. This illustrates how FM eliminates the need to add custom signals to detect specific types of problems and that monitoring the available space of features is often enough.


page 1

page 9


Drifter: Efficient Online Feature Monitoring for Improved Data Integrity in Large-Scale Recommendation Systems

Real-world production systems often grapple with maintaining data qualit...

Autonomous Deep Quality Monitoring in Streaming Environments

The common practice of quality monitoring in industry relies on manual i...

Attentive monitoring of multiple video streams driven by a Bayesian foraging strategy

In this paper we shall consider the problem of deploying attention to su...

Robust real-time monitoring of high-dimensional data streams

Robust real-time monitoring of high-dimensional data streams has many im...

Bandit Data-driven Optimization: AI for Social Good and Beyond

The use of machine learning (ML) systems in real-world applications enta...

Safely and Quickly Deploying New Features with a Staged Rollout Framework Using Sequential Test and Adaptive Experimental Design

During the rapid development cycle for Internet products (websites and m...

Detecting and Ranking Causal Anomalies in End-to-End Complex System

With the rapid development of technology, the automated monitoring syste...

Please sign up or login with your details

Forgot password? Click here to reset