Lumos: A Library for Diagnosing Metric Regressions in Web-Scale Applications

by   Jamie Pool, et al.

Web-scale applications can ship code on a daily to weekly cadence. These applications rely on online metrics to monitor the health of new releases. Regressions in metric values need to be detected and diagnosed as early as possible to reduce the disruption to users and product owners. Regressions in metrics can surface due to a variety of reasons: genuine product regressions, changes in user population, and bias due to telemetry loss (or processing) are among the common causes. Diagnosing the cause of these metric regressions is costly for engineering teams as they need to invest time in finding the root cause of the issue as soon as possible. We present Lumos, a Python library built using the principles of AB testing to systematically diagnose metric regressions to automate such analysis. Lumos has been deployed across the component teams in Microsoft's Real-Time Communication applications Skype and Microsoft Teams. It has enabled engineering teams to detect 100s of real changes in metrics and reject 1000s of false alarms detected by anomaly detectors. The application of Lumos has resulted in freeing up as much as 95 of the time allocated to metric-based investigations. In this work, we open source Lumos and present our results from applying it to two different components within the RTC group over millions of sessions. This general library can be coupled with any production system to manage the volume of alerting efficiently.


page 4

page 5

page 8


Greykite: Deploying Flexible Forecasting at Scale at LinkedIn

Forecasts help businesses allocate resources and achieve objectives. At ...

Moving Metric Detection and Alerting System at eBay

At eBay, there are thousands of product health metrics for different dom...

Practical data monitoring in the internet-services domain

Large-scale monitoring, anomaly detection, and root cause analysis of me...

ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric

Estimation of perceptual quality in audio and speech is possible using a...

Safely and Quickly Deploying New Features with a Staged Rollout Framework Using Sequential Test and Adaptive Experimental Design

During the rapid development cycle for Internet products (websites and m...

Finding Needle in a Million Metrics: Anomaly Detection in a Large-scale Computational Advertising Platform

Online media offers opportunities to marketers to deliver brand messages...

Silent Data Corruptions at Scale

Silent Data Corruption (SDC) can have negative impact on large-scale inf...

Please sign up or login with your details

Forgot password? Click here to reset