Did the Model Change? Efficiently Assessing Machine Learning API Shifts

by   Lingjiao Chen, et al.

Machine learning (ML) prediction APIs are increasingly widely used. An ML API can change over time due to model updates or retraining. This presents a key challenge in the usage of the API because it is often not clear to the user if and how the ML model has changed. Model shifts can affect downstream application performance and also create oversight issues (e.g. if consistency is desired). In this paper, we initiate a systematic investigation of ML API shifts. We first quantify the performance shifts from 2020 to 2021 of popular ML APIs from Google, Microsoft, Amazon, and others on a variety of datasets. We identified significant model shifts in 12 out of 36 cases we investigated. Interestingly, we found several datasets where the API's predictions became significantly worse over time. This motivated us to formulate the API shift assessment problem at a more fine-grained level as estimating how the API model's confusion matrix changes over time when the data distribution is constant. Monitoring confusion matrix shifts using standard random sampling can require a large number of samples, which is expensive as each API call costs a fee. We propose a principled adaptive sampling algorithm, MASA, to efficiently estimate confusion matrix shifts. MASA can accurately estimate the confusion matrix shifts in commercial ML APIs using up to 90 random sampling. This work establishes ML API shifts as an important problem to study and provides a cost-effective approach to monitor such shifts.


page 3

page 8

page 32


Improving the Learnability of Machine Learning APIs by Semi-Automated API Wrapping

A major hurdle for students and professional software developers who wan...

WILDS: A Benchmark of in-the-Wild Distribution Shifts

Distribution shifts can cause significant degradation in a broad range o...

FrugalML: How to Use ML Prediction APIs More Accurately and Cheaply

Prediction APIs offered for a fee are a fast-growing industry and an imp...

RoCourseNet: Distributionally Robust Training of a Prediction Aware Recourse Model

Counterfactual (CF) explanations for machine learning (ML) models are pr...

Model Assertions for Monitoring and Improving ML Model

ML models are increasingly deployed in settings with real world interact...

Model Assertions for Monitoring and Improving ML Models

ML models are increasingly deployed in settings with real world interact...

FrugalMCT: Efficient Online ML API Selection for Multi-Label Classification Tasks

Multi-label classification tasks such as OCR and multi-object recognitio...