FrugalMCT: Efficient Online ML API Selection for Multi-Label Classification Tasks

02/18/2021
by   Lingjiao Chen, et al.
13

Multi-label classification tasks such as OCR and multi-object recognition are a major focus of the growing machine learning as a service industry. While many multi-label prediction APIs are available, it is challenging for users to decide which API to use for their own data and budget, due to the heterogeneity in those APIs' price and performance. Recent work shows how to select from single-label prediction APIs. However the computation complexity of the previous approach is exponential in the number of labels and hence is not suitable for settings like OCR. In this work, we propose FrugalMCT, a principled framework that adaptively selects the APIs to use for different data in an online fashion while respecting user's budget. The API selection problem is cast as an integer linear program, which we show has a special structure that we leverage to develop an efficient online API selector with strong performance guarantees. We conduct systematic experiments using ML APIs from Google, Microsoft, Amazon, IBM, Tencent and other providers for tasks including multi-label image classification, scene text recognition and named entity recognition. Across diverse tasks, FrugalMCT can achieve over 90 reduction while matching the accuracy of the best single API, or up to 8 better accuracy while matching the best API's cost.

READ FULL TEXT

Authors

page 8

page 38

page 39

06/12/2020

FrugalML: How to Use ML Prediction APIs More Accurately and Cheaply

Prediction APIs offered for a fee are a fast-growing industry and an imp...
11/09/2018

Automated Multi-Label Classification based on ML-Plan

Automated machine learning (AutoML) has received increasing attention in...
11/13/2018

ML-Net: multi-label classification of biomedical texts with deep neural networks

Background: Multi-label text classification is one type of text classifi...
11/23/2020

The Emerging Trends of Multi-Label Learning

Exabytes of data are generated daily by humans, leading to the growing n...
06/20/2022

DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations

Solving multi-label recognition (MLR) for images in the low-label regime...
07/29/2021

Did the Model Change? Efficiently Assessing Machine Learning API Shifts

Machine learning (ML) prediction APIs are increasingly widely used. An M...
07/06/2013

Ensemble Methods for Multi-label Classification

Ensemble methods have been shown to be an effective tool for solving mul...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.