DeepAI AI Chat
Log In Sign Up

Which Model Shall I Choose? Cost/Quality Trade-offs for Text Classification Tasks

by   Shi Zong, et al.
University of Waterloo

Industry practitioners always face the problem of choosing the appropriate model for deployment under different considerations, such as to maximize a metric that is crucial for production, or to reduce the total cost given financial concerns. In this work, we focus on the text classification task and present a quantitative analysis for this challenge. Using classification accuracy as the main metric, we evaluate the classifiers' performances for a variety of models, including large language models, along with their associated costs, including the annotation cost, training (fine-tuning) cost, and inference cost. We then discuss the model choices for situations like having a large number of samples needed for inference. We hope our work will help people better understand the cost/quality trade-offs for the text classification task.


page 1

page 2

page 3

page 4


Performance-Efficiency Trade-Offs in Adapting Language Models to Text Classification Tasks

Pre-trained language models (LMs) obtain state-of-the-art performance wh...

A WL-SPPIM Semantic Model for Document Classification

In this paper, we explore SPPIM-based text classification method, and th...

Analyzing Text Representations under Tight Annotation Budgets: Measuring Structural Alignment

Annotating large collections of textual data can be time consuming and e...

Model Stability with Continuous Data Updates

In this paper, we study the "stability" of machine learning (ML) models ...

Revisiting Softmax for Uncertainty Approximation in Text Classification

Uncertainty approximation in text classification is an important area wi...

Computer-Assisted Creation of Boolean Search Rules for Text Classification in the Legal Domain

In this paper, we present a method of building strong, explainable class...