DeepAI AI Chat
Log In Sign Up

Which Model Shall I Choose? Cost/Quality Trade-offs for Text Classification Tasks

01/17/2023
by   Shi Zong, et al.
University of Waterloo
0

Industry practitioners always face the problem of choosing the appropriate model for deployment under different considerations, such as to maximize a metric that is crucial for production, or to reduce the total cost given financial concerns. In this work, we focus on the text classification task and present a quantitative analysis for this challenge. Using classification accuracy as the main metric, we evaluate the classifiers' performances for a variety of models, including large language models, along with their associated costs, including the annotation cost, training (fine-tuning) cost, and inference cost. We then discuss the model choices for situations like having a large number of samples needed for inference. We hope our work will help people better understand the cost/quality trade-offs for the text classification task.

READ FULL TEXT

page 1

page 2

page 3

page 4

10/21/2022

Performance-Efficiency Trade-Offs in Adapting Language Models to Text Classification Tasks

Pre-trained language models (LMs) obtain state-of-the-art performance wh...
05/26/2017

A WL-SPPIM Semantic Model for Document Classification

In this paper, we explore SPPIM-based text classification method, and th...
10/11/2022

Analyzing Text Representations under Tight Annotation Budgets: Measuring Structural Alignment

Annotating large collections of textual data can be time consuming and e...
01/14/2022

Model Stability with Continuous Data Updates

In this paper, we study the "stability" of machine learning (ML) models ...
10/25/2022

Revisiting Softmax for Uncertainty Approximation in Text Classification

Uncertainty approximation in text classification is an important area wi...
12/10/2021

Computer-Assisted Creation of Boolean Search Rules for Text Classification in the Legal Domain

In this paper, we present a method of building strong, explainable class...