DeepAI AI Chat
Log In Sign Up

Are All the Datasets in Benchmark Necessary? A Pilot Study of Dataset Evaluation for Text Classification

by   Yang Xiao, et al.
National University of Singapore
Carnegie Mellon University

In this paper, we ask the research question of whether all the datasets in the benchmark are necessary. We approach this by first characterizing the distinguishability of datasets when comparing different systems. Experiments on 9 datasets and 36 systems show that several existing benchmark datasets contribute little to discriminating top-scoring systems, while those less used datasets exhibit impressive discriminative power. We further, taking the text classification task as a case study, investigate the possibility of predicting dataset discrimination based on its properties (e.g., average sentence length). Our preliminary experiments promisingly show that given a sufficient number of training experimental records, a meaningful predictor can be learned to estimate dataset discrimination over unseen datasets. We released all datasets with features explored in this work on DataLab: <>.


page 1

page 2

page 3

page 4


A WL-SPPIM Semantic Model for Document Classification

In this paper, we explore SPPIM-based text classification method, and th...

Improve Text Classification Accuracy with Intent Information

Text classification, a core component of task-oriented dialogue systems,...

Identifying Semantically Difficult Samples to Improve Text Classification

In this paper, we investigate the effect of addressing difficult samples...

LSHTC: A Benchmark for Large-Scale Text Classification

LSHTC is a series of challenges which aims to assess the performance of ...

Evolutionary Data Measures: Understanding the Difficulty of Text Classification Tasks

Classification tasks are usually analysed and improved through new model...