MILO: Model-Agnostic Subset Selection Framework for Efficient Model Training and Tuning

01/30/2023

∙

by   Kirshnateja Killamsetty, et al.

∙

ibm

∙

The University of Texas at Dallas

∙

0

∙

share

Training deep networks and tuning hyperparameters on large datasets is computationally intensive. One of the primary research directions for efficient training is to reduce training costs by selecting well-generalizable subsets of training data. Compared to simple adaptive random subset selection baselines, existing intelligent subset selection approaches are not competitive due to the time-consuming subset selection step, which involves computing model-dependent gradients and feature embeddings and applies greedy maximization of submodular objectives. Our key insight is that removing the reliance on downstream model parameters enables subset selection as a pre-processing step and enables one to train multiple models at no additional cost. In this work, we propose MILO, a model-agnostic subset selection framework that decouples the subset selection from model training while enabling superior model convergence and performance by using an easy-to-hard curriculum. Our empirical results indicate that MILO can train models 3× - 10 × faster and tune hyperparameters 20× - 75 × faster than full-dataset training or tuning without compromising performance.