A linearized framework and a new benchmark for model selection for fine-tuning

by   Aditya Deshpande, et al.

Fine-tuning from a collection of models pre-trained on different domains (a "model zoo") is emerging as a technique to improve test accuracy in the low-data regime. However, model selection, i.e. how to pre-select the right model to fine-tune from a model zoo without performing any training, remains an open topic. We use a linearized framework to approximate fine-tuning, and introduce two new baselines for model selection – Label-Gradient and Label-Feature Correlation. Since all model selection algorithms in the literature have been tested on different use-cases and never compared directly, we introduce a new comprehensive benchmark for model selection comprising of: i) A model zoo of single and multi-domain models, and ii) Many target tasks. Our benchmark highlights accuracy gain with model zoo compared to fine-tuning Imagenet models. We show our model selection baseline can select optimal models to fine-tune in few selections and has the highest ranking correlation to fine-tuning accuracy compared to existing algorithms.


Memorization in NLP Fine-tuning Methods

Large language models are shown to present privacy risks through memoriz...

Scalable Diverse Model Selection for Accessible Transfer Learning

With the preponderance of pretrained deep learning models available off-...

Neural Capacitance: A New Perspective of Neural Network Selection via Edge Dynamics

Efficient model selection for identifying a suitable pre-trained neural ...

A Flexible Selection Scheme for Minimum-Effort Transfer Learning

Fine-tuning is a popular way of exploiting knowledge contained in a pre-...

Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization

In Domain Generalization (DG) settings, models trained on a given set of...

Tune: A Research Platform for Distributed Model Selection and Training

Modern machine learning algorithms are increasingly computationally dema...

Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference

The task of abductive natural language inference (αnli), to decide which...