Simple Control Baselines for Evaluating Transfer Learning

02/07/2022
by   Andrei Atanov, et al.
0

Transfer learning has witnessed remarkable progress in recent years, for example, with the introduction of augmentation-based contrastive self-supervised learning methods. While a number of large-scale empirical studies on the transfer performance of such models have been conducted, there is not yet an agreed-upon set of control baselines, evaluation practices, and metrics to report, which often hinders a nuanced and calibrated understanding of the real efficacy of the methods. We share an evaluation standard that aims to quantify and communicate transfer learning performance in an informative and accessible setup. This is done by baking a number of simple yet critical control baselines in the evaluation method, particularly the blind-guess (quantifying the dataset bias), scratch-model (quantifying the architectural contribution), and maximal-supervision (quantifying the upper-bound). To demonstrate how the evaluation standard can be employed, we provide an example empirical study investigating a few basic questions about self-supervised learning. For example, using this standard, the study shows the effectiveness of existing self-supervised pre-training methods is skewed towards image classification tasks versus dense pixel-wise predictions. In general, we encourage using/reporting the suggested control baselines in evaluating transfer learning in order to gain a more meaningful and informative understanding.

READ FULL TEXT

page 4

page 7

research
07/01/2020

A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural Networks

Deep neural networks are typically trained under a supervised learning f...
research
04/14/2022

DeiT III: Revenge of the ViT

A Vision Transformer (ViT) is a simple neural architecture amenable to s...
research
11/03/2020

Learning Visual Representations for Transfer Learning by Suppressing Texture

Recent literature has shown that features obtained from supervised train...
research
11/22/2021

Benchmarking Detection Transfer Learning with Vision Transformers

Object detection is a central downstream task used to test if pre-traine...
research
03/30/2021

Benchmarking Representation Learning for Natural World Image Collections

Recent progress in self-supervised learning has resulted in models that ...
research
06/02/2022

Expressiveness and Learnability: A Unifying View for Evaluating Self-Supervised Learning

We propose a unifying view to analyze the representation quality of self...
research
09/06/2021

Training Deep Networks from Zero to Hero: avoiding pitfalls and going beyond

Training deep neural networks may be challenging in real world data. Usi...

Please sign up or login with your details

Forgot password? Click here to reset