DeepAI AI Chat
Log In Sign Up

How stable are Transferability Metrics evaluations?

by   Andrea Agostinelli, et al.

Transferability metrics is a maturing field with increasing interest, which aims at providing heuristics for selecting the most suitable source models to transfer to a given target dataset, without fine-tuning them all. However, existing works rely on custom experimental setups which differ across papers, leading to inconsistent conclusions about which transferability metrics work best. In this paper we conduct a large-scale study by systematically constructing a broad range of 715k experimental setup variations. We discover that even small variations to an experimental setup lead to different conclusions about the superiority of a transferability metric over another. Then we propose better evaluations by aggregating across many experiments, enabling to reach more stable conclusions. As a result, we reveal the superiority of LogME at selecting good source datasets to transfer from in a semantic segmentation scenario, NLEEP at selecting good source architectures in an image classification scenario, and GBC at determining which target task benefits most from a given source model. Yet, no single transferability metric works best in all scenarios.


page 1

page 2

page 3

page 4


Transferability Metrics for Selecting Source Model Ensembles

We address the problem of ensemble selection in transfer learning: Given...

Towards Estimating Transferability using Hard Subsets

As transfer learning techniques are increasingly used to transfer knowle...

Building a Winning Team: Selecting Source Model Ensembles using a Submodular Transferability Estimation Approach

Estimating the transferability of publicly available pretrained models t...

Transferability Metrics for Object Detection

Transfer learning aims to make the most of existing pre-trained models t...

Transferability Estimation Based On Principal Gradient Expectation

Deep transfer learning has been widely used for knowledge transmission i...

The Performance of Transferability Metrics does not Translate to Medical Tasks

Transfer learning boosts the performance of medical image analysis by en...

A Quantification Approach for Transferability in Lifelike Computing Systems

The basic idea of lifelike computing systems is the transfer of concepts...