Rethinking Two Consensuses of the Transferability in Deep Learning

12/01/2022
by   Yixiong Chen, et al.
0

Deep transfer learning (DTL) has formed a long-term quest toward enabling deep neural networks (DNNs) to reuse historical experiences as efficiently as humans. This ability is named knowledge transferability. A commonly used paradigm for DTL is firstly learning general knowledge (pre-training) and then reusing (fine-tuning) them for a specific target task. There are two consensuses of transferability of pre-trained DNNs: (1) a larger domain gap between pre-training and downstream data brings lower transferability; (2) the transferability gradually decreases from lower layers (near input) to higher layers (near output). However, these consensuses were basically drawn from the experiments based on natural images, which limits their scope of application. This work aims to study and complement them from a broader perspective by proposing a method to measure the transferability of pre-trained DNN parameters. Our experiments on twelve diverse image classification datasets get similar conclusions to the previous consensuses. More importantly, two new findings are presented, i.e., (1) in addition to the domain gap, a larger data amount and huge dataset diversity of downstream target task also prohibit the transferability; (2) although the lower layers learn basic image features, they are usually not the most transferable layers due to their domain sensitivity.

READ FULL TEXT
research
09/08/2021

On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets

Pre-training language models (LMs) on large-scale unlabeled text data ma...
research
01/15/2022

Transferability in Deep Learning: A Survey

The success of deep learning algorithms generally depends on large-scale...
research
10/14/2021

Omni-Training for Data-Efficient Deep Learning

Learning a generalizable deep model from a few examples in a short time ...
research
11/23/2020

Ranking Neural Checkpoints

This paper is concerned with ranking many pre-trained deep neural networ...
research
04/24/2020

How Much Off-The-Shelf Knowledge Is Transferable From Natural Images To Pathology Images?

Deep learning has achieved a great success in natural image classificati...
research
03/17/2020

DEPARA: Deep Attribution Graph for Deep Knowledge Transferability

Exploring the intrinsic interconnections between the knowledge encoded i...
research
11/06/2014

How transferable are features in deep neural networks?

Many deep neural networks trained on natural images exhibit a curious ph...

Please sign up or login with your details

Forgot password? Click here to reset