Zoo-Tuning: Adaptive Transfer from a Zoo of Models

06/29/2021
by   Yang Shu, et al.
0

With the development of deep networks on various large-scale datasets, a large zoo of pretrained models are available. When transferring from a model zoo, applying classic single-model based transfer learning methods to each source model suffers from high computational burden and cannot fully utilize the rich knowledge in the zoo. We propose Zoo-Tuning to address these challenges, which learns to adaptively transfer the parameters of pretrained models to the target task. With the learnable channel alignment layer and adaptive aggregation layer, Zoo-Tuning adaptively aggregates channel aligned pretrained parameters to derive the target model, which promotes knowledge transfer by simultaneously adapting multiple source models to downstream tasks. The adaptive aggregation substantially reduces the computation cost at both training and inference. We further propose lite Zoo-Tuning with the temporal ensemble of batch average gating values to reduce the storage cost at the inference time. We evaluate our approach on a variety of tasks, including reinforcement learning, image classification, and facial landmark detection. Experiment results demonstrate that the proposed adaptive transfer learning approach can transfer knowledge from a zoo of models more effectively and efficiently.

READ FULL TEXT
research
03/06/2023

Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning

Prompt tuning, in which a base pretrained model is adapted to each task ...
research
06/08/2022

Hub-Pathway: Transfer Learning from A Hub of Pre-trained Models

Transfer learning aims to leverage knowledge from pre-trained models to ...
research
07/08/2022

Beyond Transfer Learning: Co-finetuning for Action Localisation

Transfer learning is the predominant paradigm for training deep networks...
research
10/12/2022

Prompt Generation Networks for Efficient Adaptation of Frozen Vision Transformers

Large-scale pretrained models, especially those trained from vision-lang...
research
09/12/2021

Facial Anatomical Landmark Detection using Regularized Transfer Learning with Application to Fetal Alcohol Syndrome Recognition

Fetal alcohol syndrome (FAS) caused by prenatal alcohol exposure can res...
research
04/14/2020

A Transfer Learning approach to Heatmap Regression for Action Unit intensity estimation

Action Units (AUs) are geometrically-based atomic facial muscle movement...
research
09/24/2020

Unsupervised Transfer Learning for Spatiotemporal Predictive Networks

This paper explores a new research problem of unsupervised transfer lear...

Please sign up or login with your details

Forgot password? Click here to reset