Parameter Transfer Unit for Deep Neural Networks

by   Yinghua Zhang, et al.
The Hong Kong University of Science and Technology

Parameters in deep neural networks which are trained on large-scale databases can generalize across multiple domains, which is referred as "transferability". Unfortunately, the transferability is usually defined as discrete states and it differs with domains and network architectures. Existing works usually heuristically apply parameter-sharing or fine-tuning, and there is no principled approach to learn a parameter transfer strategy. To address the gap, a parameter transfer unit (PTU) is proposed in this paper. The PTU learns a fine-grained nonlinear combination of activations from both the source and the target domain networks, and subsumes hand-crafted discrete transfer states. In the PTU, the transferability is controlled by two gates which are artificial neurons and can be learned from data. The PTU is a general and flexible module which can be used in both CNNs and RNNs. Experiments are conducted with various network architectures and multiple transfer domain pairs. Results demonstrate the effectiveness of the PTU as it outperforms heuristic parameter-sharing and fine-tuning in most settings.


On Transferability of Prompt Tuning for Natural Language Understanding

Prompt tuning (PT) is a promising parameter-efficient method to utilize ...

Multifaceted Analysis of Fine-Tuning in Deep Model for Visual Recognition

In recent years, convolutional neural networks (CNNs) have achieved impr...

PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptation

Prompt-tuning, which freezes pretrained language models (PLMs) and only ...

Learning What and Where to Transfer

As the application of deep learning has expanded to real-world problems ...

Domain-adaptive deep network compression

Deep Neural Networks trained on large datasets can be easily transferred...

How transferable are features in deep neural networks?

Many deep neural networks trained on natural images exhibit a curious ph...

DRESS: Dynamic REal-time Sparse Subnets

The limited and dynamically varied resources on edge devices motivate us...

Please sign up or login with your details

Forgot password? Click here to reset