Efficient Transfer Learning via Joint Adaptation of Network Architecture and Weight

05/19/2021
by   Ming Sun, et al.
0

Transfer learning can boost the performance on the targettask by leveraging the knowledge of the source domain. Recent worksin neural architecture search (NAS), especially one-shot NAS, can aidtransfer learning by establishing sufficient network search space. How-ever, existing NAS methods tend to approximate huge search spaces byexplicitly building giant super-networks with multiple sub-paths, anddiscard super-network weights after a child structure is found. Both thecharacteristics of existing approaches causes repetitive network trainingon source tasks in transfer learning. To remedy the above issues, we re-duce the super-network size by randomly dropping connection betweennetwork blocks while embedding a larger search space. Moreover, wereuse super-network weights to avoid redundant training by proposinga novel framework consisting of two modules, the neural architecturesearch module for architecture transfer and the neural weight searchmodule for weight transfer. These two modules conduct search on thetarget task based on a reduced super-networks, so we only need to trainonce on the source task. We experiment our framework on both MS-COCO and CUB-200 for the object detection and fine-grained imageclassification tasks, and show promising improvements with onlyO(CN)super-network complexity.

READ FULL TEXT

page 11

page 14

research
06/11/2020

Few-shot Neural Architecture Search

To improve the search efficiency for Neural Architecture Search (NAS), O...
research
06/19/2019

Transfer NAS: Knowledge Transfer between Search Spaces with Transformer Agents

Recent advances in Neural Architecture Search (NAS) have produced state-...
research
10/30/2017

Transfer Learning to Learn with Multitask Neural Model Search

Deep learning models require extensive architecture design exploration a...
research
04/23/2021

Inter-choice dependent super-network weights

The automatic design of architectures for neural networks, Neural Archit...
research
05/11/2022

AutoKE: An automatic knowledge embedding framework for scientific machine learning

Imposing physical constraints on neural networks as a method of knowledg...
research
11/18/2019

ImmuNeCS: Neural Committee Search by an Artificial Immune System

Current Neural Architecture Search techniques can suffer from a few shor...
research
06/23/2019

One-Shot Neural Architecture Search Through A Posteriori Distribution Guided Sampling

The emergence of one-shot approaches has greatly advanced the research o...

Please sign up or login with your details

Forgot password? Click here to reset