Task-Adaptive Neural Network Retrieval with Meta-Contrastive Learning

03/02/2021
by   Wonyong Jeong, et al.
10

Most conventional Neural Architecture Search (NAS) approaches are limited in that they only generate architectures (network topologies) without searching for optimal parameters. While some NAS methods handle this issue by utilizing a supernet trained on a large-scale dataset such as ImageNet, they may be suboptimal if the target tasks are highly dissimilar from the dataset the supernet is trained on. To tackle this issue, we propose a novel neural network retrieval method, which retrieves the most optimal pre-trained network for a given task and constraints (e.g. number of parameters) from a model zoo. We train this framework by meta-learning a cross-modal latent space with contrastive loss, to maximize the similarity between a dataset and a network that obtains high performance on it, and minimize the similarity between an irrelevant dataset-network pair. We validate the efficacy of our method on ten real-world datasets, against existing NAS baselines. The results show that our method instantly retrieves networks that outperforms models obtained with the baselines with significantly fewer training steps to reach the target performance.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 2

page 6

page 15

07/02/2021

Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets

Despite the success of recent Neural Architecture Search (NAS) methods o...
10/12/2021

Across-Task Neural Architecture Search via Meta Learning

Adequate labeled data and expensive compute resources are the prerequisi...
09/10/2021

Rapid Model Architecture Adaption for Meta-Learning

Network Architecture Search (NAS) methods have recently gathered much at...
06/07/2019

One-Shot Neural Architecture Search via Compressive Sensing

Neural architecture search (NAS), or automated design of neural network ...
12/03/2021

Data-Free Neural Architecture Search via Recursive Label Calibration

This paper aims to explore the feasibility of neural architecture search...
12/01/2019

MetAdapt: Meta-Learned Task-Adaptive Architecture for Few-Shot Classification

Few-Shot Learning (FSL) is a topic of rapidly growing interest. Typicall...
12/17/2019

Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data

This paper investigates the intriguing question of whether we can create...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.