Task-Adaptive Neural Network Retrieval with Meta-Contrastive Learning

by   Wonyong Jeong, et al.

Most conventional Neural Architecture Search (NAS) approaches are limited in that they only generate architectures (network topologies) without searching for optimal parameters. While some NAS methods handle this issue by utilizing a supernet trained on a large-scale dataset such as ImageNet, they may be suboptimal if the target tasks are highly dissimilar from the dataset the supernet is trained on. To tackle this issue, we propose a novel neural network retrieval method, which retrieves the most optimal pre-trained network for a given task and constraints (e.g. number of parameters) from a model zoo. We train this framework by meta-learning a cross-modal latent space with contrastive loss, to maximize the similarity between a dataset and a network that obtains high performance on it, and minimize the similarity between an irrelevant dataset-network pair. We validate the efficacy of our method on ten real-world datasets, against existing NAS baselines. The results show that our method instantly retrieves networks that outperforms models obtained with the baselines with significantly fewer training steps to reach the target performance.



There are no comments yet.


page 2

page 6

page 15


Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets

Despite the success of recent Neural Architecture Search (NAS) methods o...

Across-Task Neural Architecture Search via Meta Learning

Adequate labeled data and expensive compute resources are the prerequisi...

Rapid Model Architecture Adaption for Meta-Learning

Network Architecture Search (NAS) methods have recently gathered much at...

One-Shot Neural Architecture Search via Compressive Sensing

Neural architecture search (NAS), or automated design of neural network ...

Data-Free Neural Architecture Search via Recursive Label Calibration

This paper aims to explore the feasibility of neural architecture search...

MetAdapt: Meta-Learned Task-Adaptive Architecture for Few-Shot Classification

Few-Shot Learning (FSL) is a topic of rapidly growing interest. Typicall...

Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data

This paper investigates the intriguing question of whether we can create...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.