AutoHAS: Differentiable Hyper-parameter and Architecture Search

06/05/2020
by   Xuanyi Dong, et al.
0

Neural Architecture Search (NAS) has achieved significant progress in pushing state-of-the-art performance. While previous NAS methods search for different network architectures with the same hyper-parameters, we argue that such search would lead to sub-optimal results. We empirically observe that different architectures tend to favor their own hyper-parameters. In this work, we extend NAS to a broader and more practical space by combining hyper-parameter and architecture search. As architecture choices are often categorical whereas hyper-parameter choices are often continuous, a critical challenge here is how to handle these two types of values in a joint search space. To tackle this challenge, we propose AutoHAS, a differentiable hyper-parameter and architecture search approach, with the idea of discretizing the continuous space into a linear combination of multiple categorical basis. A key element of AutoHAS is the use of weight sharing across all architectures and hyper-parameters which enables efficient search over the large joint search space. Experimental results on MobileNet/ResNet/EfficientNet/BERT show that AutoHAS significantly improves accuracy up to 2 0.4 on SQuAD 1.1, with search cost comparable to training a single model. Compared to other AutoML methods, such as random search or Bayesian methods, AutoHAS can achieve better accuracy with 10x less compute cost.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/05/2022

Search Space Adaptation for Differentiable Neural Architecture Search in Image Classification

As deep neural networks achieve unprecedented performance in various tas...
research
05/20/2020

Rethinking Performance Estimation in Neural Architecture Search

Neural architecture search (NAS) remains a challenging problem, which is...
research
09/13/2021

DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture

Automated machine learning (AutoML) usually involves several crucial com...
research
04/23/2023

HKNAS: Classification of Hyperspectral Imagery Based on Hyper Kernel Neural Architecture Search

Recent neural architecture search (NAS) based approaches have made great...
research
04/03/2020

Neural Architecture Generator Optimization

Neural Architecture Search (NAS) was first proposed to achieve state-of-...
research
01/21/2021

PyGlove: Symbolic Programming for Automated Machine Learning

Neural networks are sensitive to hyper-parameter and architecture choice...
research
09/22/2020

AutoRC: Improving BERT Based Relation Classification Models via Architecture Search

Although BERT based relation classification (RC) models have achieved si...

Please sign up or login with your details

Forgot password? Click here to reset