AdvantageNAS: Efficient Neural Architecture Search with Credit Assignment

12/11/2020
by   Rei Sato, et al.
0

Neural architecture search (NAS) is an approach for automatically designing a neural network architecture without human effort or expert knowledge. However, the high computational cost of NAS limits its use in commercial applications. Two recent NAS paradigms, namely one-shot and sparse propagation, which reduce the time and space complexities, respectively, provide clues for solving this problem. In this paper, we propose a novel search strategy for one-shot and sparse propagation NAS, namely AdvantageNAS, which further reduces the time complexity of NAS by reducing the number of search iterations. AdvantageNAS is a gradient-based approach that improves the search efficiency by introducing credit assignment in gradient estimation for architecture updates. Experiments on the NAS-Bench-201 and PTB dataset show that AdvantageNAS discovers an architecture with higher performance under a limited time budget compared to existing sparse propagation NAS. To further reveal the reliabilities of AdvantageNAS, we investigate it theoretically and find that it monotonically improves the expected loss and thus converges.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/24/2018

SNAS: Stochastic Neural Architecture Search

We propose Stochastic Neural Architecture Search (SNAS), an economical e...
research
06/04/2021

Neural Architecture Search via Bregman Iterations

We propose a novel strategy for Neural Architecture Search (NAS) based o...
research
11/23/2022

NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension

One-shot neural architecture search (NAS) substantially improves the sea...
research
07/07/2020

Hyperparameter Optimization in Neural Networks via Structured Sparse Recovery

In this paper, we study two important problems in the automated design o...
research
07/21/2022

Efficient Search of Multiple Neural Architectures with Different Complexities via Importance Sampling

Neural architecture search (NAS) aims to automate architecture design pr...
research
10/25/2022

NAS-PRNet: Neural Architecture Search generated Phase Retrieval Net for Off-axis Quantitative Phase Imaging

Single neural networks have achieved simultaneous phase retrieval with a...
research
11/02/2022

Speeding up NAS with Adaptive Subset Selection

A majority of recent developments in neural architecture search (NAS) ha...

Please sign up or login with your details

Forgot password? Click here to reset