One-Shot Neural Architecture Search Through A Posteriori Distribution Guided Sampling

06/23/2019
by   Yizhou Zhou, et al.
5

The emergence of one-shot approaches has greatly advanced the research on neural architecture search (NAS). Recent approaches train an over-parameterized super-network (one-shot model) and then sample and evaluate a number of sub-networks, which inherit weights from the one-shot model. The overall searching cost is significantly reduced as training is avoided for sub-networks. However, the network sampling process is casually treated and the inherited weights from an independently trained super-network perform sub-optimally for sub-networks. In this paper, we propose a novel one-shot NAS scheme to address the above issues. The key innovation is to explicitly estimate the joint a posteriori distribution over network architecture and weights, and sample networks for evaluation according to it. This brings two benefits. First, network sampling under the guidance of a posteriori probability is more efficient than conventional random or uniform sampling. Second, the network architecture and its weights are sampled as a pair to alleviate the sub-optimal weights problem. Note that estimating the joint a posteriori distribution is not a trivial problem. By adopting variational methods and introducing a hybrid network representation, we convert the distribution approximation problem into an end-to-end neural network training problem which is neatly approached by variational dropout. As a result, the proposed method reduces the number of sampled sub-networks by orders of magnitude. We validate our method on the fundamental image classification task. Results on Cifar-10, Cifar-100 and ImageNet show that our method strikes the best trade-off between precision and speed among NAS methods. On Cifar-10, we speed up the searching process by 20x and achieve a higher precision than the best network found by existing NAS methods.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/11/2020

Few-shot Neural Architecture Search

To improve the search efficiency for Neural Architecture Search (NAS), O...
research
05/13/2019

BayesNAS: A Bayesian Approach for Neural Architecture Search

One-Shot Neural Architecture Search (NAS) is a promising method to signi...
research
01/28/2021

Neural Architecture Search with Random Labels

In this paper, we investigate a new variant of neural architecture searc...
research
05/19/2022

A Hardware-Aware Framework for Accelerating Neural Architecture Search Across Modalities

Recent advances in Neural Architecture Search (NAS) such as one-shot NAS...
research
07/17/2023

ShiftNAS: Improving One-shot NAS via Probability Shift

One-shot Neural architecture search (One-shot NAS) has been proposed as ...
research
05/19/2021

Efficient Transfer Learning via Joint Adaptation of Network Architecture and Weight

Transfer learning can boost the performance on the targettask by leverag...
research
04/23/2021

Inter-choice dependent super-network weights

The automatic design of architectures for neural networks, Neural Archit...

Please sign up or login with your details

Forgot password? Click here to reset