Zero-Cost Proxies Meet Differentiable Architecture Search

06/12/2021
by   Lichuan Xiang, et al.
4

Differentiable neural architecture search (NAS) has attracted significant attention in recent years due to its ability to quickly discover promising architectures of deep neural networks even in very large search spaces. Despite its success, DARTS lacks robustness in certain cases, e.g. it may degenerate to trivial architectures with excessive parametric-free operations such as skip connection or random noise, leading to inferior performance. In particular, operation selection based on the magnitude of architectural parameters was recently proven to be fundamentally wrong showcasing the need to rethink this aspect. On the other hand, zero-cost proxies have been recently studied in the context of sample-based NAS showing promising results – speeding up the search process drastically in some cases but also failing on some of the large search spaces typical for differentiable NAS. In this work we propose a novel operation selection paradigm in the context of differentiable NAS which utilises zero-cost proxies. Our perturbation-based zero-cost operation selection (Zero-Cost-PT) improves searching time and, in many cases, accuracy compared to the best available differentiable architecture search, regardless of the search space size. Specifically, we are able to find comparable architectures to DARTS-PT on the DARTS CNN search space while being over 40x faster (total searching time 25 minutes on a single GPU).

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/16/2019

UNAS: Differentiable Architecture Search Meets Reinforcement Learning

Neural architecture search (NAS) aims to discover network architectures ...
research
11/08/2021

Approximate Neural Architecture Search via Operation Distribution Learning

The standard paradigm in Neural Architecture Search (NAS) is to search f...
research
03/03/2022

β-DARTS: Beta-Decay Regularization for Differentiable Architecture Search

Neural Architecture Search (NAS) has attracted increasingly more attenti...
research
07/18/2023

An Evaluation of Zero-Cost Proxies – from Neural Architecture Performance to Model Robustness

Zero-cost proxies are nowadays frequently studied and used to search for...
research
12/24/2021

DARTS without a Validation Set: Optimizing the Marginal Likelihood

The success of neural architecture search (NAS) has historically been li...
research
09/02/2020

Understanding the wiring evolution in differentiable neural architecture search

Controversy exists on whether differentiable neural architecture search ...
research
05/06/2019

Differentiable Architecture Search with Ensemble Gumbel-Softmax

For network architecture search (NAS), it is crucial but challenging to ...

Please sign up or login with your details

Forgot password? Click here to reset