NATS-Bench: Benchmarking NAS algorithms for Architecture Topology and Size

by   Xuanyi Dong, et al.

Neural architecture search (NAS) has attracted a lot of attention and has been illustrated to bring tangible benefits in a large number of applications in the past few years. Network topology and network size have been regarded as two of the most important aspects for the performance of deep learning models and the community has spawned lots of searching algorithms for both aspects of the neural architectures. However, the performance gain from these searching algorithms is achieved under different search spaces and training setups. This makes the overall performance of the algorithms to some extent incomparable and the improvement from a sub-module of the searching model unclear. In this paper, we propose NATS-Bench, a unified benchmark on searching for both topology and size, for (almost) any up-to-date NAS algorithm. NATS-Bench includes the search space of 15,625 neural cell candidates for architecture topology and 32,768 for architecture size on three datasets. We analyse the validity of our benchmark in terms of various criteria and performance comparison of all candidates in the search space. We also show the versatility of NATS-Bench by benchmarking 13 recent state-of-the-art NAS algorithms on it. All logs and diagnostic information trained using the same setup for each candidate are provided. This facilitates a much larger community of researchers to focus on developing better NAS algorithms in a more comparable and computationally cost friendly environment. All codes are publicly available at: .



There are no comments yet.


page 5

page 6

page 7


NAS-Bench-102: Extending the Scope of Reproducible Neural Architecture Search

Neural architecture search (NAS) has achieved breakthrough success in a ...

Powering One-shot Topological NAS with Stabilized Share-parameter Proxy

One-shot NAS method has attracted much interest from the research commun...

Neural Architecture Search with Random Labels

In this paper, we investigate a new variant of neural architecture searc...

FP-NAS: Fast Probabilistic Neural Architecture Search

Differential Neural Architecture Search (NAS) requires all layer choices...

Prune and Replace NAS

While recent NAS algorithms are thousands of times faster than the pione...

BenchENAS: A Benchmarking Platform for Evolutionary Neural Architecture Search

Neural architecture search (NAS), which automatically designs the archit...

NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters

The benchmark datasets for neural architecture search (NAS) have been de...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.