Neural Network Design: Learning from Neural Architecture Search

11/01/2020
by   Bas van Stein, et al.
142

Neural Architecture Search (NAS) aims to optimize deep neural networks' architecture for better accuracy or smaller computational cost and has recently gained more research interests. Despite various successful approaches proposed to solve the NAS task, the landscape of it, along with its properties, are rarely investigated. In this paper, we argue for the necessity of studying the landscape property thereof and propose to use the so-called Exploratory Landscape Analysis (ELA) techniques for this goal. Taking a broad set of designs of the deep convolutional network, we conduct extensive experimentation to obtain their performance. Based on our analysis of the experimental results, we observed high similarities between well-performing architecture designs, which is then used to significantly narrow the search space to improve the efficiency of any NAS algorithm. Moreover, we extract the ELA features over the NAS landscapes on three common image classification data sets, MNIST, Fashion, and CIFAR-10, which shows that the NAS landscape can be distinguished for those three data sets. Also, when comparing to the ELA features of the well-known Black-Box Optimization Benchmarking (BBOB) problem set, we found out that the NAS landscapes surprisingly form a new problem class on its own, which can be separated from all 24 BBOB problems. Given this interesting observation, we, therefore, state the importance of further investigation on selecting an efficient optimizer for the NAS landscape as well as the necessity of augmenting the current benchmark problem set.

READ FULL TEXT

page 5

page 6

page 8

research
07/09/2023

Carbon-Efficient Neural Architecture Search

This work presents a novel approach to neural architecture search (NAS) ...
research
07/04/2021

Mutation is all you need

Neural architecture search (NAS) promises to make deep learning accessib...
research
05/08/2022

αNAS: Neural Architecture Search using Property Guided Synthesis

In the past few years, neural architecture search (NAS) has become an in...
research
07/10/2022

Noisy Heuristics NAS: A Network Morphism based Neural Architecture Search using Heuristics

Network Morphism based Neural Architecture Search (NAS) is one of the mo...
research
11/02/2021

Fitness Landscape Footprint: A Framework to Compare Neural Architecture Search Problems

Neural architecture search is a promising area of research dedicated to ...
research
08/23/2023

A Benchmark Study on Calibration

Deep neural networks are increasingly utilized in various machine learni...
research
06/02/2022

A Local Optima Network Analysis of the Feedforward Neural Architecture Space

This study investigates the use of local optima network (LON) analysis, ...

Please sign up or login with your details

Forgot password? Click here to reset