Improving Neural Architecture Search Image Classifiers via Ensemble Learning

03/14/2019
by   Vladimir Macko, et al.
0

Finding the best neural network architecture requires significant time, resources, and human expertise. These challenges are partially addressed by neural architecture search (NAS) which is able to find the best convolutional layer or cell that is then used as a building block for the network. However, once a good building block is found, manual design is still required to assemble the final architecture as a combination of multiple blocks under a predefined parameter budget constraint. A common solution is to stack these blocks into a single tower and adjust the width and depth to fill the parameter budget. However, these single tower architectures may not be optimal. Instead, in this paper we present the AdaNAS algorithm, that uses ensemble techniques to compose a neural network as an ensemble of smaller networks automatically. Additionally, we introduce a novel technique based on knowledge distillation to iteratively train the smaller networks using the previous ensemble as a teacher. Our experiments demonstrate that ensembles of networks improve accuracy upon a single neural network while keeping the same number of parameters. Our models achieve comparable results with the state-of-the-art on CIFAR-10 and sets a new state-of-the-art on CIFAR-100.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/19/2023

RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation

Deep Neural Networks are vulnerable to adversarial attacks. Neural Archi...
research
08/18/2020

Feature Products Yield Efficient Networks

We introduce Feature-Product networks (FP-nets) as a novel deep-network ...
research
11/04/2020

Channel Planting for Deep Neural Networks using Knowledge Distillation

In recent years, deeper and wider neural networks have shown excellent p...
research
02/03/2023

Enhancing Once-For-All: A Study on Parallel Blocks, Skip Connections and Early Exits

The use of Neural Architecture Search (NAS) techniques to automate the d...
research
05/14/2020

Deep Ensembles on a Fixed Memory Budget: One Wide Network or Several Thinner Ones?

One of the generally accepted views of modern deep learning is that incr...
research
07/01/2020

Group Ensemble: Learning an Ensemble of ConvNets in a single ConvNet

Ensemble learning is a general technique to improve accuracy in machine ...
research
10/27/2018

Training Frankenstein's Creature to Stack: HyperTree Architecture Search

We propose HyperTrees for the low cost automatic design of multiple-inpu...

Please sign up or login with your details

Forgot password? Click here to reset