Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks

02/17/2021
by   Lemeng Wu, et al.
13

We propose firefly neural architecture descent, a general framework for progressively and dynamically growing neural networks to jointly optimize the networks' parameters and architectures. Our method works in a steepest descent fashion, which iteratively finds the best network within a functional neighborhood of the original network that includes a diverse set of candidate network structures. By using Taylor approximation, the optimal network structure in the neighborhood can be found with a greedy selection procedure. We show that firefly descent can flexibly grow networks both wider and deeper, and can be applied to learn accurate but resource-efficient neural architectures that avoid catastrophic forgetting in continual learning. Empirically, firefly descent achieves promising results on both neural architecture search and continual learning. In particular, on a challenging continual image classification task, it learns networks that are smaller in size but have higher average accuracy than those learned by the state-of-the-art methods.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/07/2020

Efficient Architecture Search for Continual Learning

Continual learning with neural networks is an important learning framewo...
research
03/31/2019

Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting

Addressing catastrophic forgetting is one of the key challenges in conti...
research
11/21/2020

Learn to Bind and Grow Neural Structures

Task-incremental learning involves the challenging problem of learning n...
research
04/14/2021

Neural Architecture Search of Deep Priors: Towards Continual Learning without Catastrophic Interference

In this paper we analyze the classification performance of neural networ...
research
10/17/2021

Growing Representation Learning

Machine learning continues to grow in popularity due to its ability to l...
research
03/23/2020

Steepest Descent Neural Architecture Optimization: Escaping Local Optimum with Signed Neural Splitting

We propose signed splitting steepest descent (S3D), which progressively ...
research
06/11/2022

A Review on Plastic Artificial Neural Networks: Exploring the Intersection between Neural Architecture Search and Continual Learning

Despite the significant advances achieved in Artificial Neural Networks ...

Please sign up or login with your details

Forgot password? Click here to reset