Controlling Model Complexity in Probabilistic Model-Based Dynamic Optimization of Neural Network Structures

07/15/2019
by   Shota Saito, et al.
7

A method of simultaneously optimizing both the structure of neural networks and the connection weights in a single training loop can reduce the enormous computational cost of neural architecture search. We focus on the probabilistic model-based dynamic neural network structure optimization that considers the probability distribution of structure parameters and simultaneously optimizes both the distribution parameters and connection weights based on gradient methods. Since the existing algorithm searches for the structures that only minimize the training loss, this method might find overly complicated structures. In this paper, we propose the introduction of a penalty term to control the model complexity of obtained structures. We formulate a penalty term using the number of weights or units and derive its analytical natural gradient. The proposed method minimizes the objective function injected the penalty term based on the stochastic gradient descent. We apply the proposed method in the unit selection of a fully-connected neural network and the connection selection of a convolutional neural network. The experimental results show that the proposed method can control model complexity while maintaining performance.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/23/2018

Dynamic Optimization of Neural Network Structures Using Probabilistic Modeling

Deep neural networks (DNNs) are powerful machine learning models and hav...
research
12/14/2017

Nonparametric Neural Networks

Automatically determining the optimal size of a neural network for a giv...
research
03/26/2021

Explore the Knowledge contained in Network Weights to Obtain Sparse Neural Networks

Sparse neural networks are important for achieving better generalization...
research
10/08/2019

Differentiable Sparsification for Deep Neural Networks

A deep neural network has relieved the burden of feature engineering by ...
research
10/11/2021

Mining the Weights Knowledge for Optimizing Neural Network Structures

Knowledge embedded in the weights of the artificial neural network can b...
research
07/07/2021

Mitigating Performance Saturation in Neural Marked Point Processes: Architectures and Loss Functions

Attributed event sequences are commonly encountered in practice. A recen...
research
05/26/2021

On the Exploitation of Neuroevolutionary Information: Analyzing the Past for a More Efficient Future

Neuroevolutionary algorithms, automatic searches of neural network struc...

Please sign up or login with your details

Forgot password? Click here to reset