Towards Unifying Neural Architecture Space Exploration and Generalization

10/02/2019
by   Kartikeya Bhardwaj, et al.
0

In this paper, we address a fundamental research question of significant practical interest: Can certain theoretical characteristics of CNN architectures indicate a priori (i.e., without training) which models with highly different number of parameters and layers achieve a similar generalization performance? To answer this question, we model CNNs from a network science perspective and introduce a new, theoretically-grounded, architecture-level metric called NN-Mass. We also integrate, for the first time, the PAC-Bayes theory of generalization with small-world networks to discover new synergies among our proposed NN-Mass metric, architecture characteristics, and model generalization. With experiments on real datasets such as CIFAR-10/100, we provide extensive empirical evidence for our theoretical findings. Finally, we exploit these new insights for model compression and achieve up to 3x fewer parameters and FLOPS, while losing minimal accuracy (e.g., 96.82 dataset.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/14/2020

Understanding Generalization in Deep Learning via Tensor Methods

Deep neural networks generalize well on unseen data though the number of...
research
06/10/2020

Better Together: Resnet-50 accuracy with 13x fewer parameters and at 3x speed

Recent research on compressing deep neural networks has focused on reduc...
research
05/10/2023

Compressing neural network by tensor network with exponentially fewer variational parameters

Neural network (NN) designed for challenging machine learning tasks is i...
research
08/23/2017

A Neural Network Approach for Mixing Language Models

The performance of Neural Network (NN)-based language models is steadily...
research
04/20/2019

ChoiceNet: CNN learning through choice of multiple feature map representations

We introduce a new architecture called ChoiceNet where each layer of the...
research
03/31/2022

Exploiting Explainable Metrics for Augmented SGD

Explaining the generalization characteristics of deep learning is an eme...
research
09/19/2018

Identifying Generalization Properties in Neural Networks

While it has not yet been proven, empirical evidence suggests that model...

Please sign up or login with your details

Forgot password? Click here to reset