Neural Capacitance: A New Perspective of Neural Network Selection via Edge Dynamics

01/11/2022
by   Chunheng Jiang, et al.
0

Efficient model selection for identifying a suitable pre-trained neural network to a downstream task is a fundamental yet challenging task in deep learning. Current practice requires expensive computational costs in model training for performance prediction. In this paper, we propose a novel framework for neural network selection by analyzing the governing dynamics over synaptic connections (edges) during training. Our framework is built on the fact that back-propagation during neural network training is equivalent to the dynamical evolution of synaptic connections. Therefore, a converged neural network is associated with an equilibrium state of a networked system composed of those edges. To this end, we construct a network mapping ϕ, converting a neural network G_A to a directed line graph G_B that is defined on those edges in G_A. Next, we derive a neural capacitance metric β_ eff as a predictive measure universally capturing the generalization capability of G_A on the downstream task using only a handful of early training results. We carried out extensive experiments using 17 popular pre-trained ImageNet models and five benchmark datasets, including CIFAR10, CIFAR100, SVHN, Fashion MNIST and Birds, to evaluate the fine-tuning performance of our framework. Our neural capacitance metric is shown to be a powerful indicator for model selection based only on early training results and is more efficient than state-of-the-art methods.

READ FULL TEXT
research
01/29/2021

A linearized framework and a new benchmark for model selection for fine-tuning

Fine-tuning from a collection of models pre-trained on different domains...
research
06/15/2023

LOVM: Language-Only Vision Model Selection

Pre-trained multi-modal vision-language models (VLMs) are becoming incre...
research
01/03/2018

ScreenerNet: Learning Curriculum for Neural Networks

We propose to learn a curriculum or a syllabus for supervised learning w...
research
09/19/2020

Enabling Resource-Aware Mapping of Spiking Neural Networks via Spatial Decomposition

With growing model complexity, mapping Spiking Neural Network (SNN)-base...
research
08/07/2021

NASOA: Towards Faster Task-oriented Online Fine-tuning with a Zoo of Models

Fine-tuning from pre-trained ImageNet models has been a simple, effectiv...
research
07/31/2021

Provably Efficient Lottery Ticket Discovery

The lottery ticket hypothesis (LTH) claims that randomly-initialized, de...
research
10/22/2019

Composite Neural Network: Theory and Application to PM2.5 Prediction

This work investigates the framework and performance issues of the compo...

Please sign up or login with your details

Forgot password? Click here to reset