SuperNet – An efficient method of neural networks ensembling

03/29/2020
by   Ludwik Bukowski, et al.
0

The main flaw of neural network ensembling is that it is exceptionally demanding computationally, especially, if the individual sub-models are large neural networks, which must be trained separately. Having in mind that modern DNNs can be very accurate, they are already the huge ensembles of simple classifiers, and that one can construct more thrifty compressed neural net of a similar performance for any ensemble, the idea of designing the expensive SuperNets can be questionable. The widespread belief that ensembling increases the prediction time, makes it not attractive and can be the reason that the main stream of ML research is directed towards developing better loss functions and learning strategies for more advanced and efficient neural networks. On the other hand, all these factors make the architectures more complex what may lead to overfitting and high computational complexity, that is, to the same flaws for which the highly parametrized SuperNets ensembles are blamed. The goal of the master thesis is to speed up the execution time required for ensemble generation. Instead of training K inaccurate sub-models, each of them can represent various phases of training (representing various local minima of the loss function) of a single DNN [Huang et al., 2017; Gripov et al., 2018]. Thus, the computational performance of the SuperNet can be comparable to the maximum CPU time spent on training its single sub-model, plus usually much shorter CPU time required for training the SuperNet coupling factors.

READ FULL TEXT
research
02/19/2018

Understanding the Loss Surface of Neural Networks for Binary Classification

It is widely conjectured that the reason that training algorithms for ne...
research
06/27/2022

Transfer learning for ensembles: reducing computation time and keeping the diversity

Transferring a deep neural network trained on one problem to another req...
research
06/27/2022

Effective training-time stacking for ensembling of deep neural networks

Ensembling is a popular and effective method for improving machine learn...
research
06/20/2020

Collective Learning by Ensembles of Altruistic Diversifying Neural Networks

Combining the predictions of collections of neural networks often outper...
research
10/05/2020

Multi-Loss Sub-Ensembles for Accurate Classification with Uncertainty Estimation

Deep neural networks (DNNs) have made a revolution in numerous fields du...
research
02/03/2022

A Note on "Assessing Generalization of SGD via Disagreement"

Jiang et al. (2021) give empirical evidence that the average test error ...
research
12/11/2018

Synergy Effect between Convolutional Neural Networks and the Multiplicity of SMILES for Improvement of Molecular Prediction

In our study, we demonstrate the synergy effect between convolutional ne...

Please sign up or login with your details

Forgot password? Click here to reset