DeepAI AI Chat
Log In Sign Up

Dodging the Sparse Double Descent

by   Victor Quétu, et al.

This paper presents an approach to addressing the issue of over-parametrization in deep neural networks, more specifically by avoiding the “sparse double descent” phenomenon. The authors propose a learning framework that allows avoidance of this phenomenon and improves generalization, an entropy measure to provide more insights on its insurgence, and provide a comprehensive quantitative analysis of various factors such as re-initialization methods, model width and depth, and dataset noise. The proposed approach is supported by experimental results achieved using typical adversarial learning setups. The source code to reproduce the experiments is provided in the supplementary materials and will be publicly released upon acceptance of the paper.


Double Descent of Discrepancy: A Task-, Data-, and Model-Agnostic Phenomenon

In this paper, we studied two identically-trained neural networks (i.e. ...

Sparse Double Descent: Where Network Pruning Aggravates Overfitting

People usually believe that network pruning not only reduces the computa...

Geometric Regularization from Overparameterization explains Double Descent and other findings

The volume of the distribution of possible weight configurations associa...

Can we avoid Double Descent in Deep Neural Networks?

Finding the optimal size of deep learning models is very actual and of b...

Do Deeper Convolutional Networks Perform Better?

Over-parameterization is a recent topic of much interest in the machine ...

Understanding the double descent curve in Machine Learning

The theory of bias-variance used to serve as a guide for model selection...

Some Results of Experimental Check of The Model of the Object Innovativeness Quantitative Evaluation

The paper presents the results of the experiments that were conducted to...