-
Optimal Regularization Can Mitigate Double Descent
Recent empirical and theoretical studies have shown that many learning a...
read it
-
Deep Double Descent: Where Bigger Models and More Data Hurt
We show that a variety of modern deep learning tasks exhibit a "double-d...
read it
-
Benign Overfitting and Noisy Features
Modern machine learning often operates in the regime where the number of...
read it
-
LambdaOpt: Learn to Regularize Recommender Models in Finer Levels
Recommendation models mainly deal with categorical variables, such as us...
read it
-
Exact expressions for double descent and implicit regularization via surrogate random design
Double descent refers to the phase transition that is exhibited by the g...
read it
-
Early Stopping in Deep Networks: Double Descent and How to Eliminate it
Over-parameterized models, in particular deep networks, often exhibit a ...
read it
-
Optimizing generalization on the train set: a novel gradient-based framework to train parameters and hyperparameters simultaneously
Generalization is a central problem in Machine Learning. Most prediction...
read it
Avoiding The Double Descent Phenomenon of Random Feature Models Using Hybrid Regularization
We demonstrate the ability of hybrid regularization methods to automatically avoid the double descent phenomenon arising in the training of random feature models (RFM). The hallmark feature of the double descent phenomenon is a spike in the regularization gap at the interpolation threshold, i.e. when the number of features in the RFM equals the number of training samples. To close this gap, the hybrid method considered in our paper combines the respective strengths of the two most common forms of regularization: early stopping and weight decay. The scheme does not require hyperparameter tuning as it automatically selects the stopping iteration and weight decay hyperparameter by using generalized cross-validation (GCV). This also avoids the necessity of a dedicated validation set. While the benefits of hybrid methods have been well-documented for ill-posed inverse problems, our work presents the first use case in machine learning. To expose the need for regularization and motivate hybrid methods, we perform detailed numerical experiments inspired by image classification. In those examples, the hybrid scheme successfully avoids the double descent phenomenon and yields RFMs whose generalization is comparable with classical regularization approaches whose hyperparameters are tuned optimally using the test data. We provide our MATLAB codes for implementing the numerical experiments in this paper at https://github.com/EmoryMLIP/HybridRFM.
READ FULL TEXT
Comments
There are no comments yet.