The Vanishing Decision Boundary Complexity and the Strong First Component

11/25/2022
by   Hengshuai Yao, et al.
0

We show that unlike machine learning classifiers, there are no complex boundary structures in the decision boundaries for well-trained deep models. However, we found that the complicated structures do appear in training but they vanish shortly after shaping. This is a pessimistic news if one seeks to capture different levels of complexity in the decision boundary for understanding generalization, which works well in machine learning. Nonetheless, we found that the decision boundaries of predecessor models on the training data are reflective of the final model's generalization. We show how to use the predecessor decision boundaries for studying the generalization of deep models. We have three major findings. One is on the strength of the first principle component of deep models, another about the singularity of optimizers, and the other on the effects of the skip connections in ResNets. Code is at https://github.com/hengshu1/decision_boundary_github.

READ FULL TEXT

page 1

page 3

page 5

research
12/18/2020

ROBY: Evaluating the Robustness of a Deep Model by its Decision Boundaries

With the successful application of deep learning models in many real-wor...
research
03/15/2022

Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective

We discuss methods for visualizing neural network decision boundaries an...
research
08/07/2019

Investigating Decision Boundaries of Trained Neural Networks

Deep learning models have been the subject of study from various perspec...
research
02/15/2020

Hold me tight! Influence of discriminative features on deep network boundaries

Important insights towards the explainability of neural networks and the...
research
06/12/2023

Unveiling the Hessian's Connection to the Decision Boundary

Understanding the properties of well-generalizing minima is at the heart...
research
01/15/2021

Heating up decision boundaries: isocapacitory saturation, adversarial scenarios and generalization bounds

In the present work we study classifiers' decision boundaries via Browni...
research
03/27/2023

Multiphysics discovery with moving boundaries using Ensemble SINDy and Peridynamic Differential Operator

This study proposes a novel framework for learning the underlying physic...

Please sign up or login with your details

Forgot password? Click here to reset