Trap of Feature Diversity in the Learning of MLPs

by   Dongrui Liu, et al.
Shanghai Jiao Tong University

In this paper, we discover a two-phase phenomenon in the learning of multi-layer perceptrons (MLPs). I.e., in the first phase, the training loss does not decrease significantly, but the similarity of features between different samples keeps increasing, which hurts the feature diversity. We explain such a two-phase phenomenon in terms of the learning dynamics of the MLP. Furthermore, we propose two normalization operations to eliminate the two-phase phenomenon, which avoids the decrease of the feature diversity and speeds up the training process.


page 1

page 2

page 3

page 4


Phase Diagram of Initial Condensation for Two-layer Neural Networks

The phenomenon of distinct behaviors exhibited by neural networks under ...

Polarization Diversity-enabled LOS/NLOS Identification via Carrier Phase Measurements

Provision of accurate localization is an increasingly important feature ...

Two-Phase Learning for Overcoming Noisy Labels

To counter the challenge associated with noise labels, the learning stra...

Improving the Diversity of Top-N Recommendation via Determinantal Point Process

Recommender systems take the key responsibility to help users discover i...

Phase transition in the knapsack problem

We examine the phase transition phenomenon for the Knapsack problem from...

A Study of Neural Collapse Phenomenon: Grassmannian Frame, Symmetry, Generalization

In this paper, we extends original Neural Collapse Phenomenon by proving...

RD-DPP: Rate-Distortion Theory Meets Determinantal Point Process to Diversify Learning Data Samples

In some practical learning tasks, such as traffic video analysis, the nu...

Please sign up or login with your details

Forgot password? Click here to reset