Emergence of hierarchical modes from deep learning

08/21/2022
by   Chan Li, et al.
0

Large-scale deep neural networks consume expensive training costs, but the training results in less-interpretable weight matrices constructing the networks. Here, we propose a mode decomposition learning that can interpret the weight matrices as a hierarchy of latent modes. These modes are akin to patterns in physics studies of memory networks. The mode decomposition learning not only saves a significant large amount of training costs, but also explains the network performance with the leading modes. The mode learning scheme shows a progressively compact latent space across the network hierarchy, and the least number of modes increases only logarithmically with the network width. Our mode decomposition learning is also studied in an analytic on-line learning setting, which reveals multi-stage of learning dynamics. Therefore, the proposed mode decomposition learning points to a cheap and interpretable route towards the magical deep learning.

READ FULL TEXT

page 2

page 3

page 4

research
08/10/2021

Deep Learning Enhanced Dynamic Mode Decomposition

Koopman operator theory shows how nonlinear dynamical systems can be rep...
research
07/19/2019

Kernel Mode Decomposition and programmable/interpretable regression networks

Mode decomposition is a prototypical pattern recognition problem that ca...
research
06/30/2019

Directivity Modes of Earthquake Populations with Unsupervised Learning

We present a novel approach for resolving modes of rupture directivity i...
research
11/25/2020

Automatic Identification of MHD Modes in Magnetic Fluctuations Spectrograms using Deep Learning Techniques

The control and mitigation of MHD oscillations modes is an open problem ...
research
12/18/2022

The Underlying Correlated Dynamics in Neural Training

Training of neural networks is a computationally intensive task. The sig...
research
07/17/2017

Current-mode Memristor Crossbars for Neuromemristive Systems

Motivated by advantages of current-mode design, this brief contribution ...
research
07/08/2022

Convolution Neural Network based Mode Decomposition for Degenerated Modes via Multiple Images from Polarizers

In this paper, a mode decomposition (MD) method for degenerated modes ha...

Please sign up or login with your details

Forgot password? Click here to reset