-
Gradient Starvation: A Learning Proclivity in Neural Networks
We identify and formalize a fundamental gradient descent phenomenon resu...
11/18/2020 ∙ by Mohammad Pezeshki, et al. ∙
1
∙
share
read it
-
LEAD: Least-Action Dynamics for Min-Max Optimization
Adversarial formulations in machine learning have rekindled interest in ...
10/26/2020 ∙ by Reyhane Askari Hemmat, et al. ∙
22
∙
share
read it
-
Implicit Regularization in Deep Learning: A View from Function Space
We approach the problem of implicit regularization in deep learning from...
08/03/2020 ∙ by Aristide Baratin, et al. ∙
39
∙
share
read it
-
Learning to Combine Top-Down and Bottom-Up Signals in Recurrent Neural Networks with Attention over Modules
Robust perception relies on both bottom-up and top-down signals. Bottom-...
06/30/2020 ∙ by Sarthak Mittal, et al. ∙
11
∙
share
read it
-
On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools
Recurrent neural networks (RNNs) have been successfully applied to a var...
06/25/2020 ∙ by Ryan Vogt, et al. ∙
0
∙
share
read it
-
Advantages of biologically-inspired adaptive neural activation in RNNs during learning
Dynamic adaptation in single-neuron response plays a fundamental role in...
06/22/2020 ∙ by Victor Geadah, et al. ∙
18
∙
share
read it
-
Untangling tradeoffs between recurrence and self-attention in neural networks
Attention and self-attention mechanisms, inspired by cognitive processes...
06/16/2020 ∙ by Giancarlo Kerg, et al. ∙
0
∙
share
read it
-
Internal representation dynamics and geometry in recurrent neural networks
The efficiency of recurrent neural networks (RNNs) in dealing with seque...
01/09/2020 ∙ by Stefan Horoi, et al. ∙
0
∙
share
read it
-
Dimensionality compression and expansion in Deep Neural Networks
Datasets such as images, text, or movies are embedded in high-dimensiona...
06/02/2019 ∙ by Stefano Recanatesi, et al. ∙
0
∙
share
read it
-
Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
A recent strategy to circumvent the exploding and vanishing gradient pro...
05/28/2019 ∙ by Giancarlo Kerg, et al. ∙
1
∙
share
read it
