
-
Gradient Starvation: A Learning Proclivity in Neural Networks
We identify and formalize a fundamental gradient descent phenomenon resu...
read it
-
LEAD: Least-Action Dynamics for Min-Max Optimization
Adversarial formulations in machine learning have rekindled interest in ...
read it
-
Implicit Regularization in Deep Learning: A View from Function Space
We approach the problem of implicit regularization in deep learning from...
read it
-
Learning to Combine Top-Down and Bottom-Up Signals in Recurrent Neural Networks with Attention over Modules
Robust perception relies on both bottom-up and top-down signals. Bottom-...
read it
-
On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools
Recurrent neural networks (RNNs) have been successfully applied to a var...
read it
-
Advantages of biologically-inspired adaptive neural activation in RNNs during learning
Dynamic adaptation in single-neuron response plays a fundamental role in...
read it
-
Untangling tradeoffs between recurrence and self-attention in neural networks
Attention and self-attention mechanisms, inspired by cognitive processes...
read it
-
Internal representation dynamics and geometry in recurrent neural networks
The efficiency of recurrent neural networks (RNNs) in dealing with seque...
read it
-
Dimensionality compression and expansion in Deep Neural Networks
Datasets such as images, text, or movies are embedded in high-dimensiona...
read it
-
Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
A recent strategy to circumvent the exploding and vanishing gradient pro...
read it