-
Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization
The early phase of training has been shown to be important in two ways f...
12/28/2020 ∙ by Stanisław Jastrzębski, et al. ∙
16
∙
share
read it
-
Advantages of biologically-inspired adaptive neural activation in RNNs during learning
Dynamic adaptation in single-neuron response plays a fundamental role in...
06/22/2020 ∙ by Victor Geadah, et al. ∙
18
∙
share
read it
-
Untangling tradeoffs between recurrence and self-attention in neural networks
Attention and self-attention mechanisms, inspired by cognitive processes...
06/16/2020 ∙ by Giancarlo Kerg, et al. ∙
0
∙
share
read it
-
Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
A recent strategy to circumvent the exploding and vanishing gradient pro...
05/28/2019 ∙ by Giancarlo Kerg, et al. ∙
1
∙
share
read it
-
h-detach: Modifying the LSTM Gradient Towards Better Optimization
Recurrent neural networks are known for their notorious exploding and va...
10/06/2018 ∙ by Devansh Arpit, et al. ∙
0
∙
share
read it
