Stochastic Gradient Descent Learns State Equations with Nonlinear Activations

09/09/2018

∙

We study discrete time dynamical systems governed by the state equation h_t+1=ϕ(Ah_t+Bu_t). Here A,B are weight matrices, ϕ is an activation function, and u_t is the input data. This relation is the backbone of recurrent neural networks (e.g. LSTMs) which have broad applications in sequential learning tasks. We utilize stochastic gradient descent to learn the weight matrices from a finite input/state trajectory (u_t,h_t)_t=0^N. We prove that SGD estimate linearly converges to the ground truth weights while using near-optimal sample size. Our results apply to increasing activations whose derivatives are bounded away from zero. The analysis is based on i) a novel SGD convergence result with nonlinear activations and ii) careful statistical characterization of the state vector. Numerical experiments verify the fast convergence of SGD on ReLU and leaky ReLU in consistence with our theory.

READ FULL TEXT

Stochastic Gradient Descent Learns State Equations with Nonlinear Activations

Online Stochastic Gradient Descent Learns Linear Dynamical Systems from A Single Trajectory

Non-asymptotic and Accurate Learning of Nonlinear Dynamical Systems

Convergence of SGD in Learning ReLU Models with Separable Data

On the Stochastic Gradient Descent and Inverse Variance-flatness Relation in Artificial Neural Networks

Fitting ReLUs via SGD and Quantized SGD

A Letter on Convergence of In-Parameter-Linear Nonlinear Neural Architectures with Gradient Learnings

Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach

Stochastic Gradient Descent Learns State Equations with Nonlinear Activations

Related Research

Online Stochastic Gradient Descent Learns Linear Dynamical Systems from A Single Trajectory

Non-asymptotic and Accurate Learning of Nonlinear Dynamical Systems

Convergence of SGD in Learning ReLU Models with Separable Data

On the Stochastic Gradient Descent and Inverse Variance-flatness Relation in Artificial Neural Networks

Fitting ReLUs via SGD and Quantized SGD

A Letter on Convergence of In-Parameter-Linear Nonlinear Neural Architectures with Gradient Learnings

Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach