Generalized Tensor Models for Recurrent Neural Networks

01/30/2019
by   Valentin Khrulkov, et al.
8

Recurrent Neural Networks (RNNs) are very successful at solving challenging problems with sequential data. However, this observed efficiency is not yet entirely explained by theory. It is known that a certain class of multiplicative RNNs enjoys the property of depth efficiency --- a shallow network of exponentially large width is necessary to realize the same score function as computed by such an RNN. Such networks, however, are not very often applied to real life tasks. In this work, we attempt to reduce the gap between theory and practice by extending the theoretical analysis to RNNs which employ various nonlinearities, such as Rectified Linear Unit (ReLU), and show that they also benefit from properties of universality and depth efficiency. Our theoretical results are verified by a series of extensive computational experiments.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/20/2013

How to Construct Deep Recurrent Neural Networks

In this paper, we explore different ways to extend a recurrent neural ne...
research
02/28/2018

Tensor Decomposition for Compressing Recurrent Neural Network

In the machine learning fields, Recurrent Neural Network (RNN) has becom...
research
11/25/2022

Minimal Width for Universal Property of Deep RNN

A recurrent neural network (RNN) is a widely used deep-learning network ...
research
02/09/2022

Stability Analysis of Recurrent Neural Networks by IQC with Copositive Mutipliers

This paper is concerned with the stability analysis of the recurrent neu...
research
02/26/2016

Architectural Complexity Measures of Recurrent Neural Networks

In this paper, we systematically analyze the connecting architectures of...
research
09/18/2020

Recurrent Graph Tensor Networks

Recurrent Neural Networks (RNNs) are among the most successful machine l...
research
03/10/2022

Robustness Analysis of Classification Using Recurrent Neural Networks with Perturbed Sequential Input

For a given stable recurrent neural network (RNN) that is trained to per...

Please sign up or login with your details

Forgot password? Click here to reset