Connecting Weighted Automata, Tensor Networks and Recurrent Neural Networks through Spectral Learning

10/19/2020
by   Tianyu Li, et al.
0

In this paper, we present connections between three models used in different research fields: weighted finite automata (WFA) from formal languages and linguistics, recurrent neural networks used in machine learning, and tensor networks which encompasses a set of optimization techniques for high-order tensors used in quantum physics and numerical analysis. We first present an intrinsic relation between WFA and the tensor train decomposition, a particular form of tensor network. This relation allows us to exhibit a novel low rank structure of the Hankel matrix of a function computed by a WFA and to design an efficient spectral learning algorithm leveraging this structure to scale the algorithm up to very large Hankel matrices. We then unravel a fundamental connection between WFA and second-order recurrent neural networks (2-RNN): in the case of sequences of discrete symbols, WFA and 2-RNN with linear activation functions are expressively equivalent. Furthermore, we introduce the first provable learning algorithm for linear 2-RNN defined over sequences of continuous input vectors. This algorithm relies on estimating low rank sub-blocks of the Hankel tensor, from which the parameters of a linear 2-RNN can be provably recovered. The performances of the proposed learning algorithm are assessed in a simulation study on both synthetic and real-world data.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/04/2018

Connecting Weighted Automata and Recurrent Neural Networks through Spectral Learning

In this paper, we unravel a fundamental connection between weighted fini...
research
09/28/2020

Distillation of Weighted Automata from Recurrent Neural Networks using a Spectral Approach

This paper is an attempt to bridge the gap between deep learning and gra...
research
09/13/2017

Neural Network Based Nonlinear Weighted Finite Automata

Weighted finite automata (WFA) can expressively model functions defined ...
research
04/05/2019

Weighted Automata Extraction from Recurrent Neural Networks via Regression on State Spaces

We present a method to extract a weighted finite automaton (WFA) from a ...
research
10/20/2020

Quantum Tensor Networks, Stochastic Processes, and Weighted Automata

Modeling joint probability distributions over sequences has been studied...
research
08/22/2023

Low Tensor Rank Learning of Neural Dynamics

Learning relies on coordinated synaptic changes in recurrently connected...
research
07/05/2019

A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networks

We present a framework for compactly summarizing many recent results in ...

Please sign up or login with your details

Forgot password? Click here to reset