
Connecting Weighted Automata and Recurrent Neural Networks through Spectral Learning
In this paper, we unravel a fundamental connection between weighted fini...
read it

Distillation of Weighted Automata from Recurrent Neural Networks using a Spectral Approach
This paper is an attempt to bridge the gap between deep learning and gra...
read it

Neural Network Based Nonlinear Weighted Finite Automata
Weighted finite automata (WFA) can expressively model functions defined ...
read it

Weighted Automata Extraction from Recurrent Neural Networks via Regression on State Spaces
We present a method to extract a weighted finite automaton (WFA) from a ...
read it

Quantum Tensor Networks, Stochastic Processes, and Weighted Automata
Modeling joint probability distributions over sequences has been studied...
read it

A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networks
We present a framework for compactly summarizing many recent results in ...
read it

Deep Polynomial Neural Networks
Deep Convolutional Neural Networks (DCNNs) are currently the method of c...
read it
Connecting Weighted Automata, Tensor Networks and Recurrent Neural Networks through Spectral Learning
In this paper, we present connections between three models used in different research fields: weighted finite automata (WFA) from formal languages and linguistics, recurrent neural networks used in machine learning, and tensor networks which encompasses a set of optimization techniques for highorder tensors used in quantum physics and numerical analysis. We first present an intrinsic relation between WFA and the tensor train decomposition, a particular form of tensor network. This relation allows us to exhibit a novel low rank structure of the Hankel matrix of a function computed by a WFA and to design an efficient spectral learning algorithm leveraging this structure to scale the algorithm up to very large Hankel matrices. We then unravel a fundamental connection between WFA and secondorder recurrent neural networks (2RNN): in the case of sequences of discrete symbols, WFA and 2RNN with linear activation functions are expressively equivalent. Furthermore, we introduce the first provable learning algorithm for linear 2RNN defined over sequences of continuous input vectors. This algorithm relies on estimating low rank subblocks of the Hankel tensor, from which the parameters of a linear 2RNN can be provably recovered. The performances of the proposed learning algorithm are assessed in a simulation study on both synthetic and realworld data.
READ FULL TEXT
Comments
There are no comments yet.