Quaternion Recurrent Neural Networks

06/12/2018
by   Titouan Parcollet, et al.
0

Recurrent neural networks (RNNs) are powerful architectures to model sequential data, due to their capability to learn short and long-term dependencies between the basic elements of a sequence. Nonetheless, popular tasks such as speech or images recognition, involve multi-dimensional input features that are characterized by strong internal dependencies between the dimensions of the input vector. We propose a novel quaternion recurrent neural network (QRNN) that takes into account both the external relations and these internal structural dependencies with the quaternion algebra. Similarly to capsules, quaternions allow the QRNN to code internal dependencies by composing and processing multidimensional features as single entities, while the recurrent operation reveals correlations between the elements composing the sequence. We show that the QRNN achieves better performances in both a synthetic memory copy task and in a realistic application of phoneme recognition. Finally, we show that the QRNN reduces by a factor of 3x the number of free parameters needed, compared to RNNs to reach equal or even better results, leading to a more compact representation of the relevant information.

READ FULL TEXT
research
11/06/2018

Bidirectional Quaternion Long-Short Term Memory Recurrent Neural Networks for Speech Recognition

Recurrent neural networks (RNN) are at the core of modern automatic spee...
research
05/14/2007

Multi-Dimensional Recurrent Neural Networks

Recurrent neural networks (RNNs) have proved effective at one dimensiona...
research
03/07/2017

Linguistic Knowledge as Memory for Recurrent Neural Networks

Training recurrent neural networks to model long term dependencies is di...
research
01/09/2020

Internal representation dynamics and geometry in recurrent neural networks

The efficiency of recurrent neural networks (RNNs) in dealing with seque...
research
06/29/2020

Incremental Training of a Recurrent Neural Network Exploiting a Multi-Scale Dynamic Memory

The effectiveness of recurrent neural networks can be largely influenced...
research
08/02/2019

Universal Transforming Geometric Network

The recurrent geometric network (RGN), the first end-to-end differentiab...
research
03/22/2021

Alleviate Exposure Bias in Sequence Prediction with Recurrent Neural Networks

A popular strategy to train recurrent neural networks (RNNs), known as “...

Please sign up or login with your details

Forgot password? Click here to reset