Alternative structures for character-level RNNs

11/19/2015
by   Piotr Bojanowski, et al.
0

Recurrent neural networks are convenient and efficient models for language modeling. However, when applied on the level of characters instead of words, they suffer from several problems. In order to successfully model long-term dependencies, the hidden representation needs to be large. This in turn implies higher computational costs, which can become prohibitive in practice. We propose two alternative structural modifications to the classical RNN model. The first one consists on conditioning the character level representation on the previous word representation. The other one uses the character history to condition the output probability. We evaluate the performance of the two proposed modifications on challenging, multi-lingual real world data.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/13/2016

Character-Level Language Modeling with Hierarchical Recurrent Neural Networks

Recurrent neural network (RNN) based character-level language models (CL...
research
08/26/2015

Character-Aware Neural Language Models

We describe a simple neural language model that relies only on character...
research
10/26/2017

Rotational Unit of Memory

The concepts of unitary evolution matrices and associative memory have b...
research
05/24/2017

Fast-Slow Recurrent Neural Networks

Processing sequential data of variable length is a major challenge in a ...
research
08/09/2018

Character-Level Language Modeling with Deeper Self-Attention

LSTMs and other RNN variants have shown strong performance on character-...
research
11/14/2016

Attending to Characters in Neural Sequence Labeling Models

Sequence labeling architectures use word embeddings for capturing simila...
research
11/22/2019

Learning Multi-level Dependencies for Robust Word Recognition

Robust language processing systems are becoming increasingly important g...

Please sign up or login with your details

Forgot password? Click here to reset