Character-Word LSTM Language Models

04/10/2017
by   Lyan Verwimp, et al.
0

We present a Character-Word Long Short-Term Memory Language Model which both reduces the perplexity with respect to a baseline word-level language model and reduces the number of parameters of the model. Character information can reveal structural (dis)similarities between words and can even be used when a word is out-of-vocabulary, thus improving the modeling of infrequent and unknown words. By concatenating word and character embeddings, we achieve up to 2.77 improvement on English compared to a baseline model with a similar amount of parameters and 4.57 models with a larger number of parameters.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/06/2016

Gated Word-Character Recurrent Language Model

We introduce a recurrent neural network language model (RNN-LM) with lon...
research
12/10/2016

A Character-Word Compositional Neural Language Model for Finnish

Inspired by recent research, we explore ways to model the highly morphol...
research
10/13/2016

Compressing Neural Language Models by Sparse Word Representations

Neural networks are among the state-of-the-art techniques for language m...
research
04/23/2017

Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling

Fixed-vocabulary language models fail to account for one of the most cha...
research
08/18/2017

Syllable-level Neural Language Model for Agglutinative Language

Language models for agglutinative languages have always been hindered in...
research
08/31/2018

Indicatements that character language models learn English morpho-syntactic units and regularities

Character language models have access to surface morphological patterns,...
research
11/23/2022

Word-Level Representation From Bytes For Language Modeling

Modern language models mostly take sub-words as input, a design that bal...

Please sign up or login with your details

Forgot password? Click here to reset