Language Modeling with Power Low Rank Ensembles

12/26/2013
by   Ankur P. Parikh, et al.
0

We present power low rank ensembles (PLRE), a flexible framework for n-gram language modeling where ensembles of low rank matrices and tensors are used to obtain smoothed probability estimates of words in context. Our method can be understood as a generalization of n-gram modeling to non-integer n, and includes standard techniques such as absolute discounting and Kneser-Ney smoothing as special cases. PLRE training is efficient and our approach outperforms state-of-the-art modified Kneser Ney baselines in terms of perplexity on large corpora as well as on BLEU score in a downstream machine translation task.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/27/2022

CP decomposition and low-rank approximation of antisymmetric tensors

For the antisymmetric tensors the paper examines a low-rank approximatio...
research
08/17/2015

Molding CNNs for text: non-linear, non-consecutive convolutions

The success of deep learning often derives from well-chosen operational ...
research
03/07/2017

Data Noising as Smoothing in Neural Network Language Models

Data noising is an effective technique for regularizing neural network m...
research
12/04/2019

Finding entries of maximum absolute value in low-rank tensors

We present an iterative method for the search of extreme entries in low-...
research
06/23/2017

Comparison of Modified Kneser-Ney and Witten-Bell Smoothing Techniques in Statistical Language Model of Bahasa Indonesia

Smoothing is one technique to overcome data sparsity in statistical lang...
research
10/26/2022

Residual Learning of Neural Text Generation with n-gram Language Model

N-gram language models (LM) have been largely superseded by neural LMs a...
research
06/25/2018

Handling Massive N-Gram Datasets Efficiently

This paper deals with the two fundamental problems concerning the handli...

Please sign up or login with your details

Forgot password? Click here to reset