A Compositional Approach to Language Modeling

04/01/2016
by   Kushal Arora, et al.
0

Traditional language models treat language as a finite state automaton on a probability space over words. This is a very strong assumption when modeling something inherently complex such as language. In this paper, we challenge this by showing how the linear chain assumption inherent in previous work can be translated into a sequential composition tree. We then propose a new model that marginalizes over all possible composition trees thereby removing any underlying structural assumptions. As the partition function of this new model is intractable, we use a recently proposed sentence level evaluation metric Contrastive Entropy to evaluate our model. Given this new evaluation metric, we report more than 100 of the art recurrent neural network based language models.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/03/2016

Contrastive Entropy: A new evaluation metric for unnormalized language models

Perplexity (per word) is the most widely used metric for evaluating lang...
research
04/07/2019

Unsupervised Recurrent Neural Network Grammars

Recurrent neural network grammars (RNNG) are generative models of langua...
research
01/15/2017

Dialog Context Language Modeling with Recurrent Neural Networks

In this work, we propose contextual language models that incorporate dia...
research
12/21/2016

An Empirical Study of Language CNN for Image Captioning

Language Models based on recurrent neural networks have dominated recent...
research
10/24/2021

Distributed neural encoding of binding to thematic roles

A framework and method are proposed for the study of constituent composi...
research
02/11/2013

Toric grammars: a new statistical approach to natural language modeling

We propose a new statistical model for computational linguistics. Rather...
research
05/21/2018

Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers

Numeracy is the ability to understand and work with numbers. It is a nec...

Please sign up or login with your details

Forgot password? Click here to reset