Toward Better Storylines with Sentence-Level Language Models

05/11/2020
by   Daphne Ippolito, et al.
0

We propose a sentence-level language model which selects the next sentence in a story from a finite set of fluent alternatives. Since it does not need to model fluency, the sentence-level language model can focus on longer range dependencies, which are crucial for multi-sentence coherence. Rather than dealing with individual words, our method treats the story so far as a list of pre-trained sentence embeddings and predicts an embedding for the next sentence, which is more efficient than predicting word embeddings. Notably this allows us to consider a large number of candidates for the next sentence during training. We demonstrate the effectiveness of our approach with state-of-the-art accuracy on the unsupervised Story Cloze task and with promising results on larger-scale next sentence prediction tasks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/18/2023

Ditto: A Simple and Efficient Approach to Improve Sentence Embeddings

Prior studies diagnose the anisotropy problem in sentence representation...
research
02/25/2016

Automated Word Prediction in Bangla Language Using Stochastic Language Models

Word completion and word prediction are two important phenomena in typin...
research
04/23/2018

Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model

We show how to deploy recurrent neural networks within a hierarchical Ba...
research
03/15/2018

A Simple and Effective Approach to the Story Cloze Test

In the Story Cloze Test, a system is presented with a 4-sentence prompt ...
research
03/29/2023

Larger Probes Tell a Different Story: Extending Psycholinguistic Datasets Via In-Context Learning

Language model probing is often used to test specific capabilities of th...
research
04/25/2023

Compressing Sentence Representation with maximum Coding Rate Reduction

In most natural language inference problems, sentence representation is ...
research
10/13/2022

Re3: Generating Longer Stories With Recursive Reprompting and Revision

We consider the problem of automatically generating longer stories of ov...

Please sign up or login with your details

Forgot password? Click here to reset