A Neural Language Model for Dynamically Representing the Meanings of Unknown Words and Entities in a Discourse

09/06/2017
by   Sosuke Kobayashi, et al.
0

This study addresses the problem of identifying the meaning of unknown words or entities in a discourse with respect to the word embedding approaches used in neural language models. We proposed a method for on-the-fly construction and exploitation of word embeddings in both the input and output layers of a neural model by tracking contexts. This extends the dynamic entity representation used in Kobayashi et al. (2016) and incorporates a copy mechanism proposed independently by Gu et al. (2016) and Gulcehre et al. (2016). In addition, we construct a new task and dataset called Anonymized Language Modeling for evaluating the ability to capture word meanings while reading. Experiments conducted using our novel dataset show that the proposed variant of RNN language model outperformed the baseline model. Furthermore, the experiments also demonstrate that dynamic updates of an output layer help a model predict reappearing entities, whereas those of an input layer are effective to predict words following reappearing entities.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/13/2019

Character n-gram Embeddings to Improve RNN Language Models

This paper proposes a novel Recurrent Neural Network (RNN) language mode...
research
04/19/2021

When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting

Since the seminal work of Mikolov et al. (2013a) and Bojanowski et al. (...
research
10/12/2016

Language Models with Pre-Trained (GloVe) Word Embeddings

In this work we implement a training of a Language Model (LM), using Rec...
research
01/09/2023

FullStop:Punctuation and Segmentation Prediction for Dutch with Transformers

When applying automated speech recognition (ASR) for Belgian Dutch (Van ...
research
01/11/2018

Enhancing Translation Language Models with Word Embedding for Information Retrieval

In this paper, we explore the usage of Word Embedding semantic resources...
research
09/28/2018

Adaptive Input Representations for Neural Language Modeling

We introduce adaptive input representations for neural language modeling...
research
02/15/2014

word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method

The word2vec software of Tomas Mikolov and colleagues (https://code.goog...

Please sign up or login with your details

Forgot password? Click here to reset