DeepAI AI Chat
Log In Sign Up

Topic Compositional Neural Language Model

12/28/2017
by   Wenlin Wang, et al.
0

We propose a Topic Compositional Neural Language Model (TCNLM), a novel method designed to simultaneously capture both the global semantic meaning and the local word ordering structure in a document. The TCNLM learns the global semantic coherence of a document via a neural topic model, and the probability of each learned latent topic is further used to build a Mixture-of-Experts (MoE) language model, where each expert (corresponding to one topic) is a recurrent neural network (RNN) that accounts for learning the local structure of a word sequence. In order to train the MoE model efficiently, a matrix factorization method is applied, by extending each weight matrix of the RNN to be an ensemble of topic-dependent weight matrices. The degree to which each member of the ensemble is used is tied to the document-dependent probability of the corresponding topics. Experimental results on several corpora show that the proposed approach outperforms both a pure RNN-based model and other topic-guided language models. Further, our model yields sensible topics, and also has the capacity to generate meaningful sentences conditioned on given topics.

READ FULL TEXT

page 1

page 2

page 3

page 4

11/05/2016

TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency

In this paper, we propose TopicRNN, a recurrent neural network (RNN)-bas...
12/21/2019

Recurrent Hierarchical Topic-Guided Neural Language Models

To simultaneously capture syntax and global semantics from a text corpus...
04/26/2017

Topically Driven Neural Language Model

Language models are typically applied at the sentence level, without acc...
08/21/2022

GRETEL: Graph Contrastive Topic Enhanced Language Model for Long Document Extractive Summarization

Recently, neural topic models (NTMs) have been incorporated into pre-tra...
04/20/2018

Lightweight Adaptive Mixture of Neural and N-gram Language Models

It is often the case that the best performing language model is an ensem...
12/23/2018

Mixed Membership Recurrent Neural Networks

Models for sequential data such as the recurrent neural network (RNN) of...