Document Neural Autoregressive Distribution Estimation

by   Stanislas Lauly, et al.
Université de Sherbrooke

We present an approach based on feed-forward neural networks for learning the distribution of textual documents. This approach is inspired by the Neural Autoregressive Distribution Estimator(NADE) model, which has been shown to be a good estimator of the distribution of discrete-valued igh-dimensional vectors. In this paper, we present how NADE can successfully be adapted to the case of textual data, retaining from NADE the property that sampling or computing the probability of observations can be done exactly and efficiently. The approach can also be used to learn deep representations of documents that are competitive to those learned by the alternative topic modeling approaches. Finally, we describe how the approach can be combined with a regular neural network N-gram model and substantially improve its performance, by making its learned representation sensitive to the larger, document-specific context.


page 1

page 2

page 3

page 4


Neural Autoregressive Distribution Estimation

We present Neural Autoregressive Distribution Estimation (NADE) models, ...

A Deep and Autoregressive Approach for Topic Modeling of Multimodal Data

Topic modeling based on latent Dirichlet allocation (LDA) has been a fra...

Document Informed Neural Autoregressive Topic Models

Context information around words helps in determining their actual meani...

FLATM: A Fuzzy Logic Approach Topic Model for Medical Documents

One of the challenges for text analysis in medical domains is analyzing ...

A Supervised Neural Autoregressive Topic Model for Simultaneous Image Classification and Annotation

Topic modeling based on latent Dirichlet allocation (LDA) has been a fra...

Binary Paragraph Vectors

Recently Le & Mikolov described two log-linear models, called Paragraph ...

Learning Discrete Distributions by Dequantization

Media is generally stored digitally and is therefore discrete. Many succ...

Please sign up or login with your details

Forgot password? Click here to reset