Explaining Away Syntactic Structure in Semantic Document Representations

06/05/2018
by   Erik Holmer, et al.
0

Most generative document models act on bag-of-words input in an attempt to focus on the semantic content and thereby partially forego syntactic information. We argue that it is preferable to keep the original word order intact and explicitly account for the syntactic structure instead. We propose an extension to the Neural Variational Document Model (Miao et al., 2016) that does exactly that to separate local (syntactic) context from the global (semantic) representation of the document. Our model builds on the variational autoencoder framework to define a generative document model based on next-word prediction. We name our approach Sequence-Aware Variational Autoencoder since in contrast to its predecessor, it operates on the true input sequence. In a series of experiments we observe stronger topicality of the learned representations as well as increased robustness to syntactic noise in our training data.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/09/2018

Recurrent Neural Network-Based Semantic Variational Autoencoder for Sequence-to-Sequence Learning

Sequence-to-sequence (Seq2seq) models have played an import role in the ...
research
08/23/2018

Exploiting Rich Syntactic Information for Semantic Parsing with Graph-to-Sequence Model

Existing neural semantic parsers mainly utilize a sequence encoder, i.e....
research
02/25/2010

Syntactic Topic Models

The syntactic topic model (STM) is a Bayesian nonparametric model of lan...
research
06/02/2021

Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization

Abstractive summarization, the task of generating a concise summary of i...
research
03/08/2021

A Topological Approach to Compare Document Semantics Based on a New Variant of Syntactic N-grams

This paper delivers a new perspective of thinking and utilizing syntacti...
research
02/09/2019

Biadversarial Variational Autoencoder

In the original version of the Variational Autoencoder, Kingma et al. as...
research
02/06/2022

Enhancing variational generation through self-decomposition

In this article we introduce the notion of Split Variational Autoencoder...

Please sign up or login with your details

Forgot password? Click here to reset