Log In Sign Up

Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles

by   Ghazi Felhi, et al.

Linking neural representations to linguistic factors is crucial in order to build and analyze NLP models interpretable by humans. Among these factors, syntactic roles (e.g. subjects, direct objects,…) and their realizations are essential markers since they can be understood as a decomposition of predicative structures and thus the meaning of sentences. Starting from a deep probabilistic generative model with attention, we measure the interaction between latent variables and realizations of syntactic roles and show that it is possible to obtain, without supervision, representations of sentences where different syntactic roles correspond to clearly identified different latent variables. The probabilistic model we propose is an Attention-Driven Variational Autoencoder (ADVAE). Drawing inspiration from Transformer-based machine translation models, ADVAEs enable the analysis of the interactions between latent variables and input tokens through attention. We also develop an evaluation protocol to measure disentanglement with regard to the realizations of syntactic roles. This protocol is based on attention maxima for the encoder and on latent variable perturbations for the decoder. Our experiments on raw English text from the SNLI dataset show that i) disentanglement of syntactic roles can be induced without supervision, ii) ADVAE separates syntactic roles better than classical sequence VAEs and Transformer VAEs, iii) realizations of syntactic roles can be separately modified in sentences by mere intervention on the associated latent variables. Our work constitutes a first step towards unsupervised controllable content generation. The code for our work is publicly available.


page 8

page 16

page 19

page 20


Disentangling semantics in language through VAEs and a certain architectural choice

We present an unsupervised method to obtain disentangled representations...

Generative Statistical Models with Self-Emergent Grammar of Chord Sequences

Generative statistical models of chord sequences play crucial roles in m...

The heads hypothesis: A unifying statistical approach towards understanding multi-headed attention in BERT

Multi-headed attention heads are a mainstay in transformer-based models....

Syntax-Infused Variational Autoencoder for Text Generation

We present a syntax-infused variational autoencoder (SIVAE), that integr...

Factorising Meaning and Form for Intent-Preserving Paraphrasing

We propose a method for generating paraphrases of English questions that...

Exploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAEs

We propose a generative model for text generation, which exhibits disent...

Morpho-MNIST: Quantitative Assessment and Diagnostics for Representation Learning

Revealing latent structure in data is an active field of research, havin...