Disentangling Representations of Text by Masking Transformers

04/14/2021
by   Xiongyi Zhang, et al.
0

Representations from large pretrained models such as BERT encode a range of features into monolithic vectors, affording strong predictive accuracy across a multitude of downstream tasks. In this paper we explore whether it is possible to learn disentangled representations by identifying existing subnetworks within pretrained models that encode distinct, complementary aspect representations. Concretely, we learn binary masks over transformer weights or hidden units to uncover subsets of features that correlate with a specific factor of variation; this eliminates the need to train a disentangled model from scratch for a particular task. We evaluate this method with respect to its ability to disentangle representations of sentiment from genre in movie reviews, "toxicity" from dialect in Tweets, and syntax from semantics. By combining masking with magnitude pruning we find that we can identify sparse subnetworks within BERT that strongly encode particular aspects (e.g., toxicity) while only weakly encoding others (e.g., race). Moreover, despite only learning masks, we find that disentanglement-via-masking performs as well as – and often better than – previously proposed methods based on variational autoencoders and adversarial training.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/26/2020

Masking as an Efficient Alternative to Finetuning for Pretrained Language Models

We present an efficient method of utilizing pretrained language models, ...
research
10/06/2020

Are "Undocumented Workers" the Same as "Illegal Aliens"? Disentangling Denotation and Connotation in Vector Spaces

In politics, neologisms are frequently invented for partisan objectives....
research
06/01/2017

Learning Disentangled Representations with Semi-Supervised Deep Generative Models

Variational autoencoders (VAEs) learn representations of data by jointly...
research
10/31/2020

Understanding Pre-trained BERT for Aspect-based Sentiment Analysis

This paper analyzes the pre-trained hidden representations learned from ...
research
04/19/2018

Learning Disentangled Representations of Texts with Application to Biomedical Abstracts

We propose a method for learning disentangled sets of vector representat...
research
09/26/2021

Be More Active! Understanding the Differences between Mean and Sampled Representations of Variational Autoencoders

The ability of Variational Autoencoders to learn disentangled representa...
research
12/22/2017

Disentangled Representations for Manipulation of Sentiment in Text

The ability to change arbitrary aspects of a text while leaving the core...

Please sign up or login with your details

Forgot password? Click here to reset