Unsupervised Distillation of Syntactic Information from Contextualized Word Representations

10/11/2020
by   Shauli Ravfogel, et al.
6

Contextualized word representations, such as ELMo and BERT, were shown to perform well on various semantic and syntactic tasks. In this work, we tackle the task of unsupervised disentanglement between semantics and structure in neural language representations: we aim to learn a transformation of the contextualized vectors, that discards the lexical semantics, but keeps the structural information. To this end, we automatically generate groups of sentences which are structurally similar but semantically different, and use metric-learning approach to learn a transformation that emphasizes the structural component that is encoded in the vectors. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics. Finally, we demonstrate the utility of our distilled representations by showing that they outperform the original contextualized representations in a few-shot parsing setting.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/29/2017

Frame-Based Continuous Lexical Semantics through Exponential Family Tensor Factorization and Semantic Proto-Roles

We study how different frame annotations complement one another when lea...
research
03/10/2014

Parsing using a grammar of word association vectors

This paper was was first drafted in 2001 as a formalization of the syste...
research
04/29/2020

Analysing Lexical Semantic Change with Contextualised Word Representations

This paper presents the first unsupervised approach to lexical semantic ...
research
07/06/2023

Agentività e telicità in GilBERTo: implicazioni cognitive

The goal of this study is to investigate whether a Transformer-based neu...
research
11/08/2019

How Language-Neutral is Multilingual BERT?

Multilingual BERT (mBERT) provides sentence representations for 104 lang...
research
11/30/2020

UWB @ DIACR-Ita: Lexical Semantic Change Detection with CCA and Orthogonal Transformation

In this paper, we describe our method for detection of lexical semantic ...
research
10/29/2017

Personalized word representations Carrying Personalized Semantics Learned from Social Network Posts

Distributed word representations have been shown to be very useful in va...

Please sign up or login with your details

Forgot password? Click here to reset