Retrofitting Contextualized Word Embeddings with Paraphrases

09/12/2019
by   Weijia Shi, et al.
0

Contextualized word embedding models, such as ELMo, generate meaningful representations of words and their context. These models have been shown to have a great impact on downstream applications. However, in many cases, the contextualized embedding of a word changes drastically when the context is paraphrased. As a result, the downstream model is not robust to paraphrasing and other linguistic variations. To enhance the stability of contextualized word embedding models, we propose an approach to retrofitting contextualized embedding models with paraphrase contexts. Our method learns an orthogonal transformation on the input space, which seeks to minimize the variance of word representations on paraphrased contexts. Experiments show that the retrofitted model significantly outperforms the original ELMo on various sentence classification and language inference tasks.

READ FULL TEXT
research
05/12/2016

On the Convergent Properties of Word Embedding Methods

Do word embeddings converge to learn similar things over different initi...
research
01/08/2019

Deconstructing Word Embeddings

A review of Word Embedding Models through a deconstructive approach reve...
research
04/05/2017

Linear Ensembles of Word Embedding Models

This paper explores linear methods for combining several word embedding ...
research
05/06/2021

Learning to Perturb Word Embeddings for Out-of-distribution QA

QA models based on pretrained language mod-els have achieved remarkable ...
research
06/06/2020

Quantum-like Generalization of Complex Word Embedding: a lightweight approach for textual classification

In this paper, we present an extension, and an evaluation, to existing Q...
research
04/09/2019

Characterizing the impact of geometric properties of word embeddings on task performance

Analysis of word embedding properties to inform their use in downstream ...
research
09/30/2018

Zero-training Sentence Embedding via Orthogonal Basis

We propose a simple and robust training-free approach for building sente...

Please sign up or login with your details

Forgot password? Click here to reset