Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of Source Embeddings

04/26/2022
by   Danushka Bollegala, et al.
0

Given multiple source word embeddings learnt using diverse algorithms and lexical resources, meta word embedding learning methods attempt to learn more accurate and wide-coverage word embeddings. Prior work on meta-embedding has repeatedly discovered that simple vector concatenation of the source embeddings to be a competitive baseline. However, it remains unclear as to why and when simple vector concatenation can produce accurate meta-embeddings. We show that weighted concatenation can be seen as a spectrum matching operation between each source embedding and the meta-embedding, minimising the pairwise inner-product loss. Following this theoretical analysis, we propose two unsupervised methods to learn the optimal concatenation weights for creating meta-embeddings from a given set of source embeddings. Experimental results on multiple benchmark datasets show that the proposed weighted concatenated meta-embedding methods outperform previously proposed meta-embedding learning methods.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/14/2018

Frustratingly Easy Meta-Embedding -- Computing Meta-Embeddings by Averaging Source Word Embeddings

Creating accurate meta-embeddings from pre-trained source embeddings has...
research
09/19/2017

Think Globally, Embed Locally --- Locally Linear Meta-embedding of Words

Distributed word embeddings have shown superior performances in numerous...
research
05/19/2022

Gender Bias in Meta-Embeddings

Combining multiple source embeddings to create meta-embeddings is consid...
research
04/25/2022

A Survey on Word Meta-Embedding Learning

Meta-embedding (ME) learning is an emerging approach that attempts to le...
research
11/05/2020

Learning Efficient Task-Specific Meta-Embeddings with Word Prisms

Word embeddings are trained to predict word cooccurrence statistics, whi...
research
04/16/2022

Unsupervised Attention-based Sentence-Level Meta-Embeddings from Contextualised Language Models

A variety of contextualised language models have been proposed in the NL...
research
08/13/2018

Angular-Based Word Meta-Embedding Learning

Ensembling word embeddings to improve distributed word representations h...

Please sign up or login with your details

Forgot password? Click here to reset