Obtaining Better Static Word Embeddings Using Contextual Embedding Models

06/08/2021
by   Prakhar Gupta, et al.
11

The advent of contextual word embeddings – representations of words which incorporate semantic and syntactic information from their context – has led to tremendous improvements on a wide variety of NLP tasks. However, recent contextual models have prohibitively high computational cost in many use-cases and are often hard to interpret. In this work, we demonstrate that our proposed distillation method, which is a simple extension of CBOW-based training, allows to significantly improve computational efficiency of NLP applications, while outperforming the quality of existing static embeddings trained from scratch as well as those distilled from previously proposed methods. As a side-effect, our approach also allows a fair comparison of both contextual and static embeddings via standard lexical evaluation tasks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/10/2019

Better Word Embeddings by Disentangling Contextual n-Gram Information

Pre-trained word vectors are ubiquitous in Natural Language Processing a...
research
09/05/2023

Substitution-based Semantic Change Detection using Contextual Embeddings

Measuring semantic change has thus far remained a task where methods usi...
research
06/02/2023

Word Embeddings for Banking Industry

Applications of Natural Language Processing (NLP) are plentiful, from se...
research
08/27/2018

Dissecting Contextual Word Embeddings: Architecture and Representation

Contextual word representations derived from pre-trained bidirectional l...
research
06/06/2021

Combining Static Word Embeddings and Contextual Representations for Bilingual Lexicon Induction

Bilingual Lexicon Induction (BLI) aims to map words in one language to t...
research
05/04/2020

Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding words

Although models using contextual word embeddings have achieved state-of-...
research
09/18/2021

Augmenting semantic lexicons using word embeddings and transfer learning

Sentiment-aware intelligent systems are essential to a wide array of app...

Please sign up or login with your details

Forgot password? Click here to reset