Probabilistic FastText for Multi-Sense Word Embeddings

06/07/2018
by   Ben Athiwaratkun, et al.
0

We introduce Probabilistic FastText, a new model for word embeddings that can capture multiple word senses, sub-word structure, and uncertainty information. In particular, we represent each word with a Gaussian mixture density, where the mean of a mixture component is given by the sum of n-grams. This representation allows the model to share statistical strength across sub-word structures (e.g. Latin roots), producing accurate representations of rare, misspelt, or even unseen words. Moreover, each component of the mixture can capture a different word sense. Probabilistic FastText outperforms both FastText, which has no probabilistic model, and dictionary-level probabilistic embeddings, which do not incorporate subword structures, on several word-similarity benchmarks, including English RareWord and foreign language datasets. We also achieve state-of-art performance on benchmarks that measure ability to discern different meanings. Thus, the proposed model is the first to achieve multi-sense representations while having enriched semantics on rare words.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/15/2017

A Mixture Model for Learning Multi-Sense Word Embeddings

Word embeddings are now a standard technique for inducing meaning repres...
research
11/19/2015

Gaussian Mixture Embeddings for Multiple Word Prototypes

Recently, word representation has been increasingly focused on for its e...
research
11/12/2019

Learning Multi-Sense Word Distributions using Approximate Kullback-Leibler Divergence

Learning word representations has garnered greater attention in the rece...
research
01/21/2021

Multi-sense embeddings through a word sense disambiguation process

Natural Language Understanding has seen an increasing number of publicat...
research
09/12/2018

Generalizing Word Embeddings using Bag of Subwords

We approach the problem of generalizing pre-trained word embeddings beyo...
research
06/24/2016

A Game-Theoretic Approach to Word Sense Disambiguation

This paper presents a new model for word sense disambiguation formulated...
research
09/20/2021

BERT Has Uncommon Sense: Similarity Ranking for Word Sense BERTology

An important question concerning contextualized word embedding (CWE) mod...

Please sign up or login with your details

Forgot password? Click here to reset