DeepAI AI Chat
Log In Sign Up

On the Dimensionality of Word Embedding

12/11/2018
by   Zi Yin, et al.
Microsoft
0

In this paper, we provide a theoretical understanding of word embedding and its dimensionality. Motivated by the unitary-invariance of word embedding, we propose the Pairwise Inner Product (PIP) loss, a novel metric on the dissimilarity between word embeddings. Using techniques from matrix perturbation theory, we reveal a fundamental bias-variance trade-off in dimensionality selection for word embeddings. This bias-variance trade-off sheds light on many empirical observations which were previously unexplained, for example the existence of an optimal dimensionality. Moreover, new insights and discoveries, like when and how word embeddings are robust to over-fitting, are revealed. By optimizing over the bias-variance trade-off of the PIP loss, we can explicitly answer the open question of dimensionality selection for word embedding.

READ FULL TEXT

page 1

page 2

page 3

page 4

03/01/2018

Pairwise Inner Product Distance: Metric for Functionality, Stability, Dimensionality of Vector Embedding

In this paper, we present a theoretical framework for understanding vect...
05/13/2023

Frequency-aware Dimension Selection for Static Word Embedding by Mixed Product Distance

Static word embedding is still useful, particularly for context-unavaila...
03/01/2018

PIP Distance: A Unitary-invariant Metric for Understanding Functionality and Dimensionality of Vector Embeddings

In this paper, we present a theoretical framework for understanding vect...
10/27/2022

MorphTE: Injecting Morphology in Tensorized Embeddings

In the era of deep learning, word embeddings are essential when dealing ...
04/27/2022

Extremal GloVe: Theoretically Accurate Distributed Word Embedding by Tail Inference

Distributed word embeddings such as Word2Vec and GloVe have been widely ...
06/19/2019

Considerations for the Interpretation of Bias Measures of Word Embeddings

Word embedding spaces are powerful tools for capturing latent semantic r...
06/15/2023

A Bayesian approach to uncertainty in word embedding bias estimation

Multiple measures, such as WEAT or MAC, attempt to quantify the magnitud...