Analogies Explained: Towards Understanding Word Embeddings

01/28/2019
by   Carl Allen, et al.
0

Word embeddings generated by neural network methods such as word2vec (W2V) are well known to exhibit seemingly linear behaviour, e.g. the embeddings of analogy "woman is to queen as man is to king" approximately describe a parallelogram. This property is particularly intriguing since the embeddings are not trained to achieve it. Several explanations have been proposed, but each introduces assumptions that do not hold in practice. We derive a probabilistically grounded definition of paraphrasing and show it can be re-interpreted as word transformation, a mathematical description of "w_x is to w_y". From these concepts we prove existence of the linear relationship between W2V-type embeddings that underlies the analogical phenomenon, and identify explicit error terms in the relationship.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/04/2022

Word Tour: One-dimensional Word Embeddings via the Traveling Salesman Problem

Word embeddings are one of the most fundamental technologies used in nat...
research
02/26/2019

Context Vectors are Reflections of Word Vectors in Half the Dimensions

This paper takes a step towards theoretical analysis of the relationship...
research
11/05/2019

Incremental Sense Weight Training for the Interpretation of Contextualized Word Embeddings

We present a novel online algorithm that learns the essence of each dime...
research
05/27/2022

Semeval-2022 Task 1: CODWOE – Comparing Dictionaries and Word Embeddings

Word embeddings have advanced the state of the art in NLP across numerou...
research
04/17/2021

Embodying Pre-Trained Word Embeddings Through Robot Actions

We propose a promising neural network model with which to acquire a grou...
research
09/03/2015

Encoding Prior Knowledge with Eigenword Embeddings

Canonical correlation analysis (CCA) is a method for reducing the dimens...
research
03/09/2017

What can you do with a rock? Affordance extraction via word embeddings

Autonomous agents must often detect affordances: the set of behaviors en...

Please sign up or login with your details

Forgot password? Click here to reset