The Mechanism of Additive Composition

11/26/2015
by   Ran Tian, et al.
0

Additive composition (Foltz et al, 1998; Landauer and Dumais, 1997; Mitchell and Lapata, 2010) is a widely used method for computing meanings of phrases, which takes the average of vector representations of the constituent words. In this article, we prove an upper bound for the bias of additive composition, which is the first theoretical analysis on compositional frameworks from a machine learning point of view. The bound is written in terms of collocation strength; we prove that the more exclusively two successive words tend to occur together, the more accurate one can guarantee their additive composition as an approximation to the natural phrase vector. Our proof relies on properties of natural language data that are empirically verified, and can be theoretically derived from an assumption that the data is generated from a Hierarchical Pitman-Yor Process. The theory endorses additive composition as a reasonable operation for calculating meanings of phrases, and suggests ways to improve additive compositionality, including: transforming entries of distributional word vectors by a function that meets a specific condition, constructing a novel type of vector representations to make additive composition sensitive to word order, and utilizing singular value decomposition to train word vectors.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/11/2019

No Word is an Island -- A Transformation Weighting Model for Semantic Composition

Composition models of distributional semantics are used to construct phr...
research
06/18/2015

"The Sum of Its Parts": Joint Learning of Word and Phrase Representations with Autoencoders

Recently, there has been a lot of effort to represent words in continuou...
research
06/08/2016

Learning Semantically and Additively Compositional Distributional Representations

This paper connects a vector-based composition model to a formal semanti...
research
06/11/2019

A Systematic Comparison of English Noun Compound Representations

Building meaningful representations of noun compounds is not trivial sin...
research
01/04/2017

Joint Semantic Synthesis and Morphological Analysis of the Derived Word

Much like sentences are composed of words, words themselves are composed...
research
08/26/2020

Machine learning approach of Japanese composition scoring and writing aided system's design

Automatic scoring system is extremely complex for any language. Because ...
research
07/23/2017

Composing Distributed Representations of Relational Patterns

Learning distributed representations for relation instances is a central...

Please sign up or login with your details

Forgot password? Click here to reset