Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms

by   Dinghan Shen, et al.
Duke University

Many deep learning architectures have been proposed to model the compositionality in text sequences, requiring a substantial number of parameters and expensive computations. However, there has not been a rigorous evaluation regarding the added value of sophisticated compositional functions. In this paper, we conduct a point-by-point comparative study between Simple Word-Embedding-based Models (SWEMs), consisting of parameter-free pooling operations, relative to word-embedding-based RNN/CNN models. Surprisingly, SWEMs exhibit comparable or even superior performance in the majority of cases considered. Based upon this understanding, we propose two additional pooling strategies over learned word embeddings: (i) a max-pooling operation for improved interpretability; and (ii) a hierarchical pooling operation, which preserves spatial (n-gram) information within text sequences. We present experiments on 17 datasets encompassing three tasks: (i) (long) document classification; (ii) text sequence matching; and (iii) short text tasks, including classification and tagging. The source code and datasets can be obtained from https://


page 1

page 2

page 3

page 4


Word Embeddings for the Armenian Language: Intrinsic and Extrinsic Evaluation

In this work, we intrinsically and extrinsically evaluate and compare ex...

An Investigation on Word Embedding Offset Clustering as Relationship Classification

Vector representations obtained from word embedding are the source of ma...

MorphTE: Injecting Morphology in Tensorized Embeddings

In the era of deep learning, word embeddings are essential when dealing ...

A Literature Study of Embeddings on Source Code

Natural language processing has improved tremendously after the success ...

Segmentation-free compositional n-gram embedding

Applying conventional word embedding models to unsegmented languages, wh...

Revisiting Additive Compositionality: AND, OR and NOT Operations with Word Embeddings

It is well-known that typical word embedding methods such as Word2Vec an...

Deep Learning Architecture for Automatic Essay Scoring

Automatic evaluation of essay (AES) and also called automatic essay scor...

Please sign up or login with your details

Forgot password? Click here to reset