Evaluation of sentence embeddings in downstream and linguistic probing tasks

06/16/2018
by   Christian S. Perone, et al.
0

Despite the fast developmental pace of new sentence embedding methods, it is still challenging to find comprehensive evaluations of these different techniques. In the past years, we saw significant improvements in the field of sentence embeddings and especially towards the development of universal sentence encoders that could provide inductive transfer to a wide variety of downstream tasks. In this work, we perform a comprehensive evaluation of recent methods using a wide variety of downstream and linguistic feature probing tasks. We show that a simple approach using bag-of-words with a recently introduced language model for deep context-dependent word embeddings proved to yield better results in many tasks when compared to sentence encoders trained on entailment datasets. We also show, however, that we are still far away from a universal encoder that can perform consistently across several downstream tasks.

READ FULL TEXT

page 9

page 10

page 12

research
05/03/2018

What you can cram into a single vector: Probing sentence embeddings for linguistic properties

Although much effort has recently been devoted to training high-quality ...
research
03/05/2022

Just Rank: Rethinking Evaluation with Word and Sentence Similarities

Word and sentence embeddings are useful feature representations in natur...
research
09/07/2023

The Daunting Dilemma with Sentence Encoders: Success on Standard Benchmarks, Failure in Capturing Basic Semantic Properties

In this paper, we adopted a retrospective approach to examine and compar...
research
02/18/2019

CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model

Continuous Bag of Words (CBOW) is a powerful text embedding method. Due ...
research
08/29/2018

On Tree-Based Neural Sentence Modeling

Neural networks with tree-based sentence encoders have shown better resu...
research
08/25/2020

A simple method for domain adaptation of sentence embeddings

Pre-trained sentence embeddings have been shown to be very useful for a ...
research
06/09/2019

Encouraging Paragraph Embeddings to Remember Sentence Identity Improves Classification

While paragraph embedding models are remarkably effective for downstream...

Please sign up or login with your details

Forgot password? Click here to reset