On the Limits of Minimal Pairs in Contrastive Evaluation

09/15/2021
by   Jannis Vamvas, et al.
0

Minimal sentence pairs are frequently used to analyze the behavior of language models. It is often assumed that model behavior on contrastive pairs is predictive of model behavior at large. We argue that two conditions are necessary for this assumption to hold: First, a tested hypothesis should be well-motivated, since experiments show that contrastive evaluation can lead to false positives. Secondly, test data should be chosen such as to minimize distributional discrepancy between evaluation time and deployment time. For a good approximation of deployment-time decoding, we recommend that minimal pairs are created based on machine-generated text, as opposed to human-written references. We present a contrastive evaluation suite for English-German MT that implements this recommendation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/18/2021

SimCSE: Simple Contrastive Learning of Sentence Embeddings

This paper presents SimCSE, a simple contrastive learning framework that...
research
11/24/2022

Contrastive pretraining for semantic segmentation is robust to noisy positive pairs

Domain-specific variants of contrastive learning can construct positive ...
research
09/05/2023

CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning

Nowadays, the research on Large Vision-Language Models (LVLMs) has been ...
research
05/12/2023

Surfacing Biases in Large Language Models using Contrastive Input Decoding

Ensuring that large language models (LMs) are fair, robust and useful re...
research
07/07/2023

AdaptiveRec: Adaptively Construct Pairs for Contrastive Learning in Sequential Recommendation

This paper presents a solution to the challenges faced by contrastive le...
research
05/09/2022

CoCoA-MT: A Dataset and Benchmark for Contrastive Controlled MT with Application to Formality

The machine translation (MT) task is typically formulated as that of ret...
research
07/13/2022

Policy Optimization with Sparse Global Contrastive Explanations

We develop a Reinforcement Learning (RL) framework for improving an exis...

Please sign up or login with your details

Forgot password? Click here to reset