Language Models Are Poor Learners of Directional Inference

10/10/2022
by   Tianyi Li, et al.
0

We examine LMs' competence of directional predicate entailments by supervised fine-tuning with prompts. Our analysis shows that contrary to their apparent success on standard NLI, LMs show limited ability to learn such directional inference; moreover, existing datasets fail to test directionality, and/or are infested by artefacts that can be learnt as proxy for entailments, yielding over-optimistic results. In response, we present BoOQA (Boolean Open QA), a robust multi-lingual evaluation benchmark for directional predicate entailments, extrinsic to existing training sets. On BoOQA, we establish baselines and show evidence of existing LM-prompting models being incompetent directional entailment learners, in contrast to entailment graphs, however limited by sparsity.

READ FULL TEXT

page 3

page 12

research
04/16/2021

Multivalent Entailment Graphs for Question Answering

Drawing inferences between open-domain natural language predicates is a ...
research
07/30/2022

Smoothing Entailment Graphs with Language Models

The diversity and Zipfian frequency distribution of natural language pre...
research
08/03/2022

Efficient Fine-Tuning of Compressed Language Models with Learners

Fine-tuning BERT-based models is resource-intensive in memory, computati...
research
05/23/2018

Scoring Lexical Entailment with a Supervised Directional Similarity Network

We present the Supervised Directional Similarity Network (SDSN), a novel...
research
06/07/2023

From the One, Judge of the Whole: Typed Entailment Graph Construction with Predicate Generation

Entailment Graphs (EGs) have been constructed based on extracted corpora...
research
04/29/2021

Entailment as Few-Shot Learner

Large pre-trained language models (LMs) have demonstrated remarkable abi...
research
08/12/2018

A New Look at F-Tests

Directional inference for vector parameters based on higher order approx...

Please sign up or login with your details

Forgot password? Click here to reset