Zero-shot Entailment of Leaderboards for Empirical AI Research

03/29/2023
by   Salomon Kabongo, et al.
0

We present a large-scale empirical investigation of the zero-shot learning phenomena in a specific recognizing textual entailment (RTE) task category, i.e. the automated mining of leaderboards for Empirical AI Research. The prior reported state-of-the-art models for leaderboards extraction formulated as an RTE task, in a non-zero-shot setting, are promising with above 90 performances. However, a central research question remains unexamined: did the models actually learn entailment? Thus, for the experiments in this paper, two prior reported state-of-the-art models are tested out-of-the-box for their ability to generalize or their capacity for entailment, given leaderboard labels that were unseen during training. We hypothesize that if the models learned entailment, their zero-shot performances can be expected to be moderately high as well–perhaps, concretely, better than chance. As a result of this work, a zero-shot labeled dataset is created via distant labeling formulating the leaderboard extraction RTE task.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/08/2021

Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extraction

Relation extraction systems require large amounts of labeled examples wh...
research
04/22/2022

Zero and Few-shot Learning for Author Profiling

Author profiling classifies author characteristics by analyzing how lang...
research
12/29/2022

GPT Takes the Bar Exam

Nearly all jurisdictions in the United States require a professional lic...
research
11/25/2021

Near-Zero-Shot Suggestion Mining with a Little Help from WordNet

In this work, we explore the constructive side of online reviews: advice...
research
03/25/2022

ZS4IE: A toolkit for Zero-Shot Information Extraction with simple Verbalizations

The current workflow for Information Extraction (IE) analysts involves t...
research
08/31/2019

Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach

Zero-shot text classification (0Shot-TC) is a challenging NLU problem to...
research
05/03/2022

Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learning

Recent work has shown that NLP tasks such as Relation Extraction (RE) ca...

Please sign up or login with your details

Forgot password? Click here to reset