RobustLR: Evaluating Robustness to Logical Perturbation in Deductive Reasoning

05/25/2022
by   Soumya Sanyal, et al.
0

Transformers have been shown to be able to perform deductive reasoning on a logical rulebase containing rules and statements written in English natural language. While the progress is promising, it is currently unclear if these models indeed perform logical reasoning by understanding the underlying logical semantics in the language. To this end, we propose RobustLR, a suite of evaluation datasets that evaluate the robustness of these models to minimal logical edits in rulebases and some standard logical equivalence conditions. In our experiments with RoBERTa and T5, we find that the models trained in prior works do not perform consistently on the different perturbations in RobustLR, thus showing that the models are not robust to the proposed logical perturbations. Further, we find that the models find it especially hard to learn logical negation and disjunction operators. Overall, using our evaluation sets, we demonstrate some shortcomings of the deductive reasoning-based language models, which can eventually help towards designing better models for logical reasoning over natural language.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/19/2022

FaiRR: Faithful and Robust Deductive Reasoning over Natural Language

Transformers have been shown to be able to perform deductive reasoning o...
research
05/22/2023

Teaching Probabilistic Logical Reasoning to Transformers

Recent research on transformer-based language models investigates their ...
research
06/06/2014

Recursive Neural Networks Can Learn Logical Semantics

Tree-structured recursive neural networks (TreeRNNs) for sentence meanin...
research
10/22/2022

MetaLogic: Logical Reasoning Explanations with Fine-Grained Structure

In this paper, we propose a comprehensive benchmark to investigate model...
research
04/18/2021

Flexible Operations for Natural Language Deduction

An interpretable system for complex, open-domain reasoning needs an inte...
research
04/07/2023

Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4

Harnessing logical reasoning ability is a comprehensive natural language...
research
08/18/2023

How susceptible are LLMs to Logical Fallacies?

This paper investigates the rational thinking capability of Large Langua...

Please sign up or login with your details

Forgot password? Click here to reset