APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning

12/19/2022
by   Soumya Sanyal, et al.
0

Logical reasoning of text is an important ability that requires understanding the information present in the text, their interconnections, and then reasoning through them to infer new conclusions. Prior works on improving the logical reasoning ability of language models require complex processing of training data (e.g., aligning symbolic knowledge to text), yielding task-specific data augmentation solutions that restrict the learning of general logical reasoning skills. In this work, we propose APOLLO, an adaptively pretrained language model that has improved logical reasoning abilities. We select a subset of Wikipedia, based on a set of logical inference keywords, for continued pretraining of a language model. We use two self-supervised loss functions: a modified masked language modeling loss where only specific parts-of-speech words, that would likely require more reasoning than basic language understanding, are masked, and a sentence-level classification loss that teaches the model to distinguish between entailment and contradiction types of sentences. The proposed training paradigm is both simple and independent of task formats. We demonstrate the effectiveness of APOLLO by comparing it with prior baselines on two logical reasoning datasets. APOLLO performs comparably on ReClor and outperforms baselines on LogiQA.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/28/2022

Logical Fallacy Detection

Reasoning is central to human intelligence. However, fallacious argument...
research
03/01/2022

MERIt: Meta-Path Guided Contrastive Learning for Logical Reasoning

Logical reasoning is of vital importance to natural language understandi...
research
06/08/2020

Mathematical Reasoning via Self-supervised Skip-tree Training

We examine whether self-supervised language modeling applied to mathemat...
research
12/11/2019

Just Add Functions: A Neural-Symbolic Language Model

Neural network language models (NNLMs) have achieved ever-improving accu...
research
08/16/2023

Separate the Wheat from the Chaff: Model Deficiency Unlearning via Parameter-Efficient Module Operation

Large language models (LLMs) have been widely used in various applicatio...
research
05/21/2023

Contrastive Learning with Logic-driven Data Augmentation for Logical Reasoning over Text

Pre-trained large language model (LLM) is under exploration to perform N...
research
04/06/2020

Multi-Step Inference for Reasoning Over Paragraphs

Complex reasoning over text requires understanding and chaining together...

Please sign up or login with your details

Forgot password? Click here to reset