Injecting Numerical Reasoning Skills into Language Models

04/09/2020
by   Mor Geva, et al.
0

Large pre-trained language models (LMs) are known to encode substantial amounts of linguistic information. However, high-level reasoning skills, such as numerical reasoning, are difficult to learn from a language-modeling objective only. Consequently, existing models for numerical reasoning have used specialized architectures with limited flexibility. In this work, we show that numerical reasoning is amenable to automatic data generation, and thus one can inject this skill into pre-trained LMs, by generating large amounts of data, and training in a multi-task setup. We show that pre-training our model, GenBERT, on this data, dramatically improves performance on DROP (49.3 → 72.3 F1), reaching performance that matches state-of-the-art models of comparable size, while using a simple and general-purpose encoder-decoder architecture. Moreover, GenBERT generalizes well to math word problem datasets, while maintaining high performance on standard RC tasks. Our approach provides a general recipe for injecting skills into large pre-trained LMs, whenever the skill is amenable to automatic data augmentation.

READ FULL TEXT
07/15/2021

Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills

Models pre-trained with a language modeling objective possess ample worl...
04/15/2021

NT5?! Training T5 to Perform Numerical Reasoning

Numerical reasoning over text (NRoT) presents unique challenges that are...
05/18/2022

LogiGAN: Learning Logical Reasoning via Adversarial Pre-training

We present LogiGAN, an unsupervised adversarial pre-training framework f...
01/27/2022

Reasoning Like Program Executors

Reasoning over natural language is a long-standing goal for the research...
06/11/2020

Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge

To what extent can a neural network systematically reason over symbolic ...
01/31/2019

Riconoscimento ortografico per apostrofo ed espressioni polirematiche

The work presents two algorithms of manipulation and comparison between ...
05/05/2022

Assistive Recipe Editing through Critiquing

There has recently been growing interest in the automatic generation of ...

Code Repositories

injecting_numeracy

The accompanying code for "Injecting Numerical Reasoning Skills into Language Models" (Mor Geva*, Ankit Gupta* and Jonathan Berant, ACL 2020).


view repo