Mathematical Reasoning via Self-supervised Skip-tree Training

06/08/2020
by   Markus N. Rabe, et al.
1

We examine whether self-supervised language modeling applied to mathematical formulas enables logical reasoning. We suggest several logical reasoning tasks that can be used to evaluate language models trained on formal mathematical statements, such as type inference, suggesting missing assumptions and completing equalities. To train language models for formal mathematics, we propose a novel skip-tree task. We find that models trained on the skip-tree task show surprisingly strong mathematical reasoning abilities, and outperform models trained on standard skip-sequence tasks. We also analyze the models' ability to formulate new conjectures by measuring how often the predictions are provable and useful in other proofs.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/07/2021

Measuring and Improving BERT's Mathematical Abilities by Predicting the Order of Reasoning

Imagine you are in a supermarket. You have two bananas in your basket an...
research
12/19/2022

APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning

Logical reasoning of text is an important ability that requires understa...
research
06/06/2023

Certified Reasoning with Language Models

Language models often achieve higher accuracy when reasoning step-by-ste...
research
11/14/2022

Logical Tasks for Measuring Extrapolation and Rule Comprehension

Logical reasoning is essential in a variety of human activities. A repre...
research
02/18/2023

BERT is not The Count: Learning to Match Mathematical Statements with Proofs

We introduce a task consisting in matching a proof to a given mathematic...
research
08/16/2023

Separate the Wheat from the Chaff: Model Deficiency Unlearning via Parameter-Efficient Module Operation

Large language models (LLMs) have been widely used in various applicatio...
research
02/12/2020

The Space of Mathematical Software Systems – A Survey of Paradigmatic Systems

Mathematical software systems are becoming more and more important in pu...

Please sign up or login with your details

Forgot password? Click here to reset