Logical Tasks for Measuring Extrapolation and Rule Comprehension

11/14/2022
by   Ippei Fujisawa, et al.
11

Logical reasoning is essential in a variety of human activities. A representative example of a logical task is mathematics. Recent large-scale models trained on large datasets have been successful in various fields, but their reasoning ability in arithmetic tasks is limited, which we reproduce experimentally. Here, we recast this limitation as not unique to mathematics but common to tasks that require logical operations. We then propose a new set of tasks, termed logical tasks, which will be the next challenge to address. This higher point of view helps the development of inductive biases that have broad impact beyond the solution of individual tasks. We define and characterize logical tasks and discuss system requirements for their solution. Furthermore, we discuss the relevance of logical tasks to concepts such as extrapolation, explainability, and inductive bias. Finally, we provide directions for solving logical tasks.

READ FULL TEXT

page 11

page 12

page 14

page 24

page 25

page 26

research
06/16/2023

Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation From Deductive, Inductive and Abductive Views

Large Language Models (LLMs) have achieved great success in various natu...
research
07/14/2021

Proceedings of the Sixteenth Workshop on Logical Frameworks and Meta-Languages: Theory and Practice

Logical frameworks and meta-languages form a common substrate for repres...
research
06/08/2020

Mathematical Reasoning via Self-supervised Skip-tree Training

We examine whether self-supervised language modeling applied to mathemat...
research
02/11/2020

ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning

Recent powerful pre-trained language models have achieved remarkable per...
research
01/15/2021

LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning

While designing inductive bias in neural architectures has been widely s...
research
02/11/2022

End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking

Machine learning systems perform well on pattern matching tasks, but the...

Please sign up or login with your details

Forgot password? Click here to reset