End-to-End Neuro-Symbolic Architecture for Image-to-Image Reasoning Tasks

06/06/2021
by   Ananye Agarwal, et al.
0

Neural models and symbolic algorithms have recently been combined for tasks requiring both perception and reasoning. Neural models ground perceptual input into a conceptual vocabulary, on which a classical reasoning algorithm is applied to generate output. A key limitation is that such neural-to-symbolic models can only be trained end-to-end for tasks where the output space is symbolic. In this paper, we study neural-symbolic-neural models for reasoning tasks that require a conversion from an image input (e.g., a partially filled sudoku) to an image output (e.g., the image of the completed sudoku). While designing such a three-step hybrid architecture may be straightforward, the key technical challenge is end-to-end training – how to backpropagate without intermediate supervision through the symbolic component. We propose NSNnet, an architecture that combines an image reconstruction loss with a novel output encoder to generate a supervisory signal, develops update algorithms that leverage policy gradient methods for supervision, and optimizes loss using a novel subsampling heuristic. We experiment on problem settings where symbolic algorithms are easily specified: a visual maze solving task and a visual Sudoku solver where the supervision is in image form. Experiments show high accuracy with significantly less data compared to purely neural approaches.

READ FULL TEXT

page 2

page 14

page 15

research
09/19/2022

Knowledge-based Analogical Reasoning in Neuro-symbolic Latent Spaces

Analogical Reasoning problems challenge both connectionist and symbolic ...
research
10/17/2022

A Solver-Free Framework for Scalable Learning in Neural ILP Architectures

There is a recent focus on designing architectures that have an Integer ...
research
02/15/2023

Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?

Compositionality is a pivotal property of symbolic reasoning. However, h...
research
11/19/2021

Solving Visual Analogies Using Neural Algorithmic Reasoning

We consider a class of visual analogical reasoning problems that involve...
research
05/24/2019

Designing a Symbolic Intermediate Representation for Neural Surface Realization

Generated output from neural NLG systems often contain errors such as ha...
research
10/26/2022

Generalization Differences between End-to-End and Neuro-Symbolic Vision-Language Reasoning Systems

For vision-and-language reasoning tasks, both fully connectionist, end-t...
research
06/24/2020

Understanding Deep Architectures with Reasoning Layer

Recently, there has been a surge of interest in combining deep learning ...

Please sign up or login with your details

Forgot password? Click here to reset