Neuro-Symbolic Spatio-Temporal Reasoning

by   Jae Hee Lee, et al.

Knowledge about space and time is necessary to solve problems in the physical world: An AI agent situated in the physical world and interacting with objects often needs to reason about positions of and relations between objects; and as soon as the agent plans its actions to solve a task, it needs to consider the temporal aspect (e.g., what actions to perform over time). Spatio-temporal knowledge, however, is required beyond interacting with the physical world, and is also often transferred to the abstract world of concepts through analogies and metaphors (e.g., "a threat that is hanging over our heads"). As spatial and temporal reasoning is ubiquitous, different attempts have been made to integrate this into AI systems. In the area of knowledge representation, spatial and temporal reasoning has been largely limited to modeling objects and relations and developing reasoning methods to verify statements about objects and relations. On the other hand, neural network researchers have tried to teach models to learn spatial relations from data with limited reasoning capabilities. Bridging the gap between these two approaches in a mutually beneficial way could allow us to tackle many complex real-world problems, such as natural language processing, visual question answering, and semantic image segmentation. In this chapter, we view this integration problem from the perspective of Neuro-Symbolic AI. Specifically, we propose a synergy between logical reasoning and machine learning that will be grounded on spatial and temporal knowledge. Describing some successful applications, remaining challenges, and evaluation datasets pertaining to this direction is the main topic of this contribution.


Answer Set Programming Modulo `Space-Time'

We present ASP Modulo `Space-Time', a declarative representational and c...

Keyword-Aware Relative Spatio-Temporal Graph Networks for Video Question Answering

The main challenge in video question answering (VideoQA) is to capture a...

Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution

Spatial-temporal reasoning is a challenging task in Artificial Intellige...

Semantic Vector Spaces for Broadening Consideration of Consequences

Reasoning systems with too simple a model of the world and human intent ...

Learning Perceptual Inference by Contrasting

"Thinking in pictures," [1] i.e., spatial-temporal reasoning, effortless...

Curiosity-driven Intuitive Physics Learning

Biological infants are naturally curious and try to comprehend their phy...

A Constraint Satisfaction Framework for Executing Perceptions and Actions in Diagrammatic Reasoning

Diagrammatic reasoning (DR) is pervasive in human problem solving as a p...

Please sign up or login with your details

Forgot password? Click here to reset