Learning Contextual Causality from Time-consecutive Images

by   Hongming Zhang, et al.

Causality knowledge is crucial for many artificial intelligence systems. Conventional textual-based causality knowledge acquisition methods typically require laborious and expensive human annotations. As a result, their scale is often limited. Moreover, as no context is provided during the annotation, the resulting causality knowledge records (e.g., ConceptNet) typically do not take the context into consideration. To explore a more scalable way of acquiring causality knowledge, in this paper, we jump out of the textual domain and investigate the possibility of learning contextual causality from the visual signal. Compared with pure text-based approaches, learning causality from the visual signal has the following advantages: (1) Causality knowledge belongs to the commonsense knowledge, which is rarely expressed in the text but rich in videos; (2) Most events in the video are naturally time-ordered, which provides a rich resource for us to mine causality knowledge from; (3) All the objects in the video can be used as context to study the contextual property of causal relations. In detail, we first propose a high-quality dataset Vis-Causal and then conduct experiments to demonstrate that with good language and visual representation models as well as enough training signals, it is possible to automatically discover meaningful causal knowledge from the videos. Further analysis also shows that the contextual property of causal relations indeed exists, taking which into consideration might be crucial if we want to use the causality knowledge in real applications, and the visual signal could serve as a good resource for learning such contextual causality.


page 2

page 4

page 9

page 13


iReason: Multimodal Commonsense Reasoning using Videos and Natural Language with Interpretability

Causality knowledge is vital to building robust AI systems. Deep learnin...

Towards Causality Extraction from Requirements

System behavior is often based on causal relations between certain event...

Domain Adaptative Causality Encoder

Current approaches which are mainly based on the extraction of low-level...

TC-GAT: Graph Attention Network for Temporal Causality Discovery

The present study explores the intricacies of causal relationship extrac...

Causality Extraction based on Self-Attentive BiLSTM-CRF with Transferred Embeddings

Causality extraction from natural language texts is a challenging open p...

An overview of the quantitative causality analysis and causal graph reconstruction based on a rigorous formalism of information flow

Inference of causal relations from data now has become an important fiel...

Illusion of Causality in Visualized Data

Students who eat breakfast more frequently tend to have a higher grade p...

Please sign up or login with your details

Forgot password? Click here to reset