DeepAI AI Chat
Log In Sign Up

Multi-Scale Contrastive Co-Training for Event Temporal Relation Extraction

by   Hao-Ren Yao, et al.
National Institutes of Health
Carnegie Mellon University
Allen Institute for Artificial Intelligence

Extracting temporal relationships between pairs of events in texts is a crucial yet challenging problem for natural language understanding. Depending on the distance between the events, models must learn to differently balance information from local and global contexts surrounding the event pair for temporal relation prediction. Learning how to fuse this information has proved challenging for transformer-based language models. Therefore, we present MulCo: Multi-Scale Contrastive Co-Training, a technique for the better fusion of local and global contextualized features. Our model uses a BERT-based language model to encode local context and a Graph Neural Network (GNN) to represent global document-level syntactic and temporal characteristics. Unlike previous state-of-the-art methods, which use simple concatenation on multi-view features or select optimal sentences using sophisticated reinforcement learning approaches, our model co-trains GNN and BERT modules using a multi-scale contrastive learning objective. The GNN and BERT modules learn a synergistic parameterization by contrasting GNN multi-layer multi-hop subgraphs (i.e., global context embeddings) and BERT outputs (i.e., local context embeddings) through end-to-end back-propagation. We empirically demonstrate that MulCo provides improved ability to fuse local and global contexts encoded using BERT and GNN compared to the current state-of-the-art. Our experimental results show that MulCo achieves new state-of-the-art results on several temporal relation extraction datasets.


Extracting Temporal Event Relation with Syntactic-Guided Temporal Graph Transformer

Extracting temporal relations (e.g., before, after, concurrent) among ev...

Improving BERT Model Using Contrastive Learning for Biomedical Relation Extraction

Contrastive learning has been used to learn a high-quality representatio...

Extracting Event Temporal Relations via Hyperbolic Geometry

Detecting events and their evolution through time is a crucial task in n...

Imposing Relation Structure in Language-Model Embeddings Using Contrastive Learning

Though language model text embeddings have revolutionized NLP research, ...

Contextualized Word Embeddings Enhanced Event Temporal Relation Extraction for Story Understanding

Learning causal and temporal relationships between events is an importan...

MUSE: Multi-Scale Temporal Features Evolution for Knowledge Tracing

Transformer based knowledge tracing model is an extensively studied prob...

GNN-LM: Language Modeling based on Global Contexts via GNN

Inspired by the notion that “to copy is easier than to memorize“, in thi...