Time-Stamped Language Model: Teaching Language Models to Understand the Flow of Events

04/15/2021
by   Hossein Rajaby Faghihi, et al.
0

Tracking entities throughout a procedure described in a text is challenging due to the dynamic nature of the world described in the process. Firstly, we propose to formulate this task as a question answering problem. This enables us to use pre-trained transformer-based language models on other QA benchmarks by adapting those to the procedural text understanding. Secondly, since the transformer-based language models cannot encode the flow of events by themselves, we propose a Time-Stamped Language Model (TSLM model) to encode event information in LMs architecture by introducing the timestamp encoding. Our model evaluated on the Propara dataset shows improvements on the published state-of-the-art results with a 3.1% increase in F1 score. Moreover, our model yields better results on the location prediction task on the NPN-Cooking dataset. This result indicates that our approach is effective for procedural text understanding in general.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 4

10/07/2021

A Comparative Study of Transformer-Based Language Models on Extractive Question Answering

Question Answering (QA) is a task in natural language processing that ha...
10/25/2019

SpeechBERT: Cross-Modal Pre-trained Language Model for End-to-end Spoken Question Answering

While end-to-end models for spoken language understanding tasks have bee...
10/24/2020

Temporal Reasoning on Implicit Events from Distant Supervision

Existing works on temporal reasoning among events described in text focu...
08/19/2019

Question Answering based Clinical Text Structuring Using Pre-trained Language Model

Clinical text structuring is a critical and fundamental task for clinica...
04/20/2021

Modeling Event Plausibility with Consistent Conceptual Abstraction

Understanding natural language requires common sense, one aspect of whic...
12/15/2021

Simple Text Detoxification by Identifying a Linear Toxic Subspace in Language Model Embeddings

Large pre-trained language models are often trained on large volumes of ...
04/16/2021

proScript: Partially Ordered Scripts Generation via Pre-trained Language Models

Scripts - standardized event sequences describing typical everyday activ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.