Pretrained Language Models for Sequential Sentence Classification

09/09/2019
by   Arman Cohan, et al.
0

As a step toward better document-level understanding, we explore classification of a sequence of sentences into their corresponding categories, a task that requires understanding sentences in context of the document. Recent successful models for this task have used hierarchical models to contextualize sentence representations, and Conditional Random Fields (CRFs) to incorporate dependencies between subsequent labels. In this work, we show that pretrained language models, BERT (Devlin et al., 2018) in particular, can be used for this task to capture contextual dependencies without the need for hierarchical encoding nor a CRF. Specifically, we construct a joint sentence representation that allows BERT Transformer layers to directly utilize contextual information from all words in all sentences. Our approach achieves state-of-the-art results on four datasets, including a new dataset of structured scientific abstracts.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/19/2018

Hierarchical Neural Networks for Sequential Sentence Classification in Medical Scientific Abstracts

Prevalent models based on artificial neural network (ANN) for sentence c...
research
03/16/2022

CUE Vectors: Modular Training of Language Models Conditioned on Diverse Contextual Signals

We propose a framework to modularize the training of neural language mod...
research
07/01/2021

Leveraging Domain Agnostic and Specific Knowledge for Acronym Disambiguation

An obstacle to scientific document understanding is the extensive use of...
research
12/15/2016

Neural Networks for Joint Sentence Classification in Medical Paper Abstracts

Existing models based on artificial neural networks (ANNs) for sentence ...
research
10/07/2021

Contextual Sentence Classification: Detecting Sustainability Initiatives in Company Reports

We introduce the novel task of detecting sustainability initiatives in c...
research
07/10/2019

Can Unconditional Language Models Recover Arbitrary Sentences?

Neural network-based generative language models like ELMo and BERT can w...
research
06/12/2021

A Sentence-level Hierarchical BERT Model for Document Classification with Limited Labelled Data

Training deep learning models with limited labelled data is an attractiv...

Please sign up or login with your details

Forgot password? Click here to reset