Depth-Adaptive Graph Recurrent Network for Text Classification

02/29/2020
by   Yijin Liu, et al.
0

The Sentence-State LSTM (S-LSTM) is a powerful and high efficient graph recurrent network, which views words as nodes and performs layer-wise recurrent steps between them simultaneously. Despite its successes on text representations, the S-LSTM still suffers from two drawbacks. Firstly, given a sentence, certain words are usually more ambiguous than others, and thus more computation steps need to be taken for these difficult words and vice versa. However, the S-LSTM takes fixed computation steps for all words, irrespective of their hardness. The secondary one comes from the lack of sequential information (e.g., word order) that is inherently important for natural language. In this paper, we try to address these issues and propose a depth-adaptive mechanism for the S-LSTM, which allows the model to learn how many computational steps to conduct for different words as required. In addition, we integrate an extra RNN layer to inject sequential information, which also serves as an input feature for the decision of adaptive depths. Results on the classic text classification task (24 datasets in various sizes and domains) show that our model brings significant improvements against the conventional S-LSTM and other high-performance models (e.g., the Transformer), meanwhile achieving a good accuracy-speed trade off.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/27/2020

Explicitly Modeling Adaptive Depths for Transformer

The vanilla Transformer conducts a fixed number of computations over all...
research
05/07/2018

Sentence-State LSTM for Text Representation

Bi-directional LSTMs are a powerful tool for text representation. On the...
research
07/13/2020

Transformer with Depth-Wise LSTM

Increasing the depth of models allows neural models to model complicated...
research
01/30/2021

ShufText: A Simple Black Box Approach to Evaluate the Fragility of Text Classification Models

Text classification is the most basic natural language processing task. ...
research
03/20/2019

Neural Speed Reading with Structural-Jump-LSTM

Recurrent neural networks (RNNs) can model natural language by sequentia...
research
04/23/2017

Learning to Skim Text

Recurrent Neural Networks are showing much promise in many sub-areas of ...
research
09/26/2019

Pre-train, Interact, Fine-tune: A Novel Interaction Representation for Text Classification

Text representation can aid machines in understanding text. Previous wor...

Please sign up or login with your details

Forgot password? Click here to reset