DeepAI AI Chat
Log In Sign Up

Text Classification with Lexicon from PreAttention Mechanism

by   Qingbiao Li, et al.

A comprehensive and high-quality lexicon plays a crucial role in traditional text classification approaches. And it improves the utilization of the linguistic knowledge. Although it is helpful for the task, the lexicon has got little attention in recent neural network models. Firstly, getting a high-quality lexicon is not easy. We lack an effective automated lexicon extraction method, and most lexicons are hand crafted, which is very inefficient for big data. What's more, there is no an effective way to use a lexicon in a neural network. To address those limitations, we propose a Pre-Attention mechanism for text classification in this paper, which can learn attention of different words according to their effects in the classification tasks. The words with different attention can form a domain lexicon. Experiments on three benchmark text classification tasks show that our models get competitive result comparing with the state-of-the-art methods. We get 90.5 dataset, 93.7 model without Pre-Attention mechanism, those with Pre-Attention mechanism improve by 0.9 mechanism. In addition, the Pre-Attention mechanism performs well followed by different types of neural networks (e.g., convolutional neural networks and Long Short-Term Memory networks). For the same dataset, when we use Pre-Attention mechanism to get attention value followed by different neural networks, those words with high attention values have a high degree of coincidence, which proves the versatility and portability of the Pre-Attention mechanism. we can get stable lexicons by attention values, which is an inspiring method of information extraction.


page 7

page 8


geoGAT: Graph Model Based on Attention Mechanism for Geographic Text Classification

In the area of geographic information processing. There are few research...

On the Dynamics of Training Attention Models

The attention mechanism has been widely used in deep neural networks as ...

Exploring Distantly-Labeled Rationales in Neural Network Models

Recent studies strive to incorporate various human rationales into neura...

An Iterative Contextualization Algorithm with Second-Order Attention

Combining the representations of the words that make up a sentence into ...

Aggregated Sparse Attention for Steering Angle Prediction

In this paper, we apply the attention mechanism to autonomous driving fo...

Attention Mechanism in Neural Networks: Where it Comes and Where it Goes

A long time ago in the machine learning literature, the idea of incorpor...

Pre-training Attention Mechanisms

Recurrent neural networks with differentiable attention mechanisms have ...