Recurrent Neural Networks with Mixed Hierarchical Structures and EM Algorithm for Natural Language Processing

01/21/2022
by   Zhaoxin Luo, et al.
0

How to obtain hierarchical representations with an increasing level of abstraction becomes one of the key issues of learning with deep neural networks. A variety of RNN models have recently been proposed to incorporate both explicit and implicit hierarchical information in modeling languages in the literature. In this paper, we propose a novel approach called the latent indicator layer to identify and learn implicit hierarchical information (e.g., phrases), and further develop an EM algorithm to handle the latent indicator layer in training. The latent indicator layer further simplifies a text's hierarchical structure, which allows us to seamlessly integrate different levels of attention mechanisms into the structure. We called the resulting architecture as the EM-HRNN model. Furthermore, we develop two bootstrap strategies to effectively and efficiently train the EM-HRNN model on long text documents. Simulation studies and real data applications demonstrate that the EM-HRNN model with bootstrap training outperforms other RNN-based models in document classification tasks. The performance of the EM-HRNN model is comparable to a Transformer-based method called Bert-base, though the former is much smaller model and does not require pre-training.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/04/2021

Recurrent Neural Networks with Mixed Hierarchical Structures for Natural Language Processing

Hierarchical structures exist in both linguistics and Natural Language P...
research
04/26/2020

Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Document Matching

Many information retrieval and natural language processing problems can ...
research
12/29/2018

Weakly-Supervised Hierarchical Text Classification

Hierarchical text classification, which aims to classify text documents ...
research
04/19/2021

Few-shot Learning for Topic Modeling

Topic models have been successfully used for analyzing text documents. H...
research
10/16/2019

Injecting Hierarchy with U-Net Transformers

The Transformer architecture has become increasingly popular over the pa...
research
05/30/2019

Hierarchical Transformers for Multi-Document Summarization

In this paper, we develop a neural summarization model which can effecti...
research
01/18/2023

Discrete Latent Structure in Neural Networks

Many types of data from fields including natural language processing, co...

Please sign up or login with your details

Forgot password? Click here to reset