Modeling Local Dependence in Natural Language with Multi-channel Recurrent Neural Networks

11/13/2018
by   Chang Xu, et al.
0

Recurrent Neural Networks (RNNs) have been widely used in processing natural language tasks and achieve huge success. Traditional RNNs usually treat each token in a sentence uniformly and equally. However, this may miss the rich semantic structure information of a sentence, which is useful for understanding natural languages. Since semantic structures such as word dependence patterns are not parameterized, it is a challenge to capture and leverage structure information. In this paper, we propose an improved variant of RNN, Multi-Channel RNN (MC-RNN), to dynamically capture and leverage local semantic structure information. Concretely, MC-RNN contains multiple channels, each of which represents a local dependence pattern at a time. An attention mechanism is introduced to combine these patterns at each step, according to the semantic information. Then we parameterize structure information by adaptively selecting the most appropriate connection structures among channels. In this way, diverse local structures and dependence patterns in sentences can be well captured by MC-RNN. To verify the effectiveness of MC-RNN, we conduct extensive experiments on typical natural language processing tasks, including neural machine translation, abstractive summarization, and language modeling. Experimental results on these tasks all show significant improvements of MC-RNN over current top systems.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/09/2018

Recurrent Neural Network-Based Semantic Variational Autoencoder for Sequence-to-Sequence Learning

Sequence-to-sequence (Seq2seq) models have played an import role in the ...
research
01/06/2016

Recurrent Memory Networks for Language Modeling

Recurrent Neural Networks (RNN) have obtained excellent result in many n...
research
11/14/2019

Contextual Recurrent Units for Cloze-style Reading Comprehension

Recurrent Neural Networks (RNN) are known as powerful models for handlin...
research
09/05/2021

Learning Hierarchical Structures with Differentiable Nondeterministic Stacks

Learning hierarchical structures in sequential data – from simple algori...
research
08/05/2018

LISA: Explaining Recurrent Neural Network Judgments via Layer-wIse Semantic Accumulation and Example to Pattern Transformation

Recurrent neural networks (RNNs) are temporal networks and cumulative in...
research
09/12/2016

Knowledge as a Teacher: Knowledge-Guided Structural Attention Networks

Natural language understanding (NLU) is a core component of a spoken dia...
research
12/21/2015

Backward and Forward Language Modeling for Constrained Sentence Generation

Recent language models, especially those based on recurrent neural netwo...

Please sign up or login with your details

Forgot password? Click here to reset