Hierarchical Memory Networks

by   Sarath Chandar, et al.

Memory networks are neural networks with an explicit memory component that can be both read and written to by the network. The memory is often addressed in a soft way using a softmax function, making end-to-end training with backpropagation possible. However, this is not computationally scalable for applications which require the network to read from extremely large memories. On the other hand, it is well known that hard attention mechanisms based on reinforcement learning are challenging to train successfully. In this paper, we explore a form of hierarchical memory network, which can be considered as a hybrid between hard and soft attention memory networks. The memory is organized in a hierarchical structure such that reading from it is done with less computation than soft attention over a flat memory, while also being easier to train than hard attention over a flat memory. Specifically, we propose to incorporate Maximum Inner Product Search (MIPS) in the training and inference procedures for our hierarchical memory network. We explore the use of various state-of-the art approximate MIPS techniques and report results on SimpleQuestions, a challenging large scale factoid question answering task.


page 1

page 2

page 3

page 4


Question Generation from Paragraphs: A Tale of Two Hierarchical Models

Automatic question generation from paragraphs is an important and challe...

Metalearned Neural Memory

We augment recurrent neural networks with an external memory mechanism t...

Learning Visual Question Answering by Bootstrapping Hard Attention

Attention mechanisms in biological perception are thought to select subs...

Out-of-core Training for Extremely Large-Scale Neural Networks With Adaptive Window-Based Scheduling

While large neural networks demonstrate higher performance in various ta...

Why can't memory networks read effectively?

Memory networks have been a popular choice among neural architectures fo...

End-To-End Memory Networks

We introduce a neural network with a recurrent attention model over a po...

Near-Optimal Glimpse Sequences for Improved Hard Attention Neural Network Training

We introduce the use of Bayesian optimal experimental design techniques ...

Code Repositories


This is the code&dataset for the paper [Hierarchical Memory Networks for Answer Selection on Unknown Words. COLING 2016]

view repo


This project collects the different accepted papers and their link to Arxiv or Gitxiv

view repo

Please sign up or login with your details

Forgot password? Click here to reset