Lie-Access Neural Turing Machines

11/09/2016
by   Greg Yang, et al.
0

External neural memory structures have recently become a popular tool for algorithmic deep learning (Graves et al. 2014, Weston et al. 2014). These models generally utilize differentiable versions of traditional discrete memory-access structures (random access, stacks, tapes) to provide the storage necessary for computational tasks. In this work, we argue that these neural memory systems lack specific structure important for relative indexing, and propose an alternative model, Lie-access memory, that is explicitly designed for the neural setting. In this paradigm, memory is accessed using a continuous head in a key-space manifold. The head is moved via Lie group actions, such as shifts or rotations, generated by a controller, and memory access is performed by linear smoothing in key space. We argue that Lie groups provide a natural generalization of discrete memory structures, such as Turing machines, as they provide inverse and identity operators while maintaining differentiability. To experiment with this approach, we implement a simplified Lie-access neural Turing machine (LANTM) with different Lie groups. We find that this approach is able to perform well on a range of algorithmic tasks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/28/2016

Lie Access Neural Turing Machine

Following the recent trend in explicit neural memory structures, we pres...
research
10/14/2015

Structured Memory for Neural Turing Machines

Neural Turing Machines (NTM) contain memory component that simulates "wo...
research
12/07/2016

Neural Turing Machines: Convergence of Copy Tasks

The architecture of neural Turing machines is differentiable end to end ...
research
11/19/2015

Neural Random-Access Machines

In this paper, we propose and investigate a new neural network architect...
research
05/04/2015

Reinforcement Learning Neural Turing Machines - Revised

The Neural Turing Machine (NTM) is more expressive than all previously c...
research
01/10/2019

On the Turing Completeness of Modern Neural Network Architectures

Alternatives to recurrent neural networks, in particular, architectures ...
research
01/27/2020

Variational Optimization on Lie Groups, with Examples of Leading (Generalized) Eigenvalue Problems

The article considers smooth optimization of functions on Lie groups. By...

Please sign up or login with your details

Forgot password? Click here to reset