DeepAI AI Chat
Log In Sign Up

HyperENTM: Evolving Scalable Neural Turing Machines through HyperNEAT

10/12/2017
by   Jakob Merrild, et al.
IT University of Copenhagen
0

Recent developments within memory-augmented neural networks have solved sequential problems requiring long-term memory, which are intractable for traditional neural networks. However, current approaches still struggle to scale to large memory sizes and sequence lengths. In this paper we show how access to memory can be encoded geometrically through a HyperNEAT-based Neural Turing Machine (HyperENTM). We demonstrate that using the indirect HyperNEAT encoding allows for training on small memory vectors in a bit-vector copy task and then applying the knowledge gained from such training to speed up training on larger size memory vectors. Additionally, we demonstrate that in some instances, networks trained to copy bit-vectors of size 9 can be scaled to sizes of 1,000 without further training. While the task in this paper is simple, these results could open up the problems amendable to networks with external memories to problems with larger memory vectors and theoretically unbounded memory sizes.

READ FULL TEXT
12/07/2016

Neural Turing Machines: Convergence of Copy Tasks

The architecture of neural Turing machines is differentiable end to end ...
07/23/2018

Implementing Neural Turing Machines

Neural Turing Machines (NTMs) are an instance of Memory Augmented Neural...
10/14/2015

Structured Memory for Neural Turing Machines

Neural Turing Machines (NTM) contain memory component that simulates "wo...
02/14/2017

Survey of reasoning using Neural networks

Reason and inference require process as well as memory skills by humans....
10/27/2020

Out-of-core Training for Extremely Large-Scale Neural Networks With Adaptive Window-Based Scheduling

While large neural networks demonstrate higher performance in various ta...
10/05/2020

Robust High-dimensional Memory-augmented Neural Networks

Traditional neural networks require enormous amounts of data to build th...
06/30/2016

Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes

We extend neural Turing machine (NTM) model into a dynamic neural Turing...