Understanding Recurrent Neural State Using Memory Signatures

02/11/2018
by   Skanda Koppula, et al.
0

We demonstrate a network visualization technique to analyze the recurrent state inside the LSTMs/GRUs used commonly in language and acoustic models. Interpreting intermediate state and network activations inside end-to-end models remains an open challenge. Our method allows users to understand exactly how much and what history is encoded inside recurrent state in grapheme sequence models. Our procedure trains multiple decoders that predict prior input history. Compiling results from these decoders, a user can obtain a signature of the recurrent kernel that characterizes its memory behavior. We demonstrate this method's usefulness in revealing information divergence in the bases of recurrent factorized kernels, visualizing the character-level differences between the memory of n-gram and recurrent language models, and extracting knowledge of history encoded in the layers of grapheme-based end-to-end ASR networks.

READ FULL TEXT

page 2

page 3

page 4

research
09/13/2016

Character-Level Language Modeling with Hierarchical Recurrent Neural Networks

Recurrent neural network (RNN) based character-level language models (CL...
research
07/23/2020

Deep Learning based, end-to-end metaphor detection in Greek language with Recurrent and Convolutional Neural Networks

This paper presents and benchmarks a number of end-to-end Deep Learning ...
research
01/13/2017

End-to-End ASR-free Keyword Search from Speech

End-to-end (E2E) systems have achieved competitive results compared to c...
research
12/14/2016

Incorporating Language Level Information into Acoustic Models

This paper proposed a class of novel Deep Recurrent Neural Networks whic...
research
12/31/2021

Training and Generating Neural Networks in Compressed Weight Space

The inputs and/or outputs of some neural nets are weight matrices of oth...
research
04/05/2018

A Large-Scale Study of Language Models for Chord Prediction

We conduct a large-scale study of language models for chord prediction. ...
research
05/25/2017

Deriving Neural Architectures from Sequence and Graph Kernels

The design of neural architectures for structured objects is typically g...

Please sign up or login with your details

Forgot password? Click here to reset