A Comparison of Rule Extraction for Different Recurrent Neural Network Models and Grammatical Complexity

01/16/2018
by   Qinglong Wang, et al.
0

It has been shown that rules can be extracted from highly non-linear, recursive models such as recurrent neural networks (RNNs). The RNN models mostly investigated include both Elman networks and second-order recurrent networks. Recently, new types of RNNs have demonstrated superior power in handling many machine learning tasks, especially when structural data is involved such as language modeling. Here, we empirically evaluate different recurrent models on the task of learning deterministic finite automata (DFA), the seven Tomita grammars. We are interested in the capability of recurrent models with different architectures in learning and expressing regular grammars, which can be the building blocks for many applications dealing with structural data. Our experiments show that a second-order RNN provides the best and stablest performance of extracting DFA over all Tomita grammars and that other RNN models are greatly influenced by different Tomita grammars. To better understand these results, we provide a theoretical analysis of the "complexity" of different grammars, by introducing the entropy and the averaged edit distance of regular grammars defined in this paper. Through our analysis, we categorize all Tomita grammars into different classes, which explains the inconsistency in the performance of extraction observed across all RNN models.

READ FULL TEXT

page 4

page 15

page 18

research
01/20/2021

Synthesizing Context-free Grammars from Recurrent Neural Networks (Extended Version)

We present an algorithm for extracting a subclass of the context free gr...
research
11/12/2019

Connecting First and Second Order Recurrent Networks with Deterministic Finite Automata

We propose an approach that connects recurrent networks with different o...
research
01/25/2019

State-Regularized Recurrent Neural Networks

Recurrent neural networks are a widely used class of neural architecture...
research
03/21/2018

Jet Charge and Machine Learning

Modern machine learning techniques, such as convolutional, recurrent and...
research
07/04/2018

Connecting Weighted Automata and Recurrent Neural Networks through Spectral Learning

In this paper, we unravel a fundamental connection between weighted fini...
research
10/23/2020

State space models for building control: how deep should you go?

Power consumption in buildings show non-linear behaviors that linear mod...
research
08/04/2023

Universal Approximation of Linear Time-Invariant (LTI) Systems through RNNs: Power of Randomness in Reservoir Computing

Recurrent neural networks (RNNs) are known to be universal approximators...

Please sign up or login with your details

Forgot password? Click here to reset