Supervised level-wise pretraining for recurrent neural network initialization in multi-class classification

11/04/2019
by   Dino Ienco, et al.
0

Recurrent Neural Networks (RNNs) can be seriously impacted by the initial parameters assignment, which may result in poor generalization performances on new unseen data. With the objective to tackle this crucial issue, in the context of RNN based classification, we propose a new supervised layer-wise pretraining strategy to initialize network parameters. The proposed approach leverages a data-aware strategy that sets up a taxonomy of classification problems automatically derived by the model behavior. To the best of our knowledge, despite the great interest in RNN-based classification, this is the first data-aware strategy dealing with the initialization of such models. The proposed strategy has been tested on four benchmarks coming from two different domains, i.e., Speech Recognition and Remote Sensing. Results underline the significance of our approach and point out that data-aware strategies positively support the initialization of Recurrent Neural Network based classification models.

READ FULL TEXT
research
06/18/2020

The Recurrent Neural Tangent Kernel

The study of deep networks (DNs) in the infinite-width limit, via the so...
research
05/20/2018

Multi-Step Prediction of Dynamic Systems with Recurrent Neural Networks

Recurrent Neural Networks (RNNs) can encode rich dynamics which makes th...
research
05/01/2020

Exploring Pre-training with Alignments for RNN Transducer based End-to-End Speech Recognition

Recently, the recurrent neural network transducer (RNN-T) architecture h...
research
11/12/2015

Improving performance of recurrent neural network with relu nonlinearity

In recent years significant progress has been made in successfully train...
research
10/10/2017

Network of Recurrent Neural Networks

We describe a class of systems theory based neural networks called "Netw...
research
09/20/2021

iRNN: Integer-only Recurrent Neural Network

Recurrent neural networks (RNN) are used in many real-world text and spe...
research
09/30/2022

Sparse tree-based initialization for neural networks

Dedicated neural network (NN) architectures have been designed to handle...

Please sign up or login with your details

Forgot password? Click here to reset