Scalable Neural Tangent Kernel of Recurrent Architectures

12/09/2020
by   Sina Alemohammad, et al.
0

Kernels derived from deep neural networks (DNNs) in the infinite-width provide not only high performance in a range of machine learning tasks but also new theoretical insights into DNN training dynamics and generalization. In this paper, we extend the family of kernels associated with recurrent neural networks (RNNs), which were previously derived only for simple RNNs, to more complex architectures that are bidirectional RNNs and RNNs with average pooling. We also develop a fast GPU implementation to exploit its full practical potential. While RNNs are typically only applied to time-series data, we demonstrate that classifiers using RNN-based kernels outperform a range of baseline methods on 90 non-time-series datasets from the UCI data repository.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

07/16/2020

A fast noise filtering algorithm for time series prediction using recurrent neural networks

Recent research demonstrate that prediction of time series by recurrent ...
06/18/2020

The Recurrent Neural Tangent Kernel

The study of deep networks (DNs) in the infinite-width limit, via the so...
10/14/2021

How to train RNNs on chaotic data?

Recurrent neural networks (RNNs) are wide-spread machine learning tools ...
04/07/2015

Bidirectional Recurrent Neural Networks as Generative Models - Reconstructing Gaps in Time Series

Bidirectional recurrent neural networks (RNN) are trained to predict bot...
06/04/2019

Towards Task and Architecture-Independent Generalization Gap Predictors

Can we use deep learning to predict when deep learning works? Our result...
07/09/2019

Click-Through Rate Prediction with the User Memory Network

Click-through rate (CTR) prediction is a critical task in online adverti...
06/22/2020

Understanding Recurrent Neural Networks Using Nonequilibrium Response Theory

Recurrent neural networks (RNNs) are brain-inspired models widely used i...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.