CKConv: Continuous Kernel Convolution For Sequential Data

02/04/2021
by   David W. Romero, et al.
0

Conventional neural architectures for sequential data present important limitations. Recurrent networks suffer from exploding and vanishing gradients, small effective memory horizons, and must be trained sequentially. Convolutional networks are unable to handle sequences of unknown size and their memory horizon must be defined a priori. In this work, we show that all these problems can be solved by formulating convolutional kernels in CNNs as continuous functions. The resulting Continuous Kernel Convolution (CKConv) allows us to model arbitrarily long sequences in a parallel manner, within a single operation, and without relying on any form of recurrence. We show that Continuous Kernel Convolutional Networks (CKCNNs) obtain state-of-the-art results in multiple datasets, e.g., permuted MNIST, and, thanks to their continuous nature, are able to handle non-uniformly sampled datasets and irregularly-sampled data natively. CKCNNs match or perform better than neural ODEs designed for these purposes in a much faster and simpler manner.

READ FULL TEXT
research
12/07/2019

Dynamic Convolution: Attention over Convolution Kernels

Light-weight convolutional neural networks (CNNs) suffer performance deg...
research
10/15/2021

FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes

When designing Convolutional Neural Networks (CNNs), one must select the...
research
01/29/2016

Kernels for sequentially ordered data

We present a novel framework for kernel learning with sequential data of...
research
04/05/2023

SMPConv: Self-moving Point Representations for Continuous Convolution

Continuous convolution has recently gained prominence due to its ability...
research
10/25/2021

Exploiting Redundancy: Separable Group Convolutional Networks on Lie Groups

Group convolutional neural networks (G-CNNs) have been shown to increase...
research
05/02/2023

Sequence Modeling with Multiresolution Convolutional Memory

Efficiently capturing the long-range patterns in sequential data sources...

Please sign up or login with your details

Forgot password? Click here to reset