DeepAI AI Chat
Log In Sign Up

Efficient N-Dimensional Convolutions via Higher-Order Factorization

by   Jean Kossaifi, et al.

With the unprecedented success of deep convolutional neural networks came the quest for training always deeper networks. However, while deeper neural networks give better performance when trained appropriately, that depth also translates in memory and computation heavy models, typically with tens of millions of parameters. Several methods have been proposed to leverage redundancies in the network to alleviate this complexity. Either a pretrained network is compressed, e.g. using a low-rank tensor decomposition, or the architecture of the network is directly modified to be more effective. In this paper, we study both approaches in a unified framework, under the lens of tensor decompositions. We show how tensor decomposition applied to the convolutional kernel relates to efficient architectures such as MobileNet. Moreover, we propose a tensor-based method for efficient higher order convolutions, which can be used as a plugin replacement for N-dimensional convolutions. We demonstrate their advantageous properties both theoretically and empirically for image classification, for both 2D and 3D convolutional networks.


page 5

page 6


HOTCAKE: Higher Order Tucker Articulated Kernels for Deeper CNN Compression

The emerging edge computing has promoted immense interests in compacting...

Data-driven emergence of convolutional structure in neural networks

Exploiting data invariances is crucial for efficient learning in both ar...

End-to-end Learning of a Convolutional Neural Network via Deep Tensor Decomposition

In this paper we study the problem of learning the weights of a deep con...

Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNs

Graph neural networks that model 3D data, such as point clouds or atoms,...

Defensive Tensorization

We propose defensive tensorization, an adversarial defence technique tha...

Convolutional Learning on Simplicial Complexes

We propose a simplicial complex convolutional neural network (SCCNN) to ...