Convolution is outer product

by   Jean-Marc Andreoli, et al.

The inner product operation between tensors is the corner stone of deep neural network architectures, directly inherited from linear algebra. There is a striking contrast between the unicity of this basic construct and the extreme diversity of high level constructs which have been invented to address various application domains. This paper is interested in an intermediate construct, convolution, and its corollary, attention, which have become ubiquitous in many applications, but are still presented in an ad-hoc fashion depending on the application context. We first identify the common problem addressed by most existing forms of convolution, and show how the solution to that problem naturally involves another very generic operation of linear algebra, the outer product between tensors. We then proceed to show that attention is a form of convolution, called "content based" convolution, hence amenable to the generic formulation based on the outer product. The reader looking for yet another architecture yielding better performance results on a specific task is in for some disappointment. The reader aiming at a better, more grounded understanding of familiar concepts may find food for thought.


page 1

page 2

page 3

page 4


Tensor-Tensor Product Toolbox

Tensors are higher-order extensions of matrices. In recent work [Kilmer ...

Rank-1 Convolutional Neural Network

In this paper, we propose a convolutional neural network(CNN) with 3-D r...

Molding CNNs for text: non-linear, non-consecutive convolutions

The success of deep learning often derives from well-chosen operational ...

Spatial Autocorrelation Equation Based on Moran's Index

Based on standardized vector and globally normalized weight matrix, Mora...

Decoupled Networks

Inner product-based convolution has been a central component of convolut...

Implicit Filter-and-sum Network for Multi-channel Speech Separation

Various neural network architectures have been proposed in recent years ...

GALA: Greedy ComputAtion for Linear Algebra in Privacy-Preserved Neural Networks

Machine Learning as a Service (MLaaS) is enabling a wide range of smart ...

Please sign up or login with your details

Forgot password? Click here to reset