Probabilistic symmetry and invariant neural networks

01/18/2019
by   Benjamin Bloem-Reddy, et al.
10

In an effort to improve the performance of deep neural networks in data-scarce, non-i.i.d., or unsupervised settings, much recent research has been devoted to encoding invariance under symmetry transformations into neural network architectures. We treat the neural network input and output as random variables, and consider group invariance from the perspective of probabilistic symmetry. Drawing on tools from probability and statistics, we establish a link between functional and probabilistic symmetry, and obtain generative functional representations of joint and conditional probability distributions that are invariant or equivariant under the action of a compact group. Those representations completely characterize the structure of neural networks that can be used to model such distributions and yield a general program for constructing invariant stochastic or deterministic neural networks. We develop the details of the general program for exchangeable sequences and arrays, recovering a number of recent examples as special cases.

READ FULL TEXT
06/07/2021

Encoding Involutory Invariance in Neural Networks

In certain situations, Neural Networks (NN) are trained upon data that o...
09/14/2021

Nonlinearities in Steerable SO(2)-Equivariant CNNs

Invariance under symmetry is an important problem in machine learning. O...
03/30/2020

Detecting Symmetries with Neural Networks

Identifying symmetries in data sets is generally difficult, but knowledg...
02/27/2017

Equivariance Through Parameter-Sharing

We propose to study equivariance in deep neural networks through paramet...
05/12/2019

Rotation Invariant Householder Parameterization for Bayesian PCA

We consider probabilistic PCA and related factor models from a Bayesian ...
02/08/2019

Invariant-equivariant representation learning for multi-class data

Representations learnt through deep neural networks tend to be highly in...
06/01/2021

Symmetry-via-Duality: Invariant Neural Network Densities from Parameter-Space Correlators

Parameter-space and function-space provide two different duality frames ...