Learning Invariances using the Marginal Likelihood

08/16/2018
by   Mark van der Wilk, et al.
0

Generalising well in supervised learning tasks relies on correctly extrapolating the training data to a large region of the input space. One way to achieve this is to constrain the predictions to be invariant to transformations on the input that are known to be irrelevant (e.g. translation). Commonly, this is done through data augmentation, where the training set is enlarged by applying hand-crafted transformations to the inputs. We argue that invariances should instead be incorporated in the model structure, and learned using the marginal likelihood, which correctly rewards the reduced complexity of invariant models. We demonstrate this for Gaussian process models, due to the ease with which their marginal likelihood can be estimated. Our main contribution is a variational inference scheme for Gaussian processes containing invariances described by a sampling procedure. We learn the sampling procedure by back-propagating through it to maximise the marginal likelihood.

READ FULL TEXT
research
06/14/2021

Last Layer Marginal Likelihood for Invariance Learning

Data augmentation is often used to incorporate inductive biases into mod...
research
06/07/2023

Improving Hyperparameter Learning under Approximate Inference in Gaussian Process Models

Approximate inference in Gaussian process (GP) models with non-conjugate...
research
02/25/2022

Learning Invariant Weights in Neural Networks

Assumptions about invariances or symmetries in data can significantly in...
research
11/11/2022

Towards Improved Learning in Gaussian Processes: The Best of Two Worlds

Gaussian process training decomposes into inference of the (approximate)...
research
09/06/2017

Convolutional Gaussian Processes

We present a practical way of introducing convolutional structure into G...
research
02/21/2020

Knot Selection in Sparse Gaussian Processes

Knot-based, sparse Gaussian processes have enjoyed considerable success ...
research
07/13/2019

Learning Complex Basis Functions for Invariant Representations of Audio

Learning features from data has shown to be more successful than using h...

Please sign up or login with your details

Forgot password? Click here to reset