Learning Augmentation Distributions using Transformed Risk Minimization

11/16/2021
by   Evangelos Chatzipantazis, et al.
0

Adapting to the structure of data distributions (such as symmetry and transformation invariances) is an important challenge in machine learning. Invariances can be built into the learning process by architecture design, or by augmenting the dataset. Both require a priori knowledge about the exact nature of the symmetries. Absent this knowledge, practitioners resort to expensive and time-consuming tuning. To address this problem, we propose a new approach to learn distributions of augmentation transforms, in a new Transformed Risk Minimization (TRM) framework. In addition to predictive models, we also optimize over transformations chosen from a hypothesis space. As an algorithmic framework, our TRM method is (1) efficient (jointly learns augmentations and models in a single training loop), (2) modular (works with any training algorithm), and (3) general (handles both discrete and continuous augmentations). We theoretically compare TRM with standard risk minimization, and give a PAC-Bayes upper bound on its generalization error. We propose to optimize this bound over a rich augmentation space via a new parametrization over compositions of blocks, leading to the new Stochastic Compositional Augmentation Learning (SCALE) algorithm. We compare SCALE experimentally with prior methods (Fast AutoAugment and Augerino) on CIFAR10/100, SVHN . Additionally, we show that SCALE can correctly learn certain symmetries in the data distribution (recovering rotations on rotated MNIST) and can also improve calibration of the learned model.

READ FULL TEXT
research
06/25/2020

Learning Data Augmentation with Online Bilevel Optimization for Image Classification

Data augmentation is a key practice in machine learning for improving ge...
research
02/15/2022

A Theory of PAC Learnability under Transformation Invariances

Transformation invariances are present in many real-world problems. For ...
research
07/25/2019

Invariance reduces Variance: Understanding Data Augmentation in Deep Learning and Beyond

Many complex deep learning models have found success by exploiting symme...
research
02/27/2021

Incorporating Causal Graphical Prior Knowledge into Predictive Modeling via Simple Data Augmentation

Causal graphs (CGs) are compact representations of the knowledge of the ...
research
06/15/2023

Modularity Trumps Invariance for Compositional Robustness

By default neural networks are not robust to changes in data distributio...
research
02/05/2021

In-Loop Meta-Learning with Gradient-Alignment Reward

At the heart of the standard deep learning training loop is a greedy gra...
research
06/30/2021

Which Echo Chamber? Regions of Attraction in Learning with Decision-Dependent Distributions

As data-driven methods are deployed in real-world settings, the processe...

Please sign up or login with your details

Forgot password? Click here to reset