batchboost: regularization for stabilizing training with resistance to underfitting overfitting

01/21/2020
by   Maciej A. Czyzewski, et al.
0

Overfitting underfitting and stable training are an important challenges in machine learning. Current approaches for these issues are mixup, SamplePairing and BC learning. In our work, we state the hypothesis that mixing many images together can be more effective than just two. Batchboost pipeline has three stages: (a) pairing: method of selecting two samples. (b) mixing: how to create a new one from two samples. (c) feeding: combining mixed samples with new ones from dataset into batch (with ratio γ). Note that sample that appears in our batch propagates with subsequent iterations with less and less importance until the end of training. Pairing stage calculates the error per sample, sorts the samples and pairs with strategy: hardest with easiest one, than mixing stage merges two samples using mixup, x_1 + (1-λ)x_2. Finally, feeding stage combines new samples with mixed by ratio 1:1. Batchboost has 0.5-3 better accuracy than the current state-of-the-art mixup regularization on CIFAR-10 Fashion-MNIST. Our method is slightly better than SamplePairing technique on small datasets (up to 5 not tuned parameters (like weight decay), thus its a good method to test performance of different architectures. Source code is at: https://github.com/maciejczyzewski/batchboost

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/14/2020

PatchUp: A Regularization Technique for Convolutional Neural Networks

Large capacity deep learning models are often prone to a high generaliza...
research
05/30/2016

Stochastic Function Norm Regularization of Deep Networks

Deep neural networks have had an enormous impact on image analysis. Stat...
research
04/12/2020

A Non-Parametric Test to Detect Data-Copying in Generative Models

Detecting overfitting in generative models is an important challenge in ...
research
08/21/2022

A Unified Analysis of Mixed Sample Data Augmentation: A Loss Function Perspective

We propose the first unified theoretical analysis of mixed sample data a...
research
11/17/2021

A label efficient two-sample test

Two-sample tests evaluate whether two samples are realizations of the sa...
research
11/24/2020

InstaHide's Sample Complexity When Mixing Two Private Images

Inspired by InstaHide challenge [Huang, Song, Li and Arora'20], [Chen, S...
research
06/04/2020

Robust Sampling in Deep Learning

Deep learning requires regularization mechanisms to reduce overfitting a...

Please sign up or login with your details

Forgot password? Click here to reset