More Is More – Narrowing the Generalization Gap by Adding Classification Heads

02/09/2021
by   Roee Cates, et al.
0

Overfit is a fundamental problem in machine learning in general, and in deep learning in particular. In order to reduce overfit and improve generalization in the classification of images, some employ invariance to a group of transformations, such as rotations and reflections. However, since not all objects exhibit necessarily the same invariance, it seems desirable to allow the network to learn the useful level of invariance from the data. To this end, motivated by self-supervision, we introduce an architecture enhancement for existing neural network models based on input transformations, termed 'TransNet', together with a training algorithm suitable for it. Our model can be employed during training time only and then pruned for prediction, resulting in an equivalent architecture to the base model. Thus pruned, we show that our model improves performance on various data-sets while exhibiting improved generalization, which is achieved in turn by enforcing soft invariance on the convolutional kernels of the last layer in the base model. Theoretical analysis is provided to support the proposed method.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/10/2021

Understanding the Generalization Benefit of Model Invariance from a Data Perspective

Machine learning models that are developed to be invariant under certain...
research
02/24/2017

How ConvNets model Non-linear Transformations

In this paper, we theoretically address three fundamental problems invol...
research
07/14/2022

On the Strong Correlation Between Model Invariance and Generalization

Generalization and invariance are two essential properties of any machin...
research
10/11/2017

Subsampling large graphs and invariance in networks

Specify a randomized algorithm that, given a very large graph or network...
research
11/28/2019

A Generalization Theory based on Independent and Task-Identically Distributed Assumption

Existing generalization theories analyze the generalization performance ...
research
11/13/2019

Learning Non-Parametric Invariances from Data with Permanent Random Connectomes

One of the fundamental problems in supervised classification and in mach...
research
07/24/2023

Unsupervised Learning of Invariance Transformations

The need for large amounts of training data in modern machine learning i...

Please sign up or login with your details

Forgot password? Click here to reset