Improved Generalization Bound of Permutation Invariant Deep Neural Networks

10/15/2019
by   Akiyoshi Sannai, et al.
0

We theoretically prove that a permutation invariant property of deep neural networks largely improves its generalization performance. Learning problems with data that are invariant to permutations are frequently observed in various applications, for example, point cloud data and graph neural networks. Numerous methodologies have been developed and they achieve great performances, however, understanding a mechanism of the performance is still a developing problem. In this paper, we derive a theoretical generalization bound for invariant deep neural networks with a ReLU activation to clarify their mechanism. Consequently, our bound shows that the main term of their generalization gap is improved by √(n!) where n is a number of permuting coordinates of data. Moreover, we prove that an approximation power of invariant deep neural networks can achieve an optimal rate, though the networks are restricted to be invariant. To achieve the results, we develop several new proof techniques such as correspondence with a fundamental domain and a scale-sensitive metric entropy.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/05/2019

Universal approximations of permutation invariant/equivariant functions by deep neural networks

In this paper,we develop a theory of the relationship between permutatio...
research
10/27/2022

On the Approximation and Complexity of Deep Neural Networks to Invariant Functions

Recent years have witnessed a hot wave of deep neural networks in variou...
research
06/24/2020

Distribution-Based Invariant Deep Networks for Learning Meta-Features

Recent advances in deep learning from probability distributions enable t...
research
02/23/2021

ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks

Recently, learning algorithms motivated from sharpness of loss surface a...
research
04/26/2021

Invariant polynomials and machine learning

We present an application of invariant polynomials in machine learning. ...
research
03/06/2019

Positively Scale-Invariant Flatness of ReLU Neural Networks

It was empirically confirmed by Keskar et al.SharpMinima that flatter mi...
research
05/27/2022

Why Robust Generalization in Deep Learning is Difficult: Perspective of Expressive Power

It is well-known that modern neural networks are vulnerable to adversari...

Please sign up or login with your details

Forgot password? Click here to reset