A Functional Perspective on Learning Symmetric Functions with Neural Networks

by   Aaron Zweig, et al.

Symmetric functions, which take as input an unordered, fixed-size set, are known to be universally representable by neural networks that enforce permutation invariance. However, these architectures only give guarantees for fixed input sizes, yet in many practical scenarios, such as particle physics, a relevant notion of generalization should include varying the input size. In this paper, we embed symmetric functions (of any size) as functions over probability measures, and study the ability of neural networks defined over this space of measures to represent and learn in that space. By focusing on shallow architectures, we establish approximation and generalization bounds under different choices of regularization (such as RKHS and variation norms), that capture a hierarchy of functional spaces with increasing amount of non-linear learning. The resulting models can be learnt efficiently and enjoy generalization guarantees that extend across input sizes, as we verify empirically.


page 1

page 2

page 3

page 4


Exponential Separations in Symmetric Neural Networks

In this work we demonstrate a novel separation between symmetric neural ...

On Symmetry and Initialization for Neural Networks

This work provides an additional step in the theoretical understanding o...

Neural networks with linear threshold activations: structure and algorithms

In this article we present new results on neural networks with linear th...

Effect of Various Regularizers on Model Complexities of Neural Networks in Presence of Input Noise

Deep neural networks are over-parameterized, which implies that the numb...

Abelian Neural Networks

We study the problem of modeling a binary operation that satisfies some ...

Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measures

This paper considers the Pointer Value Retrieval (PVR) benchmark introdu...

Stochastic Deep Networks

Machine learning is increasingly targeting areas where input data cannot...