Separation Results between Fixed-Kernel and Feature-Learning Probability Metrics

06/10/2021

∙

Several works in implicit and explicit generative modeling empirically observed that feature-learning discriminators outperform fixed-kernel discriminators in terms of the sample quality of the models. We provide separation results between probability metrics with fixed-kernel and feature-learning discriminators using the function classes ℱ_2 and ℱ_1 respectively, which were developed to study overparametrized two-layer neural networks. In particular, we construct pairs of distributions over hyper-spheres that can not be discriminated by fixed kernel (ℱ_2) integral probability metric (IPM) and Stein discrepancy (SD) in high dimensions, but that can be discriminated by their feature learning (ℱ_1) counterparts. To further study the separation we provide links between the ℱ_1 and ℱ_2 IPMs with sliced Wasserstein distances. Our work suggests that fixed-kernel discriminators perform worse than their feature learning counterparts because their corresponding metrics are weaker.

READ FULL TEXT

Separation Results between Fixed-Kernel and Feature-Learning Probability Metrics

Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networks

A Theory of Feature Learning

Feature learning in neural networks and kernel machines that recursively learn features

Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks

Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks

Graph Neural Networks Provably Benefit from Structural Information: A Feature Learning Perspective

Depth and Feature Learning are Provably Beneficial for Neural Network Discriminators

Separation Results between Fixed-Kernel and Feature-Learning Probability Metrics

Related Research

Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networks

A Theory of Feature Learning

Feature learning in neural networks and kernel machines that recursively learn features

Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks

Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks

Graph Neural Networks Provably Benefit from Structural Information: A Feature Learning Perspective

Depth and Feature Learning are Provably Beneficial for Neural Network Discriminators