Separation Results between Fixed-Kernel and Feature-Learning Probability Metrics

by   Carles Domingo Enrich, et al.

Several works in implicit and explicit generative modeling empirically observed that feature-learning discriminators outperform fixed-kernel discriminators in terms of the sample quality of the models. We provide separation results between probability metrics with fixed-kernel and feature-learning discriminators using the function classes ℱ_2 and ℱ_1 respectively, which were developed to study overparametrized two-layer neural networks. In particular, we construct pairs of distributions over hyper-spheres that can not be discriminated by fixed kernel (ℱ_2) integral probability metric (IPM) and Stein discrepancy (SD) in high dimensions, but that can be discriminated by their feature learning (ℱ_1) counterparts. To further study the separation we provide links between the ℱ_1 and ℱ_2 IPMs with sliced Wasserstein distances. Our work suggests that fixed-kernel discriminators perform worse than their feature learning counterparts because their corresponding metrics are weaker.



There are no comments yet.


page 1

page 2

page 3

page 4


A Theory of Feature Learning

Feature Learning aims to extract relevant information contained in data ...

Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks

We analyze the learning dynamics of infinitely wide neural networks with...

Feature Learning in Infinite-Width Neural Networks

As its width tends to infinity, a deep neural network's behavior under g...

Rapid Feature Evolution Accelerates Learning in Neural Networks

Neural network (NN) training and generalization in the infinite-width li...

Depth and Feature Learning are Provably Beneficial for Neural Network Discriminators

We construct pairs of distributions μ_d, ν_d on ℝ^d such that the quanti...

Neural Networks as Kernel Learners: The Silent Alignment Effect

Neural networks in the lazy training regime converge to kernel machines....

Deep Speaker Verification: Do We Need End to End?

End-to-end learning treats the entire system as a whole adaptable black ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.