Moreau-Yosida f-divergences

02/26/2021
by   Dávid Terjék, et al.
5

Variational representations of f-divergences are central to many machine learning algorithms, with Lipschitz constrained variants recently gaining attention. Inspired by this, we generalize the so-called tight variational representation of f-divergences in the case of probability measures on compact metric spaces to be taken over the space of Lipschitz functions vanishing at an arbitrary base point, characterize functions achieving the supremum in the variational representation, propose a practical algorithm to calculate the tight convex conjugate of f-divergences compatible with automatic differentiation frameworks, define the Moreau-Yosida approximation of f-divergences with respect to the Wasserstein-1 metric, and derive the corresponding variational formulas, providing a generalization of a number of recent results, novel special cases of interest and a relaxation of the hard Lipschitz constraint. As an application of our theoretical results, we propose the Moreau-Yosida f-GAN, providing an implementation of the variational formulas for the Kullback-Leibler, reverse Kullback-Leibler, χ^2, reverse χ^2, squared Hellinger, Jensen-Shannon, Jeffreys, triangular discrimination and total variation divergences as GANs trained on CIFAR-10, leading to competitive results and a simple solution to the problem of uniqueness of the optimal critic.

READ FULL TEXT

page 29

page 30

page 31

page 32

page 33

page 34

page 36

page 37

research
07/02/2018

Understanding the Effectiveness of Lipschitz Constraint in Training of GANs via Gradient Analysis

This paper aims to bring a new perspective for understanding GANs, by de...
research
02/09/2018

Metric Learning via Maximizing the Lipschitz Margin Ratio

In this paper, we propose the Lipschitz margin ratio and a new metric le...
research
12/07/2020

Sobolev Wasserstein GAN

Wasserstein GANs (WGANs), built upon the Kantorovich-Rubinstein (KR) dua...
research
09/26/2017

On the regularization of Wasserstein GANs

Since their invention, generative adversarial networks (GANs) have becom...
research
04/15/2021

Lipschitz Selectors may not Yield Competitive Algorithms for Convex Body Chasing

The current best algorithms for convex body chasing problem in online al...
research
02/19/2021

On a Variational Definition for the Jensen-Shannon Symmetrization of Distances based on the Information Radius

We generalize the Jensen-Shannon divergence by considering a variational...

Please sign up or login with your details

Forgot password? Click here to reset