
A Minimax Approach to Supervised Learning
Given a task of predicting Y from X, a loss function L, and a set of probability distributions Γ on (X,Y), what is the optimal decision rule minimizing the worstcase expected loss over Γ? In this paper, we address this question by introducing a generalization of the principle of maximum entropy. Applying this principle to sets of distributions with marginal on X constrained to be the empirical marginal from the data, we develop a general minimax approach for supervised learning problems. While for some loss functions such as squarederror and log loss, the minimax approach rederives wellknwon regression models, for the 01 loss it results in a new linear classifier which we call the maximum entropy machine. The maximum entropy machine minimizes the worstcase 01 loss over the structured set of distribution, and by our numerical experiments can outperform other wellknown linear classifiers such as SVM. We also prove a bound on the generalization worstcase error in the minimax approach.
06/07/2016 ∙ by Farzan Farnia, et al. ∙ 0 ∙ shareread it

A Convex Duality Framework for GANs
Generative adversarial network (GAN) is a minimax game between a generator mimicking the true model and a discriminator distinguishing the samples produced by the generator from the real training samples. Given an unconstrained discriminator able to approximate any function, this game reduces to finding the generative model minimizing a divergence measure, e.g. the JensenShannon (JS) divergence, to the data distribution. However, in practice the discriminator is constrained to be in a smaller class F such as neural nets. Then, a natural question is how the divergence minimization interpretation changes as we constrain F. In this work, we address this question by developing a convex duality framework for analyzing GANs. For a convex set F, this duality framework interprets the original GAN formulation as finding the generative model with minimum JSdivergence to the distributions penalized to match the moments of the data distribution, with the moments specified by the discriminators in F. We show that this interpretation more generally holds for fGAN and Wasserstein GAN. As a byproduct, we apply the duality framework to a hybrid of fdivergence and Wasserstein distance. Unlike the fdivergence, we prove that the proposed hybrid divergence changes continuously with the generative model, which suggests regularizing the discriminator's Lipschitz constant in fGAN and vanilla GAN. We numerically evaluate the power of the suggested regularization schemes for improving GAN's training performance.
10/28/2018 ∙ by Farzan Farnia, et al. ∙ 0 ∙ shareread it

Generalizable Adversarial Training via Spectral Normalization
Deep neural networks (DNNs) have set benchmarks on a wide array of supervised learning tasks. Trained DNNs, however, often lack robustness to minor adversarial perturbations to the input, which undermines their true practicality. Recent works have increased the robustness of DNNs by fitting networks using adversariallyperturbed training samples, but the improved performance can still be far below the performance seen in nonadversarial settings. A significant portion of this gap can be attributed to the decrease in generalization performance due to adversarial training. In this work, we extend the notion of margin loss to adversarial settings and bound the generalization error for DNNs trained under several wellknown gradientbased attack schemes, motivating an effective regularization scheme based on spectral normalization of the DNN's weight matrices. We also provide a computationallyefficient method for normalizing the spectral norm of convolutional layers with arbitrary stride and padding schemes in deep convolutional networks. We evaluate the power of spectral normalization extensively on combinations of datasets, network architectures, and adversarial training schemes. The code is available at https://github.com/jessemzhang/dl_spectral_normalization.
11/19/2018 ∙ by Farzan Farnia, et al. ∙ 0 ∙ shareread it
Farzan Farnia
is this you? claim profile