The Optimal Noise in Noise-Contrastive Learning Is Not What You Think

03/02/2022
by   Omar Chehab, et al.
0

Learning a parametric model of a data distribution is a well-known statistical problem that has seen renewed interest as it is brought to scale in deep learning. Framing the problem as a self-supervised task, where data samples are discriminated from noise samples, is at the core of state-of-the-art methods, beginning with Noise-Contrastive Estimation (NCE). Yet, such contrastive learning requires a good noise distribution, which is hard to specify; domain-specific heuristics are therefore widely used. While a comprehensive theory is missing, it is widely assumed that the optimal noise should in practice be made equal to the data, both in distribution and proportion. This setting underlies Generative Adversarial Networks (GANs) in particular. Here, we empirically and theoretically challenge this assumption on the optimal noise. We show that deviating from this assumption can actually lead to better statistical estimators, in terms of asymptotic variance. In particular, the optimal noise distribution is different from the data's and even from a different family.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/23/2023

Optimizing the Noise in Self-Supervised Learning: from Importance Sampling to Noise-Contrastive Estimation

Self-supervised learning is an increasingly popular approach to unsuperv...
research
11/09/2020

Towards Domain-Agnostic Contrastive Learning

Despite recent success, most contrastive self-supervised learning method...
research
03/09/2023

Learning Stationary Markov Processes with Contrastive Adjustment

We introduce a new optimization algorithm, termed contrastive adjustment...
research
10/01/2022

Pitfalls of Gaussians as a noise distribution in NCE

Noise Contrastive Estimation (NCE) is a popular approach for learning pr...
research
04/04/2023

Fully Variational Noise-Contrastive Estimation

By using the underlying theory of proper scoring rules, we design a fami...
research
11/03/2022

Self-Adapting Noise-Contrastive Estimation for Energy-Based Models

Training energy-based models (EBMs) with noise-contrastive estimation (N...
research
10/21/2021

Analyzing and Improving the Optimization Landscape of Noise-Contrastive Estimation

Noise-contrastive estimation (NCE) is a statistically consistent method ...

Please sign up or login with your details

Forgot password? Click here to reset