Simple and Effective Balance of Contrastive Losses

by   Arnaud Sors, et al.

Contrastive losses have long been a key ingredient of deep metric learning and are now becoming more popular due to the success of self-supervised learning. Recent research has shown the benefit of decomposing such losses into two sub-losses which act in a complementary way when learning the representation network: a positive term and an entropy term. Although the overall loss is thus defined as a combination of two terms, the balance of these two terms is often hidden behind implementation details and is largely ignored and sub-optimal in practice. In this work, we approach the balance of contrastive losses as a hyper-parameter optimization problem, and propose a coordinate descent-based search method that efficiently find the hyper-parameters that optimize evaluation performance. In the process, we extend existing balance analyses to the contrastive margin loss, include batch size in the balance, and explain how to aggregate loss elements from the batch to maintain near-optimal performance over a larger range of batch sizes. Extensive experiments with benchmarks from deep metric learning and self-supervised learning show that optimal hyper-parameters are found faster with our method than with other common search methods.


page 3

page 6

page 12


Self-Supervised Video Representation Using Pretext-Contrastive Learning

Pretext tasks and contrastive learning have been successful in self-supe...

Tuned Contrastive Learning

In recent times, contrastive learning based loss functions have become i...

Learning the Relation between Similarity Loss and Clustering Loss in Self-Supervised Learning

Self-supervised learning enables networks to learn discriminative featur...

Deep Metric Learning with Spherical Embedding

Deep metric learning has attracted much attention in recent years, due t...

Provable Stochastic Optimization for Global Contrastive Learning: Small Batch Does Not Harm Performance

In this paper, we study contrastive learning from an optimization perspe...

Please sign up or login with your details

Forgot password? Click here to reset