Contrastive Representation Learning: A Framework and Review

10/10/2020
by   Phuc H. Le-Khac, et al.
24

Contrastive Learning has recently received interest due to its success in self-supervised representation learning in the computer vision domain. However, the origins of Contrastive Learning date as far back as the 1990s and its development has spanned across many fields and domains including Metric Learning and natural language processing. In this paper we provide a comprehensive literature review and we propose a general Contrastive Representation Learning framework that simplifies and unifies many different contrastive learning methods. We also provide a taxonomy for each of the components of contrastive learning in order to summarise it and distinguish it from other forms of machine learning. We then discuss the inductive biases which are present in any contrastive learning system and we analyse our framework under different views from various sub-fields of Machine Learning. Examples of how contrastive learning has been applied in computer vision, natural language processing, audio processing, and others, as well as in Reinforcement Learning are also presented. Finally, we discuss the challenges and some of the most promising future research directions ahead.

READ FULL TEXT

page 15

page 17

page 22

page 25

page 26

page 27

page 28

page 29

06/01/2022

Negative Sampling for Contrastive Representation Learning: A Review

The learn-to-compare paradigm of contrastive representation learning (CR...
07/14/2022

Benchmarking Omni-Vision Representation through the Lens of Visual Realms

Though impressive performance has been achieved in specific visual realm...
07/13/2022

Interactive Machine Learning: A State of the Art Review

Machine learning has proved useful in many software disciplines, includi...
03/01/2022

A Brief Overview of Unsupervised Neural Speech Representation Learning

Unsupervised representation learning for speech processing has matured g...
04/10/2020

A Review on Deep Learning Techniques for Video Prediction

The ability to predict, anticipate and reason about future outcomes is a...
02/25/2022

Raman Spectrum Matching with Contrastive Representation Learning

Raman spectroscopy is an effective, low-cost, non-intrusive technique of...
09/02/2021

An Empirical Study of Graph Contrastive Learning

Graph Contrastive Learning (GCL) establishes a new paradigm for learning...

Code Repositories

ml-visuals

🎨 ML Visuals contains figures and templates which you can reuse and customize to improve your scientific writing.


view repo