Self-supervised Representation Learning with Relative Predictive Coding

03/21/2021
by   Yao-Hung Hubert Tsai, et al.
32

This paper introduces Relative Predictive Coding (RPC), a new contrastive representation learning objective that maintains a good balance among training stability, minibatch size sensitivity, and downstream task performance. The key to the success of RPC is two-fold. First, RPC introduces the relative parameters to regularize the objective for boundedness and low variance. Second, RPC contains no logarithm and exponential score functions, which are the main cause of training instability in prior contrastive objectives. We empirically verify the effectiveness of RPC on benchmark vision and speech self-supervised learning tasks. Lastly, we relate RPC with mutual information (MI) estimation, showing RPC can be used to estimate MI with low variance.

READ FULL TEXT
research
06/10/2020

Demystifying Self-Supervised Learning: An Information-Theoretical Framework

Self-supervised representation learning adopts self-defined signals as s...
research
06/05/2021

Conditional Contrastive Learning: Removing Undesirable Information in Self-Supervised Representations

Self-supervised learning is a form of unsupervised learning that leverag...
research
12/21/2022

Similarity Contrastive Estimation for Image and Video Soft Contrastive Self-Supervised Learning

Contrastive representation learning has proven to be an effective self-s...
research
05/17/2020

Vector-Quantized Autoregressive Predictive Coding

Autoregressive Predictive Coding (APC), as a self-supervised objective, ...
research
10/22/2022

Guided contrastive self-supervised pre-training for automatic speech recognition

Contrastive Predictive Coding (CPC) is a representation learning method ...
research
10/07/2022

SAICL: Student Modelling with Interaction-level Auxiliary Contrastive Tasks for Knowledge Tracing and Dropout Prediction

Knowledge tracing and dropout prediction are crucial for online educatio...
research
06/10/2020

Self-Supervised Relational Reasoning for Representation Learning

In self-supervised learning, a system is tasked with achieving a surroga...

Please sign up or login with your details

Forgot password? Click here to reset