Aligned Contrastive Predictive Coding

04/24/2021
by   Jan Chorowski, et al.
20

We investigate the possibility of forcing a self-supervised model trained using a contrastive predictive loss to extract slowly varying latent representations. Rather than producing individual predictions for each of the future representations, the model emits a sequence of predictions shorter than that of the upcoming representations to which they will be aligned. In this way, the prediction network solves a simpler task of predicting the next symbols, but not their exact timing, while the encoding network is trained to produce piece-wise constant latent codes. We evaluate the model on a speech coding task and demonstrate that the proposed Aligned Contrastive Predictive Coding (ACPC) leads to higher linear phone prediction accuracy and lower ABX error rates, while being slightly faster to train due to the reduced number of prediction heads.

READ FULL TEXT
research
07/10/2018

Representation Learning with Contrastive Predictive Coding

While supervised learning has enabled great progress in many application...
research
10/05/2021

Unsupervised Speech Segmentation and Variable Rate Representation Learning using Segmental Contrastive Predictive Coding

Typically, unsupervised segmentation of speech into the phone and word-l...
research
05/04/2022

Zero-Episode Few-Shot Contrastive Predictive Coding: Solving intelligence tests without prior training

Video prediction models often combine three components: an encoder from ...
research
04/21/2020

Vector Quantized Contrastive Predictive Coding for Template-based Music Generation

In this work, we propose a flexible method for generating variations of ...
research
11/11/2022

Investigating Enhancements to Contrastive Predictive Coding for Human Activity Recognition

The dichotomy between the challenging nature of obtaining annotations fo...
research
06/03/2021

Segmental Contrastive Predictive Coding for Unsupervised Word Segmentation

Automatic detection of phoneme or word-like units is one of the core obj...
research
07/08/2020

Analysis of Predictive Coding Models for Phonemic Representation Learning in Small Datasets

Neural network models using predictive coding are interesting from the v...

Please sign up or login with your details

Forgot password? Click here to reset