Improving Self-Supervised Learning for Audio Representations by Feature Diversity and Decorrelation

03/07/2023

∙

Self-supervised learning (SSL) has recently shown remarkable results in closing the gap between supervised and unsupervised learning. The idea is to learn robust features that are invariant to distortions of the input data. Despite its success, this idea can suffer from a collapsing issue where the network produces a constant representation. To this end, we introduce SELFIE, a novel Self-supervised Learning approach for audio representation via Feature Diversity and Decorrelation. SELFIE avoids the collapsing issue by ensuring that the representation (i) maintains a high diversity among embeddings and (ii) decorrelates the dependencies between dimensions. SELFIE is pre-trained on the large-scale AudioSet dataset and its embeddings are validated on nine audio downstream tasks, including speech, music, and sound event recognition. Experimental results show that SELFIE outperforms existing SSL methods in several tasks.

READ FULL TEXT

Improving Self-Supervised Learning for Audio Representations by Feature Diversity and Decorrelation

Augmented Contrastive Self-Supervised Learning for Audio Invariant Representations

An empirical study of weakly supervised audio tagging embeddings for general audio representations

Self-supervised audio representation learning for mobile devices

Lightweight feature encoder for wake-up word detection based on self-supervised speech representation

RAMP: Retrieval-Augmented MOS Prediction via Confidence-based Dynamic Weighting

DeLoRes: Decorrelating Latent Spaces for Low-Resource Audio Representation Learning

Self-Supervised Learning of Music-Dance Representation through Explicit-Implicit Rhythm Synchronization

Improving Self-Supervised Learning for Audio Representations by Feature Diversity and Decorrelation

Related Research

Augmented Contrastive Self-Supervised Learning for Audio Invariant Representations

An empirical study of weakly supervised audio tagging embeddings for general audio representations

Self-supervised audio representation learning for mobile devices

Lightweight feature encoder for wake-up word detection based on self-supervised speech representation

RAMP: Retrieval-Augmented MOS Prediction via Confidence-based Dynamic Weighting

DeLoRes: Decorrelating Latent Spaces for Low-Resource Audio Representation Learning

Self-Supervised Learning of Music-Dance Representation through Explicit-Implicit Rhythm Synchronization