Contrastive Learning from Spatio-Temporal Mixed Skeleton Sequences for Self-Supervised Skeleton-Based Action Recognition

07/07/2022
by   Zhan Chen, et al.
2

Self-supervised skeleton-based action recognition with contrastive learning has attracted much attention. Recent literature shows that data augmentation and large sets of contrastive pairs are crucial in learning such representations. In this paper, we found that directly extending contrastive pairs based on normal augmentations brings limited returns in terms of performance, because the contribution of contrastive pairs from the normal data augmentation to the loss get smaller as training progresses. Therefore, we delve into hard contrastive pairs for contrastive learning. Motivated by the success of mixing augmentation strategy which improves the performance of many tasks by synthesizing novel samples, we propose SkeleMixCLR: a contrastive learning framework with a spatio-temporal skeleton mixing augmentation (SkeleMix) to complement current contrastive learning approaches by providing hard contrastive samples. First, SkeleMix utilizes the topological information of skeleton data to mix two skeleton sequences by randomly combing the cropped skeleton fragments (the trimmed view) with the remaining skeleton sequences (the truncated view). Second, a spatio-temporal mask pooling is applied to separate these two views at the feature level. Third, we extend contrastive pairs with these two views. SkeleMixCLR leverages the trimmed and truncated views to provide abundant hard contrastive pairs since they involve some context information from each other due to the graph convolution operations, which allows the model to learn better motion representations for action recognition. Extensive experiments on NTU-RGB+D, NTU120-RGB+D, and PKU-MMD datasets show that SkeleMixCLR achieves state-of-the-art performance. Codes are available at https://github.com/czhaneva/SkeleMixCLR.

READ FULL TEXT

page 1

page 4

page 9

page 11

research
08/08/2021

Skeleton-Contrastive 3D Action Representation Learning

This paper strives for self-supervised learning of a feature space suita...
research
11/24/2022

Hierarchical Consistent Contrastive Learning for Skeleton-Based Action Recognition with Growing Augmentations

Contrastive learning has been proven beneficial for self-supervised skel...
research
09/21/2023

Unveiling the Hidden Realm: Self-supervised Skeleton-based Action Recognition in Occluded Environments

To integrate action recognition methods into autonomous robotic systems,...
research
02/17/2023

Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences

Self-supervised learning has demonstrated remarkable capability in repre...
research
12/07/2021

Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition

In recent years, self-supervised representation learning for skeleton-ba...
research
03/10/2023

HYperbolic Self-Paced Learning for Self-Supervised Skeleton-based Action Representations

Self-paced learning has been beneficial for tasks where some initial kno...
research
02/05/2023

Pyramid Self-attention Polymerization Learning for Semi-supervised Skeleton-based Action Recognition

Most semi-supervised skeleton-based action recognition approaches aim to...

Please sign up or login with your details

Forgot password? Click here to reset