DeepAI AI Chat
Log In Sign Up

Few-Shot Action Recognition with Compromised Metric via Optimal Transport

by   Su Lu, et al.

Although vital to computer vision systems, few-shot action recognition is still not mature despite the wide research of few-shot image classification. Popular few-shot learning algorithms extract a transferable embedding from seen classes and reuse it on unseen classes by constructing a metric-based classifier. One main obstacle to applying these algorithms in action recognition is the complex structure of videos. Some existing solutions sample frames from a video and aggregate their embeddings to form a video-level representation, neglecting important temporal relations. Others perform an explicit sequence matching between two videos and define their distance as matching cost, imposing too strong restrictions on sequence ordering. In this paper, we propose Compromised Metric via Optimal Transport (CMOT) to combine the advantages of these two solutions. CMOT simultaneously considers semantic and temporal information in videos under Optimal Transport framework, and is discriminative for both content-sensitive and ordering-sensitive tasks. In detail, given two videos, we sample segments from them and cast the calculation of their distance as an optimal transport problem between two segment sequences. To preserve the inherent temporal ordering information, we additionally amend the ground cost matrix by penalizing it with the positional distance between a pair of segments. Empirical results on benchmark datasets demonstrate the superiority of CMOT.


page 3

page 8


An Optimal Transport Framework for Zero-Shot Learning

We present an optimal transport (OT) framework for generalized zero-shot...

Universal Prototype Transport for Zero-Shot Action Recognition and Localization

This work addresses the problem of recognizing action categories in vide...

Task-Specific Alignment and Multiple Level Transformer for Few-Shot Action Recognition

In the research field of few-shot learning, the main difference between ...

Learning with Batch-wise Optimal Transport Loss for 3D Shape Recognition

Deep metric learning is essential for visual recognition. The widely use...

Few-shot Action Recognition with Implicit Temporal Alignment and Pair Similarity Optimization

Few-shot learning aims to recognize instances from novel classes with fe...

CLTA: Contents and Length-based Temporal Attention for Few-shot Action Recognition

Few-shot action recognition has attracted increasing attention due to th...

Unsupervised Activity Segmentation by Joint Representation Learning and Online Clustering

We present a novel approach for unsupervised activity segmentation, whic...