Temporal Action Detection with Multi-level Supervision

by   Baifeng Shi, et al.

Training temporal action detection in videos requires large amounts of labeled data, yet such annotation is expensive to collect. Incorporating unlabeled or weakly-labeled data to train action detection model could help reduce annotation cost. In this work, we first introduce the Semi-supervised Action Detection (SSAD) task with a mixture of labeled and unlabeled data and analyze different types of errors in the proposed SSAD baselines which are directly adapted from the semi-supervised classification task. To alleviate the main error of action incompleteness (i.e., missing parts of actions) in SSAD baselines, we further design an unsupervised foreground attention (UFA) module utilizing the "independence" between foreground and background motion. Then we incorporate weakly-labeled data into SSAD and propose Omni-supervised Action Detection (OSAD) with three levels of supervision. An information bottleneck (IB) suppressing the scene information in non-action frames while preserving the action information is designed to help overcome the accompanying action-context confusion problem in OSAD baselines. We extensively benchmark against the baselines for SSAD and OSAD on our created data splits in THUMOS14 and ActivityNet1.2, and demonstrate the effectiveness of the proposed UFA and IB methods. Lastly, the benefit of our full OSAD-IB model under limited annotation budgets is shown by exploring the optimal annotation strategy for labeled, unlabeled and weakly-labeled data.


A Semi-Supervised Learning Approach with Two Teachers to Improve Breakdown Identification in Dialogues

Identifying breakdowns in ongoing dialogues helps to improve communicati...

Learning Temporal Action Proposals With Fewer Labels

Temporal action proposals are a common module in action detection pipeli...

Cross-Model Pseudo-Labeling for Semi-Supervised Action Recognition

Semi-supervised action recognition is a challenging but important task d...

Semi-supervised 3D Object Detection via Temporal Graph Neural Networks

3D object detection plays an important role in autonomous driving and ot...

Visual Distant Supervision for Scene Graph Generation

Scene graph generation aims to identify objects and their relations in i...

Pushing the Performance Limit of Scene Text Recognizer without Human Annotation

Scene text recognition (STR) attracts much attention over the years beca...

A Semi-supervised Graph Attentive Network for Financial Fraud Detection

With the rapid growth of financial services, fraud detection has been a ...