Temporal Convolutional Networks for Action Segmentation and Detection

11/16/2016
by   Colin Lea, et al.
0

The ability to identify and temporally segment fine-grained human actions throughout a video is crucial for robotics, surveillance, education, and beyond. Typical approaches decouple this problem by first extracting local spatiotemporal features from video frames and then feeding them into a temporal classifier that captures high-level temporal patterns. We introduce a new class of temporal models, which we call Temporal Convolutional Networks (TCNs), that use a hierarchy of temporal convolutions to perform fine-grained action segmentation or detection. Our Encoder-Decoder TCN uses pooling and upsampling to efficiently capture long-range temporal patterns whereas our Dilated TCN uses dilated convolutions. We show that TCNs are capable of capturing action compositions, segment durations, and long-range dependencies, and are over a magnitude faster to train than competing LSTM-based Recurrent Neural Networks. We apply these models to three challenging fine-grained datasets and show large improvements over the state of the art.

READ FULL TEXT

page 1

page 8

02/09/2016

Segmental Spatiotemporal CNNs for Fine-grained Action Segmentation

Joint segmentation and classification of fine-grained actions is importa...
03/05/2019

MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation

Temporally locating and classifying action segments in long untrimmed vi...
08/29/2016

Temporal Convolutional Networks: A Unified Approach to Action Segmentation

The dominant paradigm for video-based action segmentation is composed of...
05/22/2017

TricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action Segmentation

Action segmentation as a milestone towards building automatic systems to...
05/07/2020

Hierarchical Attention Network for Action Segmentation

The temporal segmentation of events is an essential task and a precursor...
02/18/2019

STCN: Stochastic Temporal Convolutional Networks

Convolutional architectures have recently been shown to be competitive o...

Code Repositories

TemporalConvolutionalNetworks

None


view repo

Please sign up or login with your details

Forgot password? Click here to reset