Spatiotemporal action localization requires incorporation of two sources of information into the designed architecture: (1) Temporal information from the previous frames and (2) spatial information from the key frame. Current state-of-the-art approaches usually extract these information with separate networks and use an extra mechanism for fusion to get detections. In this work, we present YOWO, a unified CNN architecture for real-time spatiotemporal action localization in video stream. YOWO makes use of a single neural network to extract temporal and spatial information concurrently and predict bounding boxes and action probabilities directly from video clips in one evaluation. Since the whole architecture is unified, it can be optimized end-to-end. The YOWO architecture is fast providing 34 frames-per-second on 16-frames input clips and 62 frames-per-second on 8-frames input clips. Remarkably, YOWO outperforms the previous state-of-the art results on J-HMDB-21 (71.1 UCF101-24 (75.0READ FULL TEXT VIEW PDF
We present YOLO, a new approach to object detection. Prior work on objec...
Analyzing videos of human actions involves understanding the temporal
Current state-of-the-art approaches for spatio-temporal action localizat...
This paper explores the use of convolution LSTMs to simultaneously learn...
Temporal action localization is an important task of computer vision. Th...
Computational Colour Constancy (CCC) consists of estimating the colour o...
Shot boundary detection (SBD) is an important component of many video
This is the source code of the ITSS project. The title of the project is Automatic Detection of Tennis Strokes using Spatio-Temporal Localization.
The topic of spatiotemporal human action localization has been spotlighted in recent years, which aims to not only recognize the occurrence of an action but also localize it in both time and space. In such a task, comparing with object detection in static images, temporal information plays an essential role. Finding an efficient strategy to aggregate spatial as well as temporal features makes the problem even more challenging. On the other hand, real-time human action detection is becoming increasingly crucial in numerous vision applications, such as human-computer interaction (HCI) systems, unmanned aerial vehicle (UAV) monitoring, autonomous driving, and urban security systems. Therefore, it is desirable and worthwhile to explore a more efficient framework to tackle this problem.
Inspired by the remarkable object detection architecture Faster R-CNN , most state-of-the-art works   extend the classic two-stage network architecture to action detection, where a number of proposals are produced in the first stage, then classification and localization refinement are performed in the second stage. However, these two-stage pipelines have three main shortcomings in the spatiotemporal action localization task. Firstly, the generation of action tubes which consist of bounding boxes across frames is much more complicated and time-consuming than 2D case. The classification performance is extremely dependent on these proposals, where the detected bounding boxes might be sub-optimal for the following classification task. Secondly, the action proposals focus only on features of humans in the video, neglecting the relationship between humans and some attributes in the background, which yet is able to provide considerably crucial context information for action prediction. The third problem of a two-stage architecture is that training the region proposal network and the classification network separately does not guarantee to find the global optimum. Instead, only local optimum from the combination of two stages can be found. The training cost is also higher than single-stage networks, hence it takes longer time and needs more memory.
In this paper, we propose a novel single-stage framework, YOWO (You Only Watch Once), for spatiotemporal action localization in videos. YOWO prevents all of the three shortcomings mentioned above with a single-stage architecture. The intuitive idea of YOWO arises from human’s visual cognitive system. For example, when we are absorbed into the story of a soap opera in front of the TV, each time our eyes capture a single frame. In order to understand which action each artist is performing, we have to relate current frame information (2D features from key frame) to the obtained knowledge from previous frames saved in our memory (3D features from clip). Afterwards, these two kinds of features are fused together to provide us with a reasonable conclusion. The example in Fig. 1 illustrates our inspiration.
YOWO architecture is a single-stage network with two branches. One branch extracts the spatial features of the key frame (i.e. current frame) while the other branch models the spatiotemporal features of the clip consisting of previous frames. In order to aggregate these features smoothly, the channel fusion and attention mechanism is introduced, where we get the utmost out of inter-channel dependencies. Finally, we produce frame-level detections using the fused features, and provide a linking algorithm to generate action tubes.
We carry out comprehensive experiments on J-HMDB-21 and UCF101-24 benchmarks and outperform state-of-the-art results with an improvement of 3.3% and 12.2% improvement on frame-mAP, respectively. We achieve these results operating only on RGB modality and maintaining the real-time capability, which contains utmost importance for real-world applications.
Contributions of this paper are summarized as follows:
We propose a real-time single-stage framework for spatio-temporal action localization in video streams, named YOWO, which can be trained end-to-end with high efficiency. To the best of our knowledge, this is the first work which achieves bounding box regression on features extracted by a 2D-CNN and 3D-CNN, concurrently. These two kinds of features have a complementary effect to each other for the final bounding box regression and action classification.
(ii) We propose a channel fusion and attention mechanism (CFAM) to aggregate the features smoothly from two branches above. CFAM models the inter-channel relationship within the concatenated feature maps and boosts the performance significantly by fusing features more reasonably.
(iii) We perform a detailed ablation study on the YOWO architecture. We examined the effect of 3D-CNN, 2D-CNN, their aggregation and the fusion mechanism. Moreover, we have experimented different 3D-CNN architectures and different clip lengths to explore a further trade-off between the precision and speed.
(iv) We evaluate YOWO on the UCF101-24 and J-HMDB-21 datasets. We experimentally observe that the proposed architecture outperforms state-of-the-art frame-mAP results on both datasets significantly. We also get very competitive results on video-mAP compared to the state-of-the-art results.
Action recognition with deep learning.
Since deep learning brings significant improvements in image recognition, numerous recent research efforts have been devoted to extend it for action recognition in videos. For action recognition, however, besides spatial features extracted from each individual image, temporal context across these frames also needs to be taken into account. Two-stream CNN is one effective strategy to extract spatial and temporal features separately and aggregate them together  . Most of these works are based on optical flow, which requires significant computational power to extract, resulting in a time-consuming process. An alternative option to integrate CNN features over time is the implementation of recurrent networks, whose performance, however, is not so satisfying as recent CNN-based methods . 3D-CNNs have been increasingly explored in video analysis tasks recently, which learns the features from both spatial and temporal dimensions simultaneously. 3D-CNN is first exploited to extract spatiotemporal features in  and some effective network architectures like C3D  and I3D  are explored. Inspired by the 2D-CNN residual networks , skip connections over layers are also applied to 3D-CNNs to overcome the problem of vanishing gradients . However, 3D-CNN architectures have much more parameters compared to 2D-CNNs, making them computationally expensive. In , 3D versions of some famous resource efficient CNN architectures are investigated. For resource efficiency, some other works focus on learning 2D features from single images with a 2D-CNN and then fusing them together to learn temporal features with a 3D-CNN .
in the first stage and classify the objects in these potential regions in the second stage. Despite faster R-CNN achieves state-of-the-art results in object detection, it is hard to implement it for real-time tasks due to its time-consuming two-stage architecture. Meanwhile, YOLO  and SSD  aim to simplify this process to one stage and have outstanding real-time performance. For action localization in videos, due to the success of R-CNN series most of the research approaches propose first detecting the humans in each frame and then linking these bounding boxes reasonably as action tubes [11, 24, 13]. Two-stream detectors introduce an additional stream on the base of the original classifier for optical flow modality   . Some other works produce clip tube proposals with 3D-CNNs and achieve regression as well as classification on the corresponding 3D features  , thus region proposal is necessary for them. In a recent work , authors propose a 3D capsule network for video action detection which can jointly perform pixel-wise action segmentation along with action classification. However, it is too expensive in terms of computational complexity and number of parameters since it is a U-Net  based 3D-CNN architecture.
Attention modules. Attention is an effective mechanism to capture long-range dependencies and has been attempted to be used in CNNs to boost the performance in image classification    and scene segmentation . Attention mechanism is implemented spatial-wise and channel-wise in these works, in which spatial attention addresses the inter-spatial relationship among features while channel attention enhances the most meaningful channels and weakens the others. As a remarkable work of channel-wise attention, Squeeze-and-Excitation module  is beneficial to increase CNN’s performance with little computational cost. On the other hand, for video classification tasks, non-local block  takes spatio-temporal information into account simultaneously to learn the dependencies of features across frames, which can be viewed as a self-attention strategy.
Different from previous works, we extend YOLO  in the task of spatio-temporal action localization and design a two-stream model to analyze the spatial and temporal features simultaneously. We name it as YOWO as we make use of a clip only once and detect the corresponding actions in the key frame. However, to avoid the complex optical flow computation, we use 2D features of the key frame and 3D features of the clip together. Afterwards, these two kinds of features are fused together carefully with the application of attention mechanism such that rich contextual relationships are well taken into account.
In this section, we first present YOWO’s architecture in detail, which extracts 2D features from the key frame as well as 3D features from the input clip concurrently and aggregates them together. Then the implementation of channel fusion and attention mechanism is discussed, which provides the essential performance boost. Finally we describe the details of the training process for the YOWO architecture and the improved bounding box linking strategy for generation of action tubes in untrimmed videos.
The YOWO architecture is illustrated in Fig. 2, which can be divided into four major parts: 3D-CNN, 2D-CNN, CFAM and bounding box regression parts.
Since contextual information is crucial for human action understanding, we utilize 3D-CNN for action recognition. 3D-CNNs are able to capture motion information by applying convolution operation not only in space dimension but also in time dimension. The basic 3D-CNN architecture in our framework is 3D-ResNext-101 due to its high performance in Kinetics dataset . In addition to 3D-ResNext-101, we have also experimented with different 3D-CNN models in our ablation study. For all 3D-CNN architectures, all of the layers after the last conv layer are discarded. The input to the 3D network is a clip of a video, which is composed of a sequence of successive frames in time order, and has a shape of , while the last conv layer of 3D ResNext-101 outputs a feature map of shape where , is the number of input frames, and are height and width of input images, is the number of output channels, , and . The depth dimension of the output feature map is reduced to 1 such that output volume is squeezed to in order to match the output feature map of 2D-CNN.
In the meantime, to address the spatial localization problem, 2D features of the key frame are also extracted in parallel. We employ Darknet-19  as the basic backbone of our 2D network due to its good balance between accuracy and efficiency. The key frame with the shape is the most recent frame of the input clip, thus there is no need for an additional data loader. The output feature map of Darknet-19 has a shape of where , is the number of output channels, and similar to the 3D-CNN case.
Furthermore, another important characteristic of YOWO is that both of 3D network and 2D network backbones are able to be replaced by arbitrary CNN architectures, which makes it more flexible. YOWO is designed to be simple and effort-saving to switch models.
We make the outputs of both 3D and 2D networks are of the same shape in the last two dimensions such that these two feature maps can be fused easily. We fuse the two feature maps using concatenation which simply stacks the features along channels. As a result, the fused feature map encodes both motion and appearance information which we pass as input to the CFAM module, which is inspired from .
The concatenated feature map can be regarded as an abrupt combination of 2D and 3D information, which neglects difference and interrelationship between them. To tackle this problem, we produce a new channel fusion and attention mechanism by emphasizing the inter-channel dependency of features.
As illustrated in Fig. 3, our CFAM module integrates channel attention mechanism within its structure. The concatenated feature map , where is the total number of channels in concatenated features, is firstly fed into two convolutional layers to generate a new feature map . Afterwards, several operations are performed on the feature map .
is the reshaped tensor from feature map, where
, which means that features in every single channel is vectorized to one dimension:
Then a matrix product between and its transpose is performed to produce Gram matrix , which indicates the feature correlations across channels :
where each element in the Gram matrix represents the inner product between the vectorised feature map and
. After computing the Gram matrix, a softmax layer is applied to generate channel attention map:
where is a score measuring the channel’s impact on the channel. Thus the attention map can be viewed as it summaries the inter-channel dependency of features given a feature map. To perform the impact of attention map to original features, a further matrix multiplication between the transpose of and is carried out and the result is reshaped back to 3-dimensional space , which has the same shape as the input tensor:
The output of channel attention module combines this result with the original input feature map with a trainable scalar parameter using an element-wise sum operation, and gradually learns a weight from :
The Eq. (6) shows that the final feature of each channel is a weighted sum of the features of all channels and original features, which models the long-range semantic dependencies between feature maps. Finally, the feature map is fed into two more convolutional layers to generate the output feature map of the CFAM module. Two convolutional layers at the beginning and the end of CFAM modules helps to mix the features coming from different backbones and having possibly different distributions.
Such an architecture promotes the feature representativeness in terms of inter-dependencies among channels and thus the features from different dimensions can be aggregated reasonably and smoothly. Besides, Gram matrix takes the whole feature map into consideration, where the dot product of each two flattened feature vectors presents the information about the relation between them. A larger product indicates that the features in these two channels are more correlated while a smaller product suggests that they are different from each other. For a given channel, we allocate more weights to the other channels which are much correlated and have more impact to it. By means of this mechanism, contextual relationship is emphasized and feature discriminability is enhanced.
We follow the same guidelines of YOLO  for bounding box regression. A final convolutional layer with kernels is applied to generate desired number of output channels. For each grid cell in
, 5 prior anchors are selected by k-means technique on corresponding datasets withNumCls class conditional action scores, 4 coordinates and confidence score making the final output size of YOWO . The regression of bounding boxes are then refined based on these anchors.
We have used multi-scale training while the resolution of each frame is set to 224 x 224 at test time. We select the mini-batch stochastic gradient decent algorithm with momentum and weight decay strategy to optimize the loss function, which is defined similar to the original YOLO network except that we apply smooth L loss for localization as in 
since it is less sensitive to outliers than the Lloss and focal loss  for classification loss.
. Although our architecture consists of 2D-CNN and 3D-CNN branches, the parameters are able to be updated jointly. The learning rate is initialized as 0.0001 and reduced with a factor of 0.5 after 30k, 40k, 50k and 60k iterations. For the dataset UCF101-24, the training process is completed after 5 epochs while for J-HMDB-21 after 10 epochs. The complete architecture is implemented and trained end-to-end in PyTorch.
In the trainings, because of the small number of samples in J-HMDB-21, we freeze all the 3D conv network parameters thus the convergence is faster and over-fitting risk can be reduced. In addition, for both UCF101-24 as well as J-HMDB-21, we deploy several data augmentation techniques such as flipping images horizontally in the clip, random scaling and random spatial cropping. During testing, only detected bounding boxes with confidence score larger than threshold 0.25 are selected and then post-processed with non-maximum suppression with a threshold of 0.4.
As we have already obtained frame-level action detections, next step is to link these detected bounding boxes to construct action tubes in the whole video. We apply the similar linking algorithm as described in   to simply find the optimal video-level action detections.
Assume and are two regions from consecutive frames t and t+1, the linking score for an action class is defined as
where , are class specific scores of regions and , is the intersection-over-union of these two regions, and are scalars. is a constraint which is equal to 1 if an overlap exists (), otherwise is equal to 0. We extend the linking score definition in  with an extra element , which takes the dramatic change of scores between two successive frames into account and is able to improve the performance of video detection in experiments. After all the linking scores are computed, Viterbi algorithm is deployed to find the optimal path to generate action tubes. More details are described in .
are selected. We follow the official evaluation metrics strictly to report the results and compare the performance of our method with the state of the art. Moreover, we also do a detailed ablation study in order to explore characteristics of YOWO architecture and contribution of each building block to its performance.
UCF101-24 is a subset of UCF101 , which is originally an action recognition dataset of realistic action videos. UCF101-24 contains 24 action classes and 3207 videos, for which the corresponding spatio-temporal annotations are provided. In addition, there might be multiple action instances in each video, which have the same class label but different spatial and temporal boundaries. Such a property makes video-level action detection much more challenging. As in previous works, we perform all the experiments on the first split.
J-HMDB-21 is a subset of the HMDB-51 dataset  and consists of 928 short videos with 21 action categories in daily life. Each video is well trimmed and has a single action instance across all the frames. We report our experimental results on the first split.
Evaluation metrics: We employ two popular metrics used by the the most researches in the region of spatio-temporal action detection to generate convincing evaluations. Following strictly the rule applied by the PASCAL VOC 2012 metric , frame-mAP measures the area under the precision-recall curve of the detections for each frame. On the other hand, video-mAP focuses on the action tubes . If the mean per frame intersection-over-union with the ground truth across the frames of the whole video is greater than a threshold and in the meanwhile the action label is correctly predicted, then this detected tube is regarded as a correct instance. Finally, the average precision for each class is computed and the average over all classes is reported.
3D network, 2D network or both? Depending only on its own, neither 3D-CNN nor 2D-CNN can solve the spatio-temporal localization task independently. However, if they operate simultaneously, there is potential to benefit from one another. Results on comparing the performance of different architectures are reported in Table 1. We first observe that a single 2D network can not provide a satisfying result since it does not take temporal information into account. A single 3D network is better at capturing motion information and the fusion of 2D and 3D networks (simple concatenation) can improve the performance by 6% compared to 3D network. This indicates that 2D-CNN learns finer spatial features and 3D-CNN concentrates more on the motion process yet the spatial drift of an action in the clip may lead to a lower localization accuracy. It is also shown that CFAM module further boosts the performance from 77.9% to 85.8% on UCF101-24 and from 47.1% to 64.9% on J-HMDB-21. This clearly shows the importance of the attention mechanism which strengthens the inter-dependencies among channels and helps aggregating features more reasonably.
Moreover, in order to explore the impact of each 2D-CNN, 3D-CNN and CFAM blocks, we investigate the localization and the classification performance of different architectures, which is given in Table 2. For localization, we look at the recall value, which is the ratio of the number of correctly localized actions to the total number of proposed detections. For classification, we look at the classification accuracy of the correctly localized detections. For both datasets, 2D network is better at localization while 3D network performs better at classification. It is also obvious that CFAM module boosts both localization and classification performance.
|2D + 3D||77.9||47.1|
|2D + 3D + CFAM||85.8||64.9|
|2D + 3D||93.2||93.7|
|2D + 3D + CFAM||93.5||94.5|
|2D + 3D||94.5||63.0|
|2D + 3D + CFAM||97.3||76.1|
How many frames are suitable for temporal information? For 3D-CNN branch, different clip lengths with different downsampling rates can change the performance of overall YOWO architecture . Therefore, we conduct experiments with 8-frames and 16-frames clips with different downsampling rates, which is given in Table 3. For example, 8-frames (d=3) refers to selecting 8 frames from 24 frames window with downsampling rate of 3. Specifically, we compare three downsampling rates for clip length 8-frames and two downsampling rates for 16-frames clip length.
As expected, we observe that the framework with input of frames performs better than frames since long frame sequence contains more temporal information. However, as downsampling rate is increased, the performance becomes worse. We conjecture that downsampling hinders capturing motion patterns properly and too long sequence may break the temporal contextual relationship. Especially for some quick motion classes, a long sequence may contain several unrelated frames, which can be viewed as noises.
Is it possible to save model complexity with more efficient networks? We have chosen 3D-ResNext-101  since it has multiple cardinalities thus is able to learn more complicated features. However, it is a heavy-weighted backbone with a huge number of parameters and computational complexity. Therefore, we have replaced the 3D backbone with 3D-ResNet with different depths and with some other resource efficient 3D-CNN architectures . Table 4 reports the achieved performance on both datasets. We find that even with light-weight architecture in 3D backbones, our framework is still better than 2D network. However, Table 4 clearly shows the importance of the 3D backbone. The stronger 3D-CNN architecture we use, better the achieved results.
We have compared YOWO with other state-of-the-art architectures on J-HMDB-21 and UCF101-24 datasets. For the sake of fairness, we have excluded VideoCapsuleNet  as it uses different video-mAP calculation without constructing action tubes via some linking strategies. However, YOWO still performs 9.8% and 8.6% better than VideoCapsuleNet in terms of frame-mAP @ 0.5 IoU on J-HMDB-21 and UCF101-24, respectively.
YOWO is compared with the previous state-of-the-art methods on J-HMDB-21 in Table 5. Using the standard metrics, we report the frame-mAP at IOU threshold and the video-mAP at various IOU thresholds. YOWO consistently outperforms the state-of-the-art results on dataset J-HMDB-21, with a frame-mAP increase of and a video-mAP increase of , at IOU thresholds of and , respectively.
Table 6 presents the comparison of YOWO with the state-of-the-art methods on UCF101-24. YOWO achieves with respect to frame-mAP metric, which is significantly better than the others by preceding the second best result with improvement. As for video-mAP, our framework also produces very competitive results even though we just utilize a simple linking strategy.
|Saha et al. ||4||-||36.4|
|ROAD (A) ||40||-||40.9|
Most of the state-of-the-art methods are two stage architectures, which are computationally expensive to run in real time. YOWO is a unified architecture, which can be trained end-to-end. In addition, we do not employ optical flow, which is computationally burdensome. In Table 7, we compare runtime performance of YOWO with other state-of-the-art methods. YOWO’s speed is calculated in terms of frames per second (fps) on a single NVIDIA Titan Xp GPU with a batch size of 8. It must be noted that YOWO’s 2D and 3D backbones can be replaced with any arbitrary CNN model according to the needs.
In general, YOWO architecture performs a decent job at localizing actions in videos, which is illustrated in Fig. 4. However, YOWO also has some drawbacks. Firstly, since YOWO captures all the content of the key frame and the clip, it sometimes makes some false positive detections before the actions are performed. For example, in Fig. 4 first row last image, YOWO sees a person holding a ball at a basketball court and consequently recognizes him very confidently although he is not shooting the ball yet. Secondly, YOWO needs enough temporal content to make correct action localization. If an actor starts performing action suddenly, localization at initial frames lacks temporal content and false actions are recognized consequently, as in Fig. 4 second row last image (climbing stair instead of running).
In this paper, we presented a novel unified architecture for spatiotemporal action localization in video streams. Our approach, YOWO, models the spatiotemporal context from successive frames for action understanding while extracting the fine spatial information from key frame to address the localization task in parallel. In addition, we propose a channel fusion and attention mechanism for effective aggregation of these two kinds of information. Since we do not separate human detection and action classification procedures, the whole network can be optimized by a joint loss in an end-to-end framework. We have carried out a series of comparative evaluations on two challenging representative datasets UCF101-24 and J-HMDB-21. Our approach outperforms the other state-of-the-art results while retaining real-time capability, which makes it possible to deploy it on mobile devices.
The Titan Xp used for this research was donated by the NVIDIA Corporation.
Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning.In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5659–5667, 2017.
Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6546–6555, 2018.
Tube convolutional neural network (t-cnn) for action detection in videos.In Proceedings of the IEEE International Conference on Computer Vision, pages 5822–5831, 2017.