Deep Learning for action recognition in videos
Deep 3D CNNs for video action recognition are designed to learn powerful representations in the joint spatio-temporal feature space. In practice however, because of the large number of parameters and computations involved, they may under-perform in the lack of sufficiently large datasets for training them at scale. In this paper we introduce spatial gating in spatial-temporal decomposition of 3D kernels. We implement this concept with Gate-Shift Module (GSM). GSM is lightweight and turns a 2D-CNN into a highly efficient spatio-temporal feature extractor. With GSM plugged in, a 2D-CNN learns to adaptively route features through time and combine them, at almost no additional parameters and computational overhead. We perform an extensive evaluation of the proposed module to study its effectiveness in video action recognition, achieving state-of-the-art results on Something Something-V1 and Diving48 datasets, and obtaining competitive results on EPIC-Kitchens with far less model complexity. With GSM plugged into TSN, on Something Something-V1 we obtain an absolute +32 with less than 1 trained at different temporal scales, we reach beyond 55READ FULL TEXT VIEW PDF
Deep Learning for action recognition in videos
Video action recognition is receiving increasing attention due to its potential applications in video surveillance, media analysis, and robotics, just to mention a few. Although great advances have been achieved during last years, action recognition models have not yet achieved the success of image recognition models, and the ‘AlexNet momentum for video’ has still to come.
A key challenge lies in the space-time nature of the video medium that requires temporal reasoning for fine-grained recognition. Methods based on temporal pooling of frame-level features (TSN , ActionVLAD ) process the video as a (order-less) set of still images and work well enough when the action can be discerned from objects and scene context (UCF-101, Sports-1M, THUMOS). More akin to the time dimension in video, late temporal aggregation of frame-level features can be formulated as sequence learning (LRCN , VideoLSTM ) and with attention (Attentional Pooling , LSTA ). At the other hand, early temporal processing is used to fuse short term motion features from stack of flow fields (Two-Stream ) or predicted directly from the encoded video (DMC-Net ).
Fine-grained recognition can benefit from deeper temporal modeling. Full-3D CNNs (C3D [39, 16]) process the video in space-time by expanding the kernels of a 2D ConvNet along the temporal dimension. Deep C3Ds are designed to learn powerful representations in the joint spatio-temporal feature space with more parameters (3D kernels) and computations (kernels slide over 3 densely sampled dimensions). In practice however, they may under-perform due to the lack of sufficiently large datasets for training them at scale. To cope with these issues arising from the curse of dimension one can narrow down network capacity by design. Fig. 1 shows several C3D kernel decomposition approaches proposed for spatio-temporal feature learning in video. A most intuitive approach is to factorize 3D spatio-temporal kernels into 2D spatial plus 1D temporal, resulting in a structural decomposition that disentangles spatial from temporal interactions (P3D , R(2+1)D , S3D ). An alternative design is separating channel interactions and spatio-temporal interactions via group convolution (CSN ), or modeling both spatial and spatio-temporal interactions in parallel with 2D and 3D convolution on separated channel groups (GST ). Temporal convolution can be constrained to hard-coded time-shifts that move some of the channels forward in time or backward (TSM 
). All these existing approaches learn structured kernels with a hard-wired connectivity and propagation pattern across the network. There is no data dependent decision taken at any point in the network to route features selectively through different branches, for example, group-and-shuffle patterns are fixed by design and learning how to shuffle is combinatorial complexity.
In this paper we introduce spatial gating in spatial-temporal decomposition of 3D kernels. We implement this concept with Gate-Shift Module (GSM) as shown in Fig. 1
. GSM is lightweight and turns a 2D-CNN into a highly efficient spatio-temporal feature extractor. The GSM first applies 2D convolution, then decomposes the output tensor using a learnable spatial gating into two tensors: a gated version of it, and its residual. The gated tensor goes through a 1D temporal convolution while its residual is skip-connected to its output. We implement spatial gating as group spatio-temporal convolution with single output plane per group. We use hard-coded time-shift of channel groups instead of learnable temporal convolution. With GSM plugged in, a 2D-CNN learns to adaptively route features through time and combine them, at almost no additional parameters and computational overhead. For example, when GSM is plugged into TSN, an absolute gain of percentage points in accuracy is obtained on Something Something-V1 dataset with just additional parameters and additional Floating Point Operations.
The contributions of this paper can be summarized as follows: (i) We propose a novel spatio-temporal feature extraction module that can be plugged into existing 2DConvolutional Neural Network (CNN) architectures with negligible overhead in terms of computations and memory; (ii) We perform an extensive ablation analysis of the proposed module to study its effectiveness in video action recognition; (iii) We achieve state-of-the-art or competititve results on public benchmarks with less parameters and FLOPs compared to existing approaches.
Fusing appearance and flow. A popular extension of 2D CNNs to handle video is the Two-Stream architecture by Simonyan and Zisserman . Their method consists of two separated CNNs (streams) that are trained to extract features from a sampled RGB video frame paired with the surrounding stack of optical flow images, followed by a late fusion of the prediction scores of both streams. The image stream encodes the appearance information while the optical flow stream encodes the motion information, that are often found to complement each other for action recognition. Several works followed this approach to find a suitable fusion of the streams at various depths 
and to explore the use of residual connections between them. These approaches rely on optical flow images for motion information, and a single RGB frame for appearance information, which is limiting when reasoning about the temporal context is required for video understanding.
Video as a set or sequence of frames. Later, other approaches were developed using multiple RGB frames for video classification. These approaches sparsely sample multiple frames from the video, which are applied to a 2D CNN followed by a late integration of frame-level features using average pooling 50], recurrent aggregation [5, 24], or attention [12, 35]. To boost performance, most of these approaches also combine video frame sequence with externally computed optical flow. This shows to be helpful, but computationally intensive.
Modeling short-term temporal dependencies. Other research has investigated the middle ground between late aggregation (of frame features) and early temporal processing (to get optical flow), by modeling short-term dependencies. This includes differencing of intermediate features  and combining Sobel filtering with feature differencing . Other works [6, 30] develop a differentiable network that performs TV-L1 , a popular optical flow extraction technique. The work of  instead uses a set of fixed filters for extracting motion features, thereby greatly reducing the number of parameters. DMC-Nets 
leverage motion vectors in the compressed video to synthesize discriminative motion cues for two-stream action recognition at low computational cost compared to raw flow extraction.
Video as a space-time volume. Unconstrained modeling and learning of action features is possible when considering video in space-time. Since video can be seen as a temporally dense sampled sequence of images, expanding 2D convolution operation in 2D-CNNs to 3D convolution is a most intuitive approach to spatio-temporal feature learning [39, 16, 2]. The major drawback of 3D CNNs is the huge number of parameters involved. This results in increased computations and the requirement of large scale datasets for pre-training. Carreira and Zisserman  addressed this limitation by inflating video 3D kernels with the 2D weights of a CNN trained for image recognition. Several other approaches focused on reducing the number of parameters by disentangling the spatial and temporal feature extraction operations. P3D  proposes three different choices for separating the spatial and temporal convolutions and develops a 3D-ResNet architecture whose residual units are a sequence of such three modules. R(2+1)D  and S3D-G  also show that a 2D convolution followed by 1D convolution is enough to learn discriminative features for action recognition. CoST  performs 2D convolutions, with shared parameters, along the three orthogonal dimensions of a video sequence. MultiFiber  uses multiple lightweight networks, the fibers, and multiplexer modules that facilitate information flow using point-wise convolutions across the fibers.
Spatial-temporal modeling. Recently, the focus of research is moving to the development of efficient (from a computational point of view) and effective (from a performance point of view) architectures. CNNs provide different levels of feature abstractions at different layers of the hierarchy. It has been found that the bottom layer features are less useful for extracting discriminative motion cues [36, 53, 47]. In  it is proposed to apply 1D convolution layers on top of a 2D CNN for video action recognition. The works of  and  show that it is more effective to apply full 3D and separable 3D convolutions at the top layers of a 2D CNN for extracting spatio-temporal features. These approaches resulted in performance improvement over full 3D architectures with less parameters and computations. Static features from individual frames represent scenes and objects and can also provide important cues in identifying the action. This is validated by the improved performance obtained with two-path structures that apply a parallel 2D convolution in addition to the 3D convolution [51, 27]. MiCT  is designed by adding 3D convolution branches in parallel to the 2D convolution branches of a BN-Inception-like CNN. GST  makes use of the idea of grouped convolutions for developing an efficient architecture for action recognition. They separate the features at a hierarchy across the channel dimension and separately perform 2D and 3D convolutions followed by a concatenation operation. In this way, the performance is increased while reducing the number of parameters. STM  proposes two parallel blocks for extracting motion features and spatio-temporal features. Their network rely only on 2D and 1D convolutions and feature differencing for encoding motion and spatio-temporal features. TSM  proposes to shift the features across the channel dimension as a way to perform temporal interaction between the features from adjacent frames of a video. This parameter-less approach has resulted in similar performance to 3D CNNs. However, in all previous approaches, spatial, temporal, and channel-wise interaction is hard-wired. Here, we propose the Gate-Shift Module (GSM), which control interactions in spatial-temporal decomposition and learns to adaptively route features thought time and combine them, at almost no additional parameters and computational overhead.
In this section we present Gate-Shift Networks for fine-grained action recognition. We first describe their building block, Gate-Shift Module (GSM), that turns a 2D CNN into a high performing spatio-temporal feature extractor, with minimal overhead. We then discuss and motivate the design choices leading to our final GSM architecture used in the experiments.
Fig. 2 illustrates the network schematics of 3D kernel factorization approaches (cf. Fig. 1) that have been successfully applied to video action recognition. S3D, or R(2+1)D, P3D, decompose 3D convolutions into 2D spatial plus 1D temporal convolutions. TSM replaces 1D temporal convolution with parameter-free channel-wise temporal shift operations. GST uses group convolution where one group applies 2D spatial and the other 3D spatio-temporal convolution. GST furthermore applies point-wise convolution before and after the block to allow for interactions between spatial and spatio-temporal groups, and for channel reduction and up-sampling. In these modules, the feature flow is hard-wired by design, meaning that features are forwarded from one block to the next without data-dependent pooling, gating or routing decision.
GSM design, in Fig. 2, is inspired by GST and TSM but replaces the hard-wired channel split with a learnable spatial gating block. The function of gate block, paired with fuse block, is selectively routing gated features through time-shifts and merging them with the spatially convolved residual to inject spatio-temporal interactions adaptively. GSM is lightweight as it uses 2D kernels, parameter-free time-shifts and few additional parameters to compute the spatial gating planes.
. GSM first applies spatial convolution on the layer input; this is the operation inherited from the 2D CNN base model where GSM is build in. Then, grouped spatial gating is applied, that is, gating planes are obtained for each of two channel groups, and applied on them. This separates the 2D convolution output into group-gated features and residual. The gated features are group-shifted forward and backward in time, and zero-padded. These are finally fused (added) with the residual and propagated to the next layer. This way, GSM selectively mixes spatial and temporal through a learnable spatial gating.
Gating is implemented with a single spatio-temporal 3D kernel and tanh activation. With a 3D kernel we utilize short-range spatio-temporal information in the gating. tanh provides spatial gating planes with values in the range and is motivated as follows. When the gating value at a feature location is +1 and that of the time-shifted feature was +1, then a temporal feature averaging is performed at that location. If the gating value of the time-shifted feature was -1 instead, then a temporal feature differencing is performed. Using tanh, the gating can thus learn to apply either of the two modes, location-wise. It is also found in our ablation study that tanh provides better results than e.g. sigmoid that would be the standard choice with gating, see Sec. 4.3 last paragraph.
GSM layer implementation. Let tensor be the GSM input after 2D convolution (output of blue block in Fig. 3), of shape where is the number of channels and are the spatial and temporal dimensions. Let be the group=2 split of along the channel dimension, and be the two shaped gating kernels. Then, the GSM output is computed as
where ‘’ represents convolution, ‘’ is Hadamard product, and is forward, backward temporal shift. Note that the parameter count in this GSM implementation is ; this is far less than that of a typical C2D block. E.g., the block in Fig. 3 has kernels of size where typically .
We adopt TSN as reference architecture for action recognition. TSN performs temporal pooling of frame-level features using C2D backbone. We choose BN-Inception and InceptionV3 as backbone options with TSN, and describe here how we GSM them.
As shown in Fig. 4, we insert GSM inside one of the branches of Inception blocks. We analyze the branch to which GSM is to be applied, empirically. From the experiments (Sec. 4.3, Tab. 1) we conclude that adding GSM to the branch with the least number of convolution layers performs the best. A hypothesis is that the other branches consist of spatial convolutions with larger kernel sizes and applying GSM on those branches will affect the spatial learning capacity of the network. This hypothesis is strengthened further by observing a reduced performance when GSM was added inside all the branches. Because of the presence of additional branches in Inception blocks, that encode spatial information, there is no need for a separate spatial convolution operation in GSM. That is, the GSM blocks in Fig. 4 are as in Fig. 3 without the spatial convolution block.
For clip level action classification we follow the approach of TSN, that is, we predict the action by average pooling the frame level (now spatio-temporal) scores.
BN-Inception blocks with GSM. The kernel size and stride of convolutional and pooling layers are annotated inside each block.
This section presents an extensive set of experiments to evaluate GSM.
We evaluate Gate-Shift Module (GSM) on three standard action recognition benchmarks, Something Something-V1  (Something-V1), Diving48  and EPIC-Kitchens . Something-V1 consists of 100K videos with 174 fine-grained object manipulation actions. Performance is reported on the validation set. Diving48 dataset contains around 18K videos with 48 fine-grained dive classes. EPIC-Kitchens dataset comprises 34K egocentric videos with fine-grained activity labels. We report the performance obtained on the two standard test splits. Since the test labels are withheld, the recognition scores are obtained from the submission server after we submitted the prediction scores.
All the three considered datasets are diverse in nature and require strong spatio-temporal reasoning for predicting the action classes. For instance, Something-V1 dataset does not distinguish among the objects being handled. On the other hand, EPIC-Kitchens dataset require strong spatio-temporal reasoning as well as information about the objects being handled. The videos in Diving48 dataset generally contain a uniform background with fine-grained diving actions and require strong understanding of temporal dynamics of the human body in the video.
As explained in Sec. 3.2, we choose BN-Inception and InceptionV3 as the CNN backbones. GSM is added inside each Inception block of the respective CNNs. Thus a total of 10 GSMs are added. We initialize the 3D convolution in the gating layer with zeros. Thus the model starts as a standard Temporal Segment Network (TSN
) architecture and the gating is learned during training. All models are initialized with ImageNet pre-trained weights. The entire network is trained end-to-end usingStochastic Gradient Descent (SGD) with an initial learning rate of 0.01 and momentum 0.9. We use a cosine learning rate schedule 
. The network is trained for 60 epochs on Something-V1 and EPIC-Kitchens while Diving48 is trained for 20 epochs. The first 10 epochs are used for gradual warm-up. The batch size is 32 for Something-V1 and EPIC-Kitchens, and 8 for Diving48. Dropout is applied at the classification layer at a rate of 0.5 for Something-V1 and EPIC-Kitchens and 0.7 for Diving48 dataset. Random scaling, cropping and flipping are applied as data augmentation during training. The dimension of the input is and for BN-Inception and InceptionV3, respectively. The reduced input dimension to InceptionV3 reduces the computational complexity without degradation in performance. If not specified, we use just the center crop during inference.
|BN-Inception + 1 GSM||22.7||10.46M||16.37G|
|BN-Inception + 5 GSM||43.13||10.48M||16.39G|
|BN-Inception + 10 GSM||47.24||10.5M||16.46G|
|TSN  (ECCV’16)||BN-Inception||ImageNet||16||32.73||17.52|
|MultiScale TRN  (ECCV’18)||BN-Inception||ImageNet||8||16.37||34.44|
|R(2+1)D  (CVPR’18)||ResNet-34||Sports-1M||32||152||45.7|
|R(2+1)D  from  (CVPR’19)||ResNet-34||External||32||152||51.6|
|S3D-G  (ECCV’18)||InceptionV1||ImageNet||64||71.38||48.2|
|MFNet  (ECCV’18)||ResNet-101||-||10||NA||43.9|
|TrajectoryNet  (NeurIPS’18)||ResNet-18||Kinetics||710||NA||47.8|
|TSM  (ICCV’19)||ResNet-50||Kinetics||16||65||47.2|
|STM  (ICCV’19)||ResNet-50||ImageNet||1630||66.530||50.7|
|GST  (ICCV’19)||ResNet-50||ImageNet||16||59||48.6|
|ABM  (ICCV’19)||ResNet-50||ImageNet||163||35.333||46.08|
|I3D  (CVPR’17)||ResNet-50||Kinetics||322||1082||41.6|
|Non-local  (CVPR’18)||ResNet-50||Kinetics||322||1682||44.4|
|GCN+Non-local  (ECCV’18)||ResNet-50||Kinetics||322||3032||46.1|
|ECO  (ECCV’18)||BNInc + ResNet-18||Kinetics||16||64||41.4|
|Martinez et al.  (ICCV’19)||ResNet-50||ImageNet||NA||52.17NA||50.1|
|Martinez et al.  (ICCV’19)||ResNet-152||ImageNet||NA||113.4NA||53.4|
|Model||#Frames||Params. (M)||FLOPs (G)||Accuracy (%)|
|GSM InceptionV3 En1||8+12||44.42||67.13||52.57||80.79|
|GSM InceptionV3 En2||8+12+16||66.63||120.83||54.04||81.66|
|GSM InceptionV3 En3||8+12+16+24||88.84||201.38||54.88||82.23|
|GSM InceptionV3 En3||82+122+16+24||88.84||268.47||55.16||82.49|
In this section, we report the ablation analysis performed on the validation set of Something-V1 dataset. In all the experiments, we apply 8 frames as input to the network. We first conducted an analysis to determine the Inception branch that is most suitable for adding GSM. The results of this experiment are reported in Tab. 1. We number each branch from left to right. It can be seen that the best performing model is obtained when GSM is added in branch 3. When GSM is added inside all the branches, we observed the lowest performance as this adversely affects the spatial modeling capacity of the network. From the above experiments, we conclude that GSM is most suited to be added inside the branch which contains the least number of convolutions. We follow the same design choice for InceptionV3 as well. More details regarding the architecture of InceptionV3 is provided in the Appendix.
We then compared the performance improvement by adding GSM on BN-Inception. Tab. 2 shows the ablation results. Baseline is the standard TSN architecture, with an accuracy of . We then applied GSM at the last Inception block of the CNN. This improved the recognition performance by . Increasing the number of GSMs added to the backbone consistently improved the recognition performance of the network. The final model, in which GSM is applied in all Inception blocks results in a recognition accuracy of , i.e., absolute improvement over TSN baseline, with only and overhead in parameters and complexity, respectively.
Since sigmoid is the general choice used in gating mechanims, we also analyzed the performance of GSM when sigmoid non-linearity is used inside the gating function. Compared to tanh non-linearity, sigmoid unperforms by absolute (47.24% vs 44.75%) proving the suitability of tanh for gate calibration.
|TSN  (from )||ImageNet||16.77|
|TRN  (from )||ImageNet||22.8|
|R(2+1)D  (from )||Kinetics||28.9|
|P3D  (from )||ImageNet||32.4|
|C3D  (from )||ImageNet||34.5|
|Kanojia et al. ||ImageNet||35.64|
Something-V1. The recognition performance obtained by GSM is compared with state-of-the-art approaches that just use RGB frames in Tab. 3. We also report the number of frames used by each approach during inference and the corresponding computational complexity in terms of FLOPs. The first block in the table lists the approaches that use 2D CNN and efficient 3D CNN implementation. The second block shows the approaches that use Full-3D CNNs. From the table, it can be seen that GSM results in an absolute gain of ( vs ) over the TSN baseline. GSM performs better than 3D CNNs or heavier backbones and also those approaches that use external data for pre-training, with considerably less number of FLOPs. GSM performs comparatively to the top performing method  with less number of FLOPs. It should be noted that the FLOPs of the architecture described in  is computed assuming a single clip of 16 frames. It can also be seen that by using InceptionV3, which is a larger backbone than BN-Inception, the performance of the proposed approach improves. Tab. 4 lists the action recognition accuracy, number of parameters and FLOPS obtained by ensembling the models presented in this work on Something-V1 dataset. The first and second blocks in the table list the accuracy obtained with individual models when evaluated using 1 and 2 clips, respectively. The third block shows the recognition accuracy with different ensemble models. Ensembling is done by combining GSM InceptionV3 models that are trained with different number of input frames. We average the prediction scores obtained from individual models to compute the performance of the ensemble. From the table, it can be seen that the accuracy is increasing as more models are being added. Using models with different number of input frames enables the ensemble to encode the video with different temporal resolutions. Such an ensemble has some analogy with SlowFast . With an ensemble of models trained on 8, 12, 16 and 24 frames, we achieve a state-of-the-art recognition accuracy of 55.16%. Fig. 11-12 show ’visual explanations’ of how GSM has learnt discriminative regions for making a decision (appendix, Sec A).
Diving48. Tab. 5 compares performance of GSM on Diving48 dataset with state-of-the-art approaches. We train the network using 16 frames and sample two clips during inference. We use InceptionV3 as the CNN backbone. In this dataset, the actions cannot be recognized from the scene context alone and require strong spatio-temporal reasoning. GSM achieves a recognition accuracy of , an improvement of over previous state-of-the-art .
In EPIC-Kitchens, the labels are provided as
noun pairs and the performance is evaluated on
action recognition accuracies. For this dataset, we train GSM as a multi-task problem for
action prediction. In the classification layers, we apply
action scores as bias to the
nounclassifiers, as done in LSTA . We use BN-Inception as the backbone CNN. The network is trained with 16 frames. Two clips consisting of 16 frames are sampled from each video during inference. We report the recognition accuracy obtained on the two standard test splits, S1 (seen) and S2 (unseen), in Tab. 6. The first block in the table shows the methods that use both RGB frames and optical flow as inputs while the second block lists the approaches that only use RGB images. From the table, it can be seen that GSM performs better than other approaches that use optical flow images for explicit motion encoding. The only two methods that beat GSM, R(2+1)D  and LFB , train two separate networks, one trained for
verb and the other for
noun classification, and leverage additional data for pre-training. GSM uses a single network for predicting all three labels from a video, thereby making it faster and more memory efficient. In fact, GSM performs better than R(2+1)D model pre-trained on Sports-1M dataset, suggesting that GSM can also improve its performance by pre-training on external data.
In Fig. 4(d), we show the top 10 action classes that improved the most by adding GSM to the CNN backbone of TSN. From the figure, it can be seen that the network has enhanced its ability to distinguish between action classes that are similar in appearance, such as
Unfolding something and
Putting something infront of something and
Removing something, revealing,
something behind, etc. Sample frames from some of these most improved classes are shown in Fig. 6. From these frames, we can see that reversing the order of the frames changes the action and thus the orderless pooling present in TSN fails to identify the action. On the other hand, GSM is able to improve the recognition score on these classes, providing a strong spatio-temporal reasoning. In order to validate the temporal encoding capability of GSM, we evaluated its performance by applying the video frames in the reverse order. This resulted in a drastic degradation in the recognition performance from to . On the other hand, there was no change in the recognition performance of TSN when frames reversed in time were applied.
The t-SNE plots of the features from the final layer of the CNN corresponding to the 10 action groups described in  are shown in Fig. 4(a) and 4(b). Fig. 4(c) shows the t-SNE visualization of the most improved classes compared to TSN. We sample 1800 videos from the validation split for the t-SNE visualization. It can be seen that the features from GSM show a lower intra-class and higher inter-class variability compared to those from TSN.
We also analyzed the memory requirement and computational complexity of GSM and various state-of-the-art approaches. Fig. 7 shows the accuracy, parameter and complexity trade-off computed on the validation set of Somthing-V1 dataset. The graph plots accuracy vs GFLOPs and the area of the bubbles indicate the number of parameters present in each method. From the plot, it can be seen that GSM performs competitively to the state-of-the-art  with less than one tenth the number of parameters and half the number of FLOPs.
We proposed Gate-Shift Module (GSM), a novel temporal interaction block that turns a 2D-CNN into a highly efficient spatio-temporal feature extractor. GSM introduces spatial gating to decide on exchanging information with neighboring frames. We performed an extensive evaluation to study its effectiveness in video action recognition, achieving state-of-the-art results on Something Something-V1 and Diving48 datasets, and obtaining competitive results on EPIC-Kitchens with far less model complexity. For example, when GSM is plugged into TSN, an absolute gain of +32% in recognition accuracy is obtained on Something Something-V1 dataset with just additional parameters and additional FLOPs. With an ensemble of models trained at different sampling rates, GSM achieve an an accuracy of 55%.
We show ‘visual explanations’ for the decisions made by GSM. We use the approach of saliency tubes  for generating the visualizations. In this approach, the frames and their corresponding regions that are used by the model for making a decision are visualized in a form of saliency map. Figs. 11 and 12 compare the saliency tubes generated by the TSN baseline and the proposed GSM approach on sample videos from the validation set of Something Something-V1 dataset. We use the models with BNInception backbone trained using 16 frames for generating the visualizations. Each column in the figures show the 16 frames that are applied as input to the respective networks with the saliency tubes overlaid on top. We show TSN on the left side and GSM on the right side. The classes that improved the most by plugging in GSM on TSN are chosen for visualization. These classes require strong temporal reasoning for understanding the action. From the figures, we can see that TSN focuses on the objects present in the video irrespective of where and when the action takes place, while GSM enables temporal reasoning by focusing on the active object(s) where and when an action is taking place. For example, in Fig. 10(a), an example from the class
putting something in front of something, TSN focuses on the object that is present in the scene, the pen in the first few frames and the cup in the later frames. On the other hand, GSM makes the decision from the frames where the cup is introduced into the video. Similarly, in the example from the class
taking one of many similar things on
the table shown in Fig. 10(d), TSN is focusing on the object, the matchbox, in all the frames while GSM makes the decision based on those frames where the action is taking place.
We first visualize the t-SNE plot of features for the models used in the ablation study, i.e., model with no GSM (Fig. 8(a)), model with 1 GSM (Fig. 8(b)), model with 5 GSM (Fig. 8(c)) and model with 10 GSM (Fig. 8(d)). All figures plot the features of the 10 action groups presented in . From the figures, one can see that adding GSM into the CNN results in a reduction of intra-class variability and in an increase of inter-class variability. Fig. 10 shows the t-SNE plot of features from the last four Inception blocks of BN-Inception with 10 GSM. From the figure, we can see that the semantic separation increases as we move towards the top layers of the backbone.
We provide the details of the CNN architectures used in our GSM models.
shows the architecture of GSM BN-Inception. The Inception modules used are shown in Fig. 4 of the paper. The table also lists the output size after each layer.
|Type||Kernel size/||Output size|
|Inception-GSM 1 (Inc3a)|
|Inception-GSM 1 (Inc3b)|
|Inception-GSM 2 (Inc3c)|
|Inception-GSM 1 (Inc4a)|
|Inception-GSM 1 (Inc4b)|
|Inception-GSM 1 (Inc4c)|
|Inception-GSM 1 (Inc4d)|
|Inception-GSM 2 (Inc4e)|
|Inception-GSM 1 (Inc5a)|
|Inception-GSM 1 (Inc5b)|
layer and ReLU non-linearity.is the number of classes in the dataset.
The architecture of GSM InceptionV3 is shown in Tab. 8 along with the size of the outputs after each layer. We apply an input of size instead of the standard size of . This reduces the computational complexity without affecting the performance of the model. The Inception blocks with GSM used in the model are shown in Fig. 8.
|Type||Kernel size/||Output size|
|Inception-GSM (Fig. 1a)|
|Inception-GSM (Fig. 1a)|
|Inception-GSM (Fig. 1c)|
|Inception-GSM (Fig. 1b)|
|Inception-GSM (Fig. 1b)|
|Inception-GSM (Fig. 1b)|
|Inception-GSM (Fig. 1b)|
|Inception-GSM (Fig. 1b)|
|Inception-GSM (Fig. 1d)|
|Inception-GSM (Fig. 1e)|
|Inception-GSM (Fig. 1e)|
Joint pattern recognition symposium, pages 214–223, 2007.