Gate-Shift Networks for Video Action Recognition

Deep 3D CNNs for video action recognition are designed to learn powerful representations in the joint spatio-temporal feature space. In practice however, because of the large number of parameters and computations involved, they may under-perform in the lack of sufficiently large datasets for training them at scale. In this paper we introduce spatial gating in spatial-temporal decomposition of 3D kernels. We implement this concept with Gate-Shift Module (GSM). GSM is lightweight and turns a 2D-CNN into a highly efficient spatio-temporal feature extractor. With GSM plugged in, a 2D-CNN learns to adaptively route features through time and combine them, at almost no additional parameters and computational overhead. We perform an extensive evaluation of the proposed module to study its effectiveness in video action recognition, achieving state-of-the-art results on Something Something-V1 and Diving48 datasets, and obtaining competitive results on EPIC-Kitchens with far less model complexity. With GSM plugged into TSN, on Something Something-V1 we obtain an absolute +32 with less than 1 trained at different temporal scales, we reach beyond 55


page 9

page 11

page 12

page 13

page 14


Gate-Shift-Fuse for Video Action Recognition

Convolutional Neural Networks are the de facto models for image recognit...

STSM: Spatio-Temporal Shift Module for Efficient Action Recognition

The modeling, computational cost, and accuracy of traditional Spatio-tem...

Collaborative Spatio-temporal Feature Learning for Video Action Recognition

Spatio-temporal feature learning is of central importance for action rec...

Efficient Remote Photoplethysmography with Temporal Derivative Modules and Time-Shift Invariant Loss

We present a lightweight neural model for remote heart rate estimation f...

Delving Deeper into Convolutional Networks for Learning Video Representations

We propose an approach to learn spatio-temporal features in videos from ...

GCF-Net: Gated Clip Fusion Network for Video Action Recognition

In recent years, most of the accuracy gains for video action recognition...

Interpretable Deep Feature Propagation for Early Action Recognition

Early action recognition (action prediction) from limited preliminary ob...

Code Repositories


Deep Learning for action recognition in videos

view repo

1 Introduction

Video action recognition is receiving increasing attention due to its potential applications in video surveillance, media analysis, and robotics, just to mention a few. Although great advances have been achieved during last years, action recognition models have not yet achieved the success of image recognition models, and the ‘AlexNet momentum for video’ has still to come.

A key challenge lies in the space-time nature of the video medium that requires temporal reasoning for fine-grained recognition. Methods based on temporal pooling of frame-level features (TSN [43], ActionVLAD [13]) process the video as a (order-less) set of still images and work well enough when the action can be discerned from objects and scene context (UCF-101, Sports-1M, THUMOS). More akin to the time dimension in video, late temporal aggregation of frame-level features can be formulated as sequence learning (LRCN [5], VideoLSTM [24]) and with attention (Attentional Pooling [12], LSTA [35]). At the other hand, early temporal processing is used to fuse short term motion features from stack of flow fields (Two-Stream [33]) or predicted directly from the encoded video (DMC-Net [32]).

Figure 1: 3D kernel factorization for spatio-temporal learning in video. Existing approaches decompose into channel-wise (CSN), spatial followed by temporal (S3D, TSM), or grouped spatial and spatio-temporal (GST). In all these, spatial, temporal, and channel-wise interaction is hard-wired. Our Gate-Shift Module (GSM) leverages group spatial gating (blocks in green) to control interactions in spatial-temporal decomposition. GSM is lightweight and a building block of high performing video feature extractors.

Fine-grained recognition can benefit from deeper temporal modeling. Full-3D CNNs (C3D [39, 16]) process the video in space-time by expanding the kernels of a 2D ConvNet along the temporal dimension. Deep C3Ds are designed to learn powerful representations in the joint spatio-temporal feature space with more parameters (3D kernels) and computations (kernels slide over 3 densely sampled dimensions). In practice however, they may under-perform due to the lack of sufficiently large datasets for training them at scale. To cope with these issues arising from the curse of dimension one can narrow down network capacity by design. Fig. 1 shows several C3D kernel decomposition approaches proposed for spatio-temporal feature learning in video. A most intuitive approach is to factorize 3D spatio-temporal kernels into 2D spatial plus 1D temporal, resulting in a structural decomposition that disentangles spatial from temporal interactions (P3D [31], R(2+1)D [41], S3D [47]). An alternative design is separating channel interactions and spatio-temporal interactions via group convolution (CSN [40]), or modeling both spatial and spatio-temporal interactions in parallel with 2D and 3D convolution on separated channel groups (GST [27]). Temporal convolution can be constrained to hard-coded time-shifts that move some of the channels forward in time or backward (TSM [25]

). All these existing approaches learn structured kernels with a hard-wired connectivity and propagation pattern across the network. There is no data dependent decision taken at any point in the network to route features selectively through different branches, for example, group-and-shuffle patterns are fixed by design and learning how to shuffle is combinatorial complexity.

In this paper we introduce spatial gating in spatial-temporal decomposition of 3D kernels. We implement this concept with Gate-Shift Module (GSM) as shown in Fig. 1

. GSM is lightweight and turns a 2D-CNN into a highly efficient spatio-temporal feature extractor. The GSM first applies 2D convolution, then decomposes the output tensor using a learnable spatial gating into two tensors: a gated version of it, and its residual. The gated tensor goes through a 1D temporal convolution while its residual is skip-connected to its output. We implement spatial gating as group spatio-temporal convolution with single output plane per group. We use hard-coded time-shift of channel groups instead of learnable temporal convolution. With GSM plugged in, a 2D-CNN learns to adaptively route features through time and combine them, at almost no additional parameters and computational overhead. For example, when GSM is plugged into TSN 

[43], an absolute gain of percentage points in accuracy is obtained on Something Something-V1 dataset with just additional parameters and additional Floating Point Operations.

The contributions of this paper can be summarized as follows: (i) We propose a novel spatio-temporal feature extraction module that can be plugged into existing 2D

Convolutional Neural Network (CNN) architectures with negligible overhead in terms of computations and memory; (ii) We perform an extensive ablation analysis of the proposed module to study its effectiveness in video action recognition; (iii) We achieve state-of-the-art or competititve results on public benchmarks with less parameters and FLOPs compared to existing approaches.

The rest of the paper is organized as follows. Sec. 2 reviews work on action recognition most related to ours. Sec. 3 presents the proposed approach. Experimental results are reported in Sec. 4 and Sec. 5 concludes the paper.

2 Related Work

Inspired by the performance improvements obtained with deep convolutional architectures in image recognition [17, 38], much effort has gone into extending these for video action recognition.

Fusing appearance and flow. A popular extension of 2D CNNs to handle video is the Two-Stream architecture by Simonyan and Zisserman [33]. Their method consists of two separated CNNs (streams) that are trained to extract features from a sampled RGB video frame paired with the surrounding stack of optical flow images, followed by a late fusion of the prediction scores of both streams. The image stream encodes the appearance information while the optical flow stream encodes the motion information, that are often found to complement each other for action recognition. Several works followed this approach to find a suitable fusion of the streams at various depths [9]

and to explore the use of residual connections between them 

[8]. These approaches rely on optical flow images for motion information, and a single RGB frame for appearance information, which is limiting when reasoning about the temporal context is required for video understanding.

Video as a set or sequence of frames. Later, other approaches were developed using multiple RGB frames for video classification. These approaches sparsely sample multiple frames from the video, which are applied to a 2D CNN followed by a late integration of frame-level features using average pooling [43]

, multilayer perceptrons 

[50], recurrent aggregation [5, 24], or attention [12, 35]. To boost performance, most of these approaches also combine video frame sequence with externally computed optical flow. This shows to be helpful, but computationally intensive.

Modeling short-term temporal dependencies. Other research has investigated the middle ground between late aggregation (of frame features) and early temporal processing (to get optical flow), by modeling short-term dependencies. This includes differencing of intermediate features [29] and combining Sobel filtering with feature differencing [37]. Other works [6, 30] develop a differentiable network that performs TV-L1 [48], a popular optical flow extraction technique. The work of [21] instead uses a set of fixed filters for extracting motion features, thereby greatly reducing the number of parameters. DMC-Nets [32]

leverage motion vectors in the compressed video to synthesize discriminative motion cues for two-stream action recognition at low computational cost compared to raw flow extraction.

Video as a space-time volume. Unconstrained modeling and learning of action features is possible when considering video in space-time. Since video can be seen as a temporally dense sampled sequence of images, expanding 2D convolution operation in 2D-CNNs to 3D convolution is a most intuitive approach to spatio-temporal feature learning [39, 16, 2]. The major drawback of 3D CNNs is the huge number of parameters involved. This results in increased computations and the requirement of large scale datasets for pre-training. Carreira and Zisserman [2] addressed this limitation by inflating video 3D kernels with the 2D weights of a CNN trained for image recognition. Several other approaches focused on reducing the number of parameters by disentangling the spatial and temporal feature extraction operations. P3D [31] proposes three different choices for separating the spatial and temporal convolutions and develops a 3D-ResNet architecture whose residual units are a sequence of such three modules. R(2+1)D [41] and S3D-G [47] also show that a 2D convolution followed by 1D convolution is enough to learn discriminative features for action recognition. CoST [22] performs 2D convolutions, with shared parameters, along the three orthogonal dimensions of a video sequence. MultiFiber [3] uses multiple lightweight networks, the fibers, and multiplexer modules that facilitate information flow using point-wise convolutions across the fibers.

Spatial-temporal modeling. Recently, the focus of research is moving to the development of efficient (from a computational point of view) and effective (from a performance point of view) architectures. CNNs provide different levels of feature abstractions at different layers of the hierarchy. It has been found that the bottom layer features are less useful for extracting discriminative motion cues [36, 53, 47]. In [36] it is proposed to apply 1D convolution layers on top of a 2D CNN for video action recognition. The works of [53] and [47] show that it is more effective to apply full 3D and separable 3D convolutions at the top layers of a 2D CNN for extracting spatio-temporal features. These approaches resulted in performance improvement over full 3D architectures with less parameters and computations. Static features from individual frames represent scenes and objects and can also provide important cues in identifying the action. This is validated by the improved performance obtained with two-path structures that apply a parallel 2D convolution in addition to the 3D convolution [51, 27]. MiCT [51] is designed by adding 3D convolution branches in parallel to the 2D convolution branches of a BN-Inception-like CNN. GST [27] makes use of the idea of grouped convolutions for developing an efficient architecture for action recognition. They separate the features at a hierarchy across the channel dimension and separately perform 2D and 3D convolutions followed by a concatenation operation. In this way, the performance is increased while reducing the number of parameters. STM [18] proposes two parallel blocks for extracting motion features and spatio-temporal features. Their network rely only on 2D and 1D convolutions and feature differencing for encoding motion and spatio-temporal features. TSM [25] proposes to shift the features across the channel dimension as a way to perform temporal interaction between the features from adjacent frames of a video. This parameter-less approach has resulted in similar performance to 3D CNNs. However, in all previous approaches, spatial, temporal, and channel-wise interaction is hard-wired. Here, we propose the Gate-Shift Module (GSM), which control interactions in spatial-temporal decomposition and learns to adaptively route features thought time and combine them, at almost no additional parameters and computational overhead.

3 Gate-Shift Networks

In this section we present Gate-Shift Networks for fine-grained action recognition. We first describe their building block, Gate-Shift Module (GSM), that turns a 2D CNN into a high performing spatio-temporal feature extractor, with minimal overhead. We then discuss and motivate the design choices leading to our final GSM architecture used in the experiments.

Figure 2: C3D decomposition approaches in comparison to GSM schematics. GSM is inspired by GST and TSM but replaces the hard-wired channel split with a learnable spatial gating block.

3.1 Gate-Shift Module

Figure 3: GSM implementation with group gating and forward-backward temporal shift. A gate is a single 3D convolution kernel with tanh calibration, thus very few parameters are added when GSM is used to turn a C2D base model into a spatio-temporal feature extractor.

Fig. 2 illustrates the network schematics of 3D kernel factorization approaches (cf. Fig. 1) that have been successfully applied to video action recognition. S3D, or R(2+1)D, P3D, decompose 3D convolutions into 2D spatial plus 1D temporal convolutions. TSM replaces 1D temporal convolution with parameter-free channel-wise temporal shift operations. GST uses group convolution where one group applies 2D spatial and the other 3D spatio-temporal convolution. GST furthermore applies point-wise convolution before and after the block to allow for interactions between spatial and spatio-temporal groups, and for channel reduction and up-sampling. In these modules, the feature flow is hard-wired by design, meaning that features are forwarded from one block to the next without data-dependent pooling, gating or routing decision.

GSM design, in Fig. 2, is inspired by GST and TSM but replaces the hard-wired channel split with a learnable spatial gating block. The function of gate block, paired with fuse block, is selectively routing gated features through time-shifts and merging them with the spatially convolved residual to inject spatio-temporal interactions adaptively. GSM is lightweight as it uses 2D kernels, parameter-free time-shifts and few additional parameters to compute the spatial gating planes.

Based on the conceptual design in Fig. 2, we instantiate GSM as in Fig. 3

. GSM first applies spatial convolution on the layer input; this is the operation inherited from the 2D CNN base model where GSM is build in. Then, grouped spatial gating is applied, that is, gating planes are obtained for each of two channel groups, and applied on them. This separates the 2D convolution output into group-gated features and residual. The gated features are group-shifted forward and backward in time, and zero-padded. These are finally fused (added) with the residual and propagated to the next layer. This way, GSM selectively mixes spatial and temporal through a learnable spatial gating.

Gating is implemented with a single spatio-temporal 3D kernel and tanh activation. With a 3D kernel we utilize short-range spatio-temporal information in the gating. tanh provides spatial gating planes with values in the range and is motivated as follows. When the gating value at a feature location is +1 and that of the time-shifted feature was +1, then a temporal feature averaging is performed at that location. If the gating value of the time-shifted feature was -1 instead, then a temporal feature differencing is performed. Using tanh, the gating can thus learn to apply either of the two modes, location-wise. It is also found in our ablation study that tanh provides better results than e.g. sigmoid that would be the standard choice with gating, see Sec. 4.3 last paragraph.

GSM layer implementation. Let tensor be the GSM input after 2D convolution (output of blue block in Fig. 3), of shape where is the number of channels and are the spatial and temporal dimensions. Let be the group=2 split of along the channel dimension, and be the two shaped gating kernels. Then, the GSM output is computed as


where ‘’ represents convolution, ‘’ is Hadamard product, and is forward, backward temporal shift. Note that the parameter count in this GSM implementation is ; this is far less than that of a typical C2D block. E.g., the block in Fig. 3 has kernels of size where typically .

3.2 Gate-Shift Architecture

We adopt TSN as reference architecture for action recognition. TSN performs temporal pooling of frame-level features using C2D backbone. We choose BN-Inception and InceptionV3 as backbone options with TSN, and describe here how we GSM them.

As shown in Fig. 4, we insert GSM inside one of the branches of Inception blocks. We analyze the branch to which GSM is to be applied, empirically. From the experiments (Sec. 4.3, Tab. 1) we conclude that adding GSM to the branch with the least number of convolution layers performs the best. A hypothesis is that the other branches consist of spatial convolutions with larger kernel sizes and applying GSM on those branches will affect the spatial learning capacity of the network. This hypothesis is strengthened further by observing a reduced performance when GSM was added inside all the branches. Because of the presence of additional branches in Inception blocks, that encode spatial information, there is no need for a separate spatial convolution operation in GSM. That is, the GSM blocks in Fig. 4 are as in Fig. 3 without the spatial convolution block.

For clip level action classification we follow the approach of TSN, that is, we predict the action by average pooling the frame level (now spatio-temporal) scores.

Figure 4:

BN-Inception blocks with GSM. The kernel size and stride of convolutional and pooling layers are annotated inside each block.

4 Experiments and Results

This section presents an extensive set of experiments to evaluate GSM.

4.1 Datasets

We evaluate Gate-Shift Module (GSM) on three standard action recognition benchmarks, Something Something-V1 [15] (Something-V1), Diving48 [23] and EPIC-Kitchens [4]. Something-V1 consists of 100K videos with 174 fine-grained object manipulation actions. Performance is reported on the validation set. Diving48 dataset contains around 18K videos with 48 fine-grained dive classes. EPIC-Kitchens dataset comprises 34K egocentric videos with fine-grained activity labels. We report the performance obtained on the two standard test splits. Since the test labels are withheld, the recognition scores are obtained from the submission server after we submitted the prediction scores.

All the three considered datasets are diverse in nature and require strong spatio-temporal reasoning for predicting the action classes. For instance, Something-V1 dataset does not distinguish among the objects being handled. On the other hand, EPIC-Kitchens dataset require strong spatio-temporal reasoning as well as information about the objects being handled. The videos in Diving48 dataset generally contain a uniform background with fine-grained diving actions and require strong understanding of temporal dynamics of the human body in the video.

4.2 Implementation Details

As explained in Sec. 3.2, we choose BN-Inception and InceptionV3 as the CNN backbones. GSM is added inside each Inception block of the respective CNNs. Thus a total of 10 GSMs are added. We initialize the 3D convolution in the gating layer with zeros. Thus the model starts as a standard Temporal Segment Network (TSN

) architecture and the gating is learned during training. All models are initialized with ImageNet pre-trained weights. The entire network is trained end-to-end using

Stochastic Gradient Descent (SGD) with an initial learning rate of 0.01 and momentum 0.9. We use a cosine learning rate schedule [26]

. The network is trained for 60 epochs on Something-V1 and EPIC-Kitchens while Diving48 is trained for 20 epochs. The first 10 epochs are used for gradual warm-up 

[14]. The batch size is 32 for Something-V1 and EPIC-Kitchens, and 8 for Diving48. Dropout is applied at the classification layer at a rate of 0.5 for Something-V1 and EPIC-Kitchens and 0.7 for Diving48 dataset. Random scaling, cropping and flipping are applied as data augmentation during training. The dimension of the input is and for BN-Inception and InceptionV3, respectively. The reduced input dimension to InceptionV3 reduces the computational complexity without degradation in performance. If not specified, we use just the center crop during inference.

Branch Accuracy (%)
Branch 1 45.11
Branch 2 44.98
Branch 3 45.05
Branch 4 47.24
All branches 43.5
Table 1: Ablation analysis done to determine the Inception branch that is most suitable for plugging in GSM.
Model Accuracy (%) Params. FLOPs
BN-Inception (baseline) 17.25 10.45M 16.37G
BN-Inception + 1 GSM 22.7 10.46M 16.37G
BN-Inception + 5 GSM 43.13 10.48M 16.39G
BN-Inception + 10 GSM 47.24 10.5M 16.46G
Table 2: Recognition Accuracy by varying the number of Gate-Shift Modules (GSMs) added to the backbone.
Method Backbone Pre-training #Frames GFLOPs Accuracy (%)
TSN [43] (ECCV’16) BN-Inception ImageNet 16 32.73 17.52
MultiScale TRN [50] (ECCV’18) BN-Inception ImageNet 8 16.37 34.44
R(2+1)D [41] (CVPR’18) ResNet-34 Sports-1M 32 152 45.7
R(2+1)D [41] from [11] (CVPR’19) ResNet-34 External 32 152 51.6
S3D-G [47] (ECCV’18) InceptionV1 ImageNet 64 71.38 48.2
MFNet [21] (ECCV’18) ResNet-101 - 10 NA 43.9
TrajectoryNet [49] (NeurIPS’18) ResNet-18 Kinetics 710 NA 47.8
TSM [25] (ICCV’19) ResNet-50 Kinetics 16 65 47.2
STM [18] (ICCV’19) ResNet-50 ImageNet 1630 66.530 50.7
GST [27] (ICCV’19) ResNet-50 ImageNet 16 59 48.6
ABM [52] (ICCV’19) ResNet-50 ImageNet 163 35.333 46.08
CorrNet [42] ResNet-101 - 3230 22430 51.1
I3D [2] (CVPR’17) ResNet-50 Kinetics 322 1082 41.6
Non-local [44] (CVPR’18) ResNet-50 Kinetics 322 1682 44.4
GCN+Non-local [45] (ECCV’18) ResNet-50 Kinetics 322 3032 46.1
ECO [53] (ECCV’18) BNInc + ResNet-18 Kinetics 16 64 41.4
Martinez et al[28] (ICCV’19) ResNet-50 ImageNet NA 52.17NA 50.1
Martinez et al[28] (ICCV’19) ResNet-152 ImageNet NA 113.4NA 53.4
GSM BN-Inception ImageNet 8 16.46 47.24
InceptionV3 ImageNet 8 26.85 49.01
BN-Inception ImageNet 16 32.92 49.56
InceptionV3 ImageNet 16 53.7 50.63
InceptionV3 ImageNet 162 53.72 51.68
Table 3: Comparison to state-of-the-art on Something-V1. : Computed assuming a single clip of 16 frames as input.
Model #Frames Params. (M) FLOPs (G) Accuracy (%)
Top1 Top5
GSM InceptionV3 8 22.21 26.85 49.01 76.99
GSM InceptionV3 12 22.21 40.26 51.58 79.38
GSM InceptionV3 16 22.21 53.7 50.63 79.17
GSM InceptionV3 24 22.21 80.55 49.63 78.24
GSM InceptionV3 82 22.21 53.7 50.43 78.19
GSM InceptionV3 122 22.21 80.55 51.98 79.99
GSM InceptionV3 162 22.21 107.4 51.68 79.64
GSM InceptionV3 242 22.21 161.1 50.35 78.68
GSM InceptionV3 En1 8+12 44.42 67.13 52.57 80.79
GSM InceptionV3 En2 8+12+16 66.63 120.83 54.04 81.66
GSM InceptionV3 En3 8+12+16+24 88.84 201.38 54.88 82.23
GSM InceptionV3 En3 82+122+16+24 88.84 268.47 55.16 82.49
Table 4: Recognition Accuracy obtained on Something Something-V1 dataset by ensembling different models.

4.3 Ablation Analysis

In this section, we report the ablation analysis performed on the validation set of Something-V1 dataset. In all the experiments, we apply 8 frames as input to the network. We first conducted an analysis to determine the Inception branch that is most suitable for adding GSM. The results of this experiment are reported in Tab. 1. We number each branch from left to right. It can be seen that the best performing model is obtained when GSM is added in branch 3. When GSM is added inside all the branches, we observed the lowest performance as this adversely affects the spatial modeling capacity of the network. From the above experiments, we conclude that GSM is most suited to be added inside the branch which contains the least number of convolutions. We follow the same design choice for InceptionV3 as well. More details regarding the architecture of InceptionV3 is provided in the Appendix.

We then compared the performance improvement by adding GSM on BN-Inception. Tab. 2 shows the ablation results. Baseline is the standard TSN architecture, with an accuracy of . We then applied GSM at the last Inception block of the CNN. This improved the recognition performance by . Increasing the number of GSMs added to the backbone consistently improved the recognition performance of the network. The final model, in which GSM is applied in all Inception blocks results in a recognition accuracy of , i.e., absolute improvement over TSN baseline, with only and overhead in parameters and complexity, respectively.

Since sigmoid is the general choice used in gating mechanims, we also analyzed the performance of GSM when sigmoid non-linearity is used inside the gating function. Compared to tanh non-linearity, sigmoid unperforms by absolute (47.24% vs 44.75%) proving the suitability of tanh for gate calibration.

Method Pre-training Accuracy (%)
TSN [43] (from [23]) ImageNet 16.77
TRN [50] (from [19]) ImageNet 22.8
R(2+1)D [41] (from [1]) Kinetics 28.9
DiMoFs [1] Kinetics 31.4
P3D [31] (from [27]) ImageNet 32.4
C3D [39] (from [27]) ImageNet 34.5
Kanojia et al. [19] ImageNet 35.64
CorrNet [42] - 37.7
GST ImageNet 38.8
GSM ImageNet 40.27
Table 5: Comparison to state-of-the-art on Diving48.
Method Pre-train S1 S2
Verb Noun Action Verb Noun Action
TSN [43] ImageNet 45.68 36.8 19.86 34.89 21.82 10.11
TBN [20] ImageNet 60.87 42.93 30.31 49.61 25.68 16.80
LSTA [35] ImageNet 59.55 38.35 30.33 47.32 22.16 16.63
RU-LSTM [10] ImageNet 56.93 43.05 33.06 43.67 26.77 19.49
LFB [46] Kinetics 60.0 45 32.7 50.9 31.5 21.2
R(2+1)D [11] Sports-1M 59.6 43.7 31.0 47.2 28.7 18.3
R(2+1)D [11] External 65.2 45.1 34.5 58.4 36.9 26.1
GSM ImageNet 59.41 41.83 33.45 48.28 26.15 20.18
Table 6: Comparison to state-of-the-art on EPIC-Kitchens.

4.4 State-of-the-Art Comparison

Something-V1. The recognition performance obtained by GSM is compared with state-of-the-art approaches that just use RGB frames in Tab. 3. We also report the number of frames used by each approach during inference and the corresponding computational complexity in terms of FLOPs. The first block in the table lists the approaches that use 2D CNN and efficient 3D CNN implementation. The second block shows the approaches that use Full-3D CNNs. From the table, it can be seen that GSM results in an absolute gain of ( vs ) over the TSN baseline. GSM performs better than 3D CNNs or heavier backbones and also those approaches that use external data for pre-training, with considerably less number of FLOPs. GSM performs comparatively to the top performing method [28] with less number of FLOPs. It should be noted that the FLOPs of the architecture described in [28] is computed assuming a single clip of 16 frames. It can also be seen that by using InceptionV3, which is a larger backbone than BN-Inception, the performance of the proposed approach improves. Tab. 4 lists the action recognition accuracy, number of parameters and FLOPS obtained by ensembling the models presented in this work on Something-V1 dataset. The first and second blocks in the table list the accuracy obtained with individual models when evaluated using 1 and 2 clips, respectively. The third block shows the recognition accuracy with different ensemble models. Ensembling is done by combining GSM InceptionV3 models that are trained with different number of input frames. We average the prediction scores obtained from individual models to compute the performance of the ensemble. From the table, it can be seen that the accuracy is increasing as more models are being added. Using models with different number of input frames enables the ensemble to encode the video with different temporal resolutions. Such an ensemble has some analogy with SlowFast [7]. With an ensemble of models trained on 8, 12, 16 and 24 frames, we achieve a state-of-the-art recognition accuracy of 55.16%. Fig. 11-12 show ’visual explanations’ of how GSM has learnt discriminative regions for making a decision (appendix, Sec A).

(a) t-SNE plot of TSN features
(b) t-SNE plot of GSM features
(c) t-SNE of most improved classes
(d) Most improved classes
Figure 5: t-SNE plots of the output layer features preceding the final fully connected layers for (a) TSN with BN-Inception, and (b) same TSN but with GSM built-in as in Fig. 4. In these two plots the 10 action categories described in [15] are visualized. In (d) we list the action classes with the highest improvement over TSN baseline. X-axis shows the number of corrected samples for each class. Y-axis labels are in the format true label (GSM)/predicted label (TSN). In (c) we visualize the corresponding t-SNE plot.
(a) Folding something
(b) Unfolding something
(c) Plugging something into something
(d) Plugging something into something but pulling it …
(e) Taking one of many similar things on the table
(f) Putting something similar to other things that are …
(g) Putting something infront of something
(h) Removing something, revealing something behind
Figure 6: Sample frames from Something-V1 videos that belong to the most improved classes when GSM is added.

Diving48. Tab. 5 compares performance of GSM on Diving48 dataset with state-of-the-art approaches. We train the network using 16 frames and sample two clips during inference. We use InceptionV3 as the CNN backbone. In this dataset, the actions cannot be recognized from the scene context alone and require strong spatio-temporal reasoning. GSM achieves a recognition accuracy of , an improvement of over previous state-of-the-art  [27].

EPIC-Kitchens. In EPIC-Kitchens, the labels are provided as verb-noun pairs and the performance is evaluated on verb, noun and action recognition accuracies. For this dataset, we train GSM as a multi-task problem for verb, noun and action prediction. In the classification layers, we apply action scores as bias to the verb and nounclassifiers, as done in LSTA [35]. We use BN-Inception as the backbone CNN. The network is trained with 16 frames. Two clips consisting of 16 frames are sampled from each video during inference. We report the recognition accuracy obtained on the two standard test splits, S1 (seen) and S2 (unseen), in Tab. 6. The first block in the table shows the methods that use both RGB frames and optical flow as inputs while the second block lists the approaches that only use RGB images. From the table, it can be seen that GSM performs better than other approaches that use optical flow images for explicit motion encoding. The only two methods that beat GSM, R(2+1)D [11] and LFB [46], train two separate networks, one trained for verb and the other for noun classification, and leverage additional data for pre-training. GSM uses a single network for predicting all three labels from a video, thereby making it faster and more memory efficient. In fact, GSM performs better than R(2+1)D model pre-trained on Sports-1M dataset, suggesting that GSM can also improve its performance by pre-training on external data.

4.5 Discussion

Figure 7: Accuracy-vs-complexity of state-of-the-art on Something-V1, from Tab. 3. Size indicates number of parameters (M, in millions). GSM outperforms or competes in recognition performance with far less model complexity.

In Fig. 4(d), we show the top 10 action classes that improved the most by adding GSM to the CNN backbone of TSN. From the figure, it can be seen that the network has enhanced its ability to distinguish between action classes that are similar in appearance, such as Unfolding something and Folding something, Putting something infront of something and Removing something, revealing, something behind, etc. Sample frames from some of these most improved classes are shown in Fig. 6. From these frames, we can see that reversing the order of the frames changes the action and thus the orderless pooling present in TSN fails to identify the action. On the other hand, GSM is able to improve the recognition score on these classes, providing a strong spatio-temporal reasoning. In order to validate the temporal encoding capability of GSM, we evaluated its performance by applying the video frames in the reverse order. This resulted in a drastic degradation in the recognition performance from to . On the other hand, there was no change in the recognition performance of TSN when frames reversed in time were applied.

The t-SNE plots of the features from the final layer of the CNN corresponding to the 10 action groups described in [15] are shown in Fig. 4(a) and 4(b). Fig. 4(c) shows the t-SNE visualization of the most improved classes compared to TSN. We sample 1800 videos from the validation split for the t-SNE visualization. It can be seen that the features from GSM show a lower intra-class and higher inter-class variability compared to those from TSN.

We also analyzed the memory requirement and computational complexity of GSM and various state-of-the-art approaches. Fig. 7 shows the accuracy, parameter and complexity trade-off computed on the validation set of Somthing-V1 dataset. The graph plots accuracy vs GFLOPs and the area of the bubbles indicate the number of parameters present in each method. From the plot, it can be seen that GSM performs competitively to the state-of-the-art  [28] with less than one tenth the number of parameters and half the number of FLOPs.

5 Conclusion

We proposed Gate-Shift Module (GSM), a novel temporal interaction block that turns a 2D-CNN into a highly efficient spatio-temporal feature extractor. GSM introduces spatial gating to decide on exchanging information with neighboring frames. We performed an extensive evaluation to study its effectiveness in video action recognition, achieving state-of-the-art results on Something Something-V1 and Diving48 datasets, and obtaining competitive results on EPIC-Kitchens with far less model complexity. For example, when GSM is plugged into TSN, an absolute gain of +32% in recognition accuracy is obtained on Something Something-V1 dataset with just additional parameters and additional FLOPs. With an ensemble of models trained at different sampling rates, GSM achieve an an accuracy of  55%.


Appendix A Visualization

We show ‘visual explanations’ for the decisions made by GSM. We use the approach of saliency tubes [34] for generating the visualizations. In this approach, the frames and their corresponding regions that are used by the model for making a decision are visualized in a form of saliency map. Figs. 11 and 12 compare the saliency tubes generated by the TSN baseline and the proposed GSM approach on sample videos from the validation set of Something Something-V1 dataset. We use the models with BNInception backbone trained using 16 frames for generating the visualizations. Each column in the figures show the 16 frames that are applied as input to the respective networks with the saliency tubes overlaid on top. We show TSN on the left side and GSM on the right side. The classes that improved the most by plugging in GSM on TSN are chosen for visualization. These classes require strong temporal reasoning for understanding the action. From the figures, we can see that TSN focuses on the objects present in the video irrespective of where and when the action takes place, while GSM enables temporal reasoning by focusing on the active object(s) where and when an action is taking place. For example, in Fig. 10(a), an example from the class putting something in front of something, TSN focuses on the object that is present in the scene, the pen in the first few frames and the cup in the later frames. On the other hand, GSM makes the decision from the frames where the cup is introduced into the video. Similarly, in the example from the class taking one of many similar things on the table shown in Fig. 10(d), TSN is focusing on the object, the matchbox, in all the frames while GSM makes the decision based on those frames where the action is taking place.

Appendix B t-SNE

We first visualize the t-SNE plot of features for the models used in the ablation study, i.e., model with no GSM (Fig. 8(a)), model with 1 GSM (Fig. 8(b)), model with 5 GSM (Fig. 8(c)) and model with 10 GSM (Fig. 8(d)). All figures plot the features of the 10 action groups presented in [15]. From the figures, one can see that adding GSM into the CNN results in a reduction of intra-class variability and in an increase of inter-class variability. Fig. 10 shows the t-SNE plot of features from the last four Inception blocks of BN-Inception with 10 GSM. From the figure, we can see that the semantic separation increases as we move towards the top layers of the backbone.

Appendix C Architecture Details

We provide the details of the CNN architectures used in our GSM models.

c.1 BN-Inception

Tab. 7

shows the architecture of GSM BN-Inception. The Inception modules used are shown in Fig. 4 of the paper. The table also lists the output size after each layer.

Type Kernel size/ Output size
Max Pool
Max Pool
Inception-GSM 1 (Inc3a)
Inception-GSM 1 (Inc3b)
Inception-GSM 2 (Inc3c)
Inception-GSM 1 (Inc4a)
Inception-GSM 1 (Inc4b)
Inception-GSM 1 (Inc4c)
Inception-GSM 1 (Inc4d)
Inception-GSM 2 (Inc4e)
Inception-GSM 1 (Inc5a)
Inception-GSM 1 (Inc5b)
Avg Pool
Table 7: Gate-Shift BN-Inception Architecture. All convolution layers are followed by BN

layer and ReLU non-linearity.

is the number of classes in the dataset.

c.2 InceptionV3

The architecture of GSM InceptionV3 is shown in Tab. 8 along with the size of the outputs after each layer. We apply an input of size instead of the standard size of . This reduces the computational complexity without affecting the performance of the model. The Inception blocks with GSM used in the model are shown in Fig. 8.

Figure 8: Inception blocks with GSM used in the InceptionV3 architecture.
Type Kernel size/ Output size
Max Pool
Max Pool
Inception-GSM (Fig. 1a)
Inception-GSM (Fig. 1a)
Inception-GSM (Fig. 1c)
Inception-GSM (Fig. 1b)
Inception-GSM (Fig. 1b)
Inception-GSM (Fig. 1b)
Inception-GSM (Fig. 1b)
Inception-GSM (Fig. 1b)
Inception-GSM (Fig. 1d)
Inception-GSM (Fig. 1e)
Inception-GSM (Fig. 1e)
Avg Pool
Table 8: Gate-Shift InceptionV3 Architecture. All convolution layers are followed by BN layer and ReLU non-linearity. is the number of classes in the dataset.
(a) No GSM
(b) 1 GSM
(c) 5 GSM
(d) 10 GSM
Figure 9: t-SNE visualization of features from networks that use (a) No GSM, (b) 1 GSM, (c) 5 GSMs and (d) 10 GSMs.
(a) Inc4d
(b) Inc4e
(c) Inc5a
(d) Inc5b
Figure 10: t-SNE visualization of features from intermediate layer of GSM BN-Inception.

putting something in front of something


unfolding something


removing something, revealing something behind


taking one of many similar things on the table

Figure 11: Saliency tubes generated by TSN (left) and GSM (right) on sample videos taken from the validation set of Something Something-V1 dataset. Action labels are shown as text on columns.

plugging something into something


folding something


plugging something into something but pulling it right out as you remove your hand


putting something similar to other things that are already on the table

Figure 12: Saliency tubes generated by TSN (left) and GSM (right) on sample videos taken from the validation set of Something Something-V1 dataset. Action labels are shown as text on columns.


  • [1] G. Bertasius, C. Feichtenhofer, D. Tran, J. Shi, and L. Torresani. Learning Discriminative Motion Features Through Detection. arXiv preprint arXiv:1812.04172, 2018.
  • [2] J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proc. CVPR, 2017.
  • [3] Y. Chen, Y. Kalantidis, J. Li, S. Yan, and J. Feng. Multi-fiber networks for video recognition. In Proc. ECCV, 2018.
  • [4] D. Damen, H. Doughty, G. Maria Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray. Scaling egocentric vision: The epic-kitchens dataset. In Proc. ECCV, 2018.
  • [5] J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. In Proc. CVPR, 2015.
  • [6] L. Fan, W. Huang, C. Gan, S. Ermon, B. Gong, and J. Huang. End-to-end learning of motion representation for video understanding. In Proc. CVPR, 2018.
  • [7] C. Feichtenhofer, H. Fan, J. Malik, and K. He. SlowFast Networks for Video Recognition. In Proc. ICCV, 2019.
  • [8] C. Feichtenhofer, A. Pinz, and R. Wildes. Spatiotemporal residual networks for video action recognition. In Proc. NIPS, 2016.
  • [9] C. Feichtenhofer, A. Pinz, and A. Zisserman. Convolutional two-stream network fusion for video action recognition. In Proc. CVPR, 2016.
  • [10] A. Furnari and G. M. Farinella. What Would You Expect? Anticipating Egocentric Actions With Rolling-Unrolling LSTMs and Modality Attention. In Proc. ICCV, 2019.
  • [11] D. Ghadiyaram, D. Tran, and D. Mahajan. Large-scale weakly-supervised pre-training for video action recognition. In Proc. CVPR, 2019.
  • [12] R. Girdhar and D. Ramanan. Attentional pooling for action recognition. In Proc. NIPS, 2017.
  • [13] R. Girdhar, D. Ramanan, A. Gupta, J. Sivic, and B. Russell. ActionVLAD: Learning spatio-temporal aggregation for action classification. In Proc. CVPR, 2017.
  • [14] P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arXiv preprint arXiv:1706.02677, 2017.
  • [15] R. Goyal, S.E. Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, et al. The” Something Something” Video Database for Learning and Evaluating Visual Common Sense. In Proc. ICCV, 2017.
  • [16] K. Hara, H. Kataoka, and Y. Satoh. Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet? In Proc. CVPR, 2018.
  • [17] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proc. CVPR, 2016.
  • [18] B. Jiang, M. Wang, W. Gan, W. Wu, and J. Yan. STM: SpatioTemporal and Motion Encoding for Action Recognition. Proc. ICCV, 2019.
  • [19] G. Kanojia, S. Kumawat, and S. Raman. Attentive Spatio-Temporal Representation Learning for Diving Classification. In Proc. CVPRW, 2019.
  • [20] E. Kazakos, A. Nagrani, A. Zisserman, and D. Damen. EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition. In Proc. ICCV, 2019.
  • [21] M. Lee, S. Lee, S. Son, G. Park, and N. Kwak. Motion feature network: Fixed motion filter for action recognition. In Proc. ECCV, 2018.
  • [22] C. Li, Q. Zhong, D. Xie, and S. Pu. Collaborative Spatiotemporal Feature Learning for Video Action Recognition. In Proc. CVPR, 2019.
  • [23] Y. Li, Y. Li, and N. Vasconcelos. RESOUND: Towards action recognition without representation bias. In Proc. ECCV, 2018.
  • [24] Z. Li, K. Gavrilyuk, E. Gavves, M. Jain, and C. GM Snoek. VideoLSTM convolves, attends and flows for action recognition. Computer Vision and Image Understanding, 166:41–50, 2018.
  • [25] J. Lin, C. Gan, and S. Han. Temporal Shift Module for Efficient Video Understanding. In Proc. ICCV, 2019.
  • [26] I. Loshchilov and F. Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. In Proc. ICLR, 2017.
  • [27] C. Luo and A. Yuille. Grouped Spatial-Temporal Aggregation for Efficient Action Recognition. In Proc. ICCV, 2019.
  • [28] B. Martinez, D. Modolo, Y. Xiong, and J. Tighe. Action recognition with spatial-temporal discriminative filter banks. In Proc. ICCV, 2019.
  • [29] J.Y. Ng and L.S. Davis. Temporal difference networks for video action recognition. In Proc. IEEE Winter Conference on Applications of Computer Vision (WACV), 2018.
  • [30] A. J. Piergiovanni and M. S. Ryoo. Representation flow for action recognition. In Proc. CVPR, 2019.
  • [31] Z. Qiu, T. Yao, and T. Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In Proc. ICCV, 2017.
  • [32] Z. Shou, X. Lin, Y. Kalantidis, L. Sevilla-Lara, M. Rohrbach, S. Chang, and Z. Yan. DMC-Net: Generating discriminative motion cues for fast compressed video action recognition. In Proc. CVPR, 2019.
  • [33] K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In Proc. NIPS, 2014.
  • [34] A. Stergiou, G. Kapidis, G. Kalliatakis, C. Chrysoulas, R. Veltkamp, and R. Poppe. Saliency Tubes: Visual Explanations for Spatio-Temporal Convolutions. In Proc. ICIP, 2019.
  • [35] S. Sudhakaran, S. Escalera, and O. Lanz. LSTA: Long Short-Term Attention for Egocentric Action Recognition. In Proc. CVPR, 2019.
  • [36] L. Sun, K. Jia, D. Yeung, and B. E. Shi. Human action recognition using factorized spatio-temporal convolutional networks. In Proc. ICCV, 2015.
  • [37] S. Sun, Z. Kuang, L. Sheng, W. Ouyang, and W. Zhang. Optical flow guided feature: a fast and robust motion representation for video action recognition. In Proc. CVPR, 2018.
  • [38] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proc. CVPR, 2016.
  • [39] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proc. ICCV, 2015.
  • [40] D. Tran, H. Wang, L. Torresani, and M. Feiszli. Video Classification With Channel-Separated Convolutional Networks. In Proc. ICCV, 2019.
  • [41] D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proc. CVPR, 2018.
  • [42] H. Wang, D. Tran, L. Torresani, and M. Feiszli. Video Modeling with Correlation Networks. arXiv preprint arXiv:1906.03349, 2019.
  • [43] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In Proc. ECCV, 2016.
  • [44] X. Wang, R. Girshick, A. Gupta, and K. He. Non-local neural networks. In Proc. CVPR, 2018.
  • [45] X. Wang and A. Gupta. Videos as space-time region graphs. In Proc. ECCV, 2018.
  • [46] C. Wu, C. Feichtenhofer, H. Fan, K. He, P. Krahenbuhl, and R. Girshick. Long-term feature banks for detailed video understanding. In Proc. CVPR, 2019.
  • [47] S. Xie, C. Sun, J. Huang, Z. Tu, and K. Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Proc. ECCV, 2018.
  • [48] C. Zach, T. Pock, and H. Bischof. A duality based approach for realtime TV-L 1 optical flow. In

    Joint pattern recognition symposium

    , pages 214–223, 2007.
  • [49] Y. Zhao, Y. Xiong, and D. Lin. Trajectory Convolution for Action Recognition. In Proc. NIPS, 2018.
  • [50] B. Zhou, A. Andonian, A. Oliva, and A. Torralba. Temporal relational reasoning in videos. In Proc. ECCV, 2018.
  • [51] Y. Zhou, X. Sun, Z. Zha, and W. Zeng. MiCT: Mixed 3d/2d convolutional tube for human action recognition. In Proc. CVPR, 2018.
  • [52] X. Zhu, C. Xu, L. Hui, C. Lu, and D. Tao. Approximated Bilinear Modules for Temporal Modeling. In Proc. ICCV, 2019.
  • [53] M. Zolfaghari, K. Singh, and T. Brox. ECO: Efficient Convolutional Network for Online Video Understanding. In Proc. ECCV, 2018.