The most distinctive feature of videos, from those of images, is motion. In order to grasp a full understanding of a video, we need to analyze its motion patterns as well as the appearance of objects and scenes in the video [38, 31, 20, 27]
. With significant progress of neural networks on the image domain, convolutional neural networks (CNNs) have been widely used to learn appearance features from video frames[31, 40, 35, 5] and recently extended to learn temporal features using spatio-temporal convolution across multiple frames [2, 35]. The results, however, have shown that spatio-temporal convolution alone is not enough for learning motion patterns; convolution is effective in capturing translation-equivariant patterns but not in modeling relative movement of objects [46, 39]. As a result, most state-of-the-art methods still incorporate explicit motion features, i.e., dense optical flows, extracted by an external off-the-shelf methods [31, 2, 43, 37, 21]. This causes a major computational bottleneck in video-processing models for two reasons. First, calculating optical flows frame-by-frame is a time-consuming process; obtaining optical flows of a video is typically an order of magnitude slower than feed-forwarding the video through a deep neural network. Second, processing optical flows often requires a separate stream in the model to learn motion representations , which results in doubling the number of parameters and the computational cost. To address these issues, several methods have attempted to internalize motion modeling [34, 20, 7, 27]. They, however, all either impose a heavy computation on their architectures [7, 27] or underperform other methods using external optical flows [34, 20].
To tackle the limitation of the existing methods, we propose an end-to-end trainable block, dubbed the MotionSqueeze
(MS) module, for effective motion estimation. Inserted in the middle of any neural network for video understanding, it learns to establish correspondences across adjacent frames efficiently and convert them into effective motion features. The resultant motion features are readily fed to the next downstream layer and used for final prediction. To validate the proposed MS module, we develop a video classification architecture, dubbed the MotionSqueeze network (MSNet), that is equipped with the MS module. In comparison with recent methods, shown in Figure1, the proposed method provides the best trade-off in terms of accuracy, computational cost, and model size in video understanding.
2 Related work
Video classification architectures. One of the main problems in video understanding is to categorize videos given a set of pre-defined target classes. Early methods based on deep neural networks have focused on learning spatio-temporal or motion features. Tran et al.  propose a 3D CNN (C3D) to learn spatio-temporal features while Simonyan and Zisserman  employ an independent temporal stream to learn motion features from precomputed optical flows. Carreira and Zisserman  design two-stream 3D CNNs (two-stream I3D) by integrating two former methods, and achieve the state-of-the-art performance at that time. As the two-stream 3D CNNs are powerful but computationally demanding, subsequent work has attempted to improve the efficiency. Tran et al. and Xie et al. propose to decompose 3D convolutional filters into 2D spatial and 1D temporal filters. Chen et al.  adopt group convolution techniques while Zolfaghari et al.  propose to study mixed 2D and 3D networks with the frame sampling method of temporal segment networks (TSN) . Tran et al.  analyze the effect of 3D group convolutional networks and propose the channel-separated convolutional network (CSN). Lin et al.  propose the temporal shift module (TSM) that simulates 3D convolution using 2D convolution with a part of input feature channels shifted along the temporal axis. It enables 2D convolution networks to achieve a comparable classification accuracy to 3D CNNs. Unlike these approaches, we focus on efficient learning of motion features.
Learning motions in a video. While two-stream-based architectures [31, 9, 8, 30] have demonstrated the effectiveness of pre-computed optical flows, the use of optical flows typically degrades the efficiency of video processing. To address the issue, Ng et al.  use a multi-task learning of both optical flow estimation and action classification and Stroud et al.  propose to distill motion features from pre-trained two-stream 3D CNNs. These methods do not use pre-computed optical flows during inference, but still need them at the training phase. Other methods design network architectures that learn motions internally without external optical flows [34, 16, 20, 7, 27]. Sun et al.  compute spatial and temporal gradients between appearance features to learn motion features. Lee et al.  and Jiang et al.  propose a convolutional module to extract motion features by spatial shift and subtraction operation between appearance features. Despite their computational efficiency, they do not reach the classification accuracy of two-stream networks . Fan et al.  implement the optimization process of TV-L1  as iterative neural layers, and design an end-to-end trainable architecture (TVNet). Piergiovanni and Ryoo  extend the idea of TVNet by calculating channel-wise flows of feature maps at the intermediate layers of the CNN. These variational methods achieve a good performance, but require a high computational cost due to iterative neural layers. In contrast, our method learns to extract effective motion features with a marginal increase of computation.
estimate optical flows using a convolutional neural network, which construct a correlation tensor from feature maps and regresses displacements from it. Sunet al.  use a stack of correlation layers for coarse-to-fine optical flow estimation. While these methods require dense ground-truth optical flows in training, the structure of correlation computation and subsequent displacement estimation is widely adopted in other correspondence problems with different levels of supervision. For example, recent methods for semantic correspondence, i.e., matching images with intra-class variation, typically follow a similar pipeline to learn geometric transformation between images in a more weakly-supervised regime [11, 28, 19, 24]. In this work, motivated by this line of research, we develop a motion feature module that does not require any correspondence supervision for learning.
Similarly to our work, a few recent methods [22, 45] have attempted to incorporate correspondence information for video understanding. Zhao et al.  use correlation information between feature maps of consecutive frames to replace optical flows. The size of their full model, however, is comparable to the two-stream networks . Liu et al.  propose the correspondences proposal (CP) module to learn correspondences in a video. Unlike ours, they focus on analyzing spatio-temporal relationship within the whole video, rather than motion, and the model is not fully differentiable and thus less effective in learning. In contrast, we introduce a fully-differentiable motion feature module that can be inserted in the middle of any neural network for video understanding.
The main contribution of this work is three-fold.
We propose an end-to-end trainable, model-agnostic, and lightweight module for motion feature extraction.
We develop an efficient video recognition architecture that is equipped with the proposed motion module.
We demonstrate the effectiveness of our method on four different benchmark datasets and achieve the state-of-the-art on Something-Something-V1&V2.
3 Proposed approach
The overall architecture for video understanding is illustrated in Figure 2. Let us assume a neural network that takes a video of frames as input and predicts the category of the video as output, where convolutional layers are used to transform input frames into frame-wise appearance features. The proposed motion feature module, dubbed MotionSqueeze (MS) module, is inserted to produce frame-wise motion features using pairs of adjacent appearance features. The resultant motion features are added to the appearance features for final prediction. In this section, we first explain the MS module, and describe the details of our network architecture for video understanding.
3.1 MotionSqueeze (MS) module
The MS module is a learnable motion feature extractor, which can replace the use of explicit optical flows for video understanding. As described in Figure 3, given two feature maps from adjacent frames, it learns to extract effective motion features in three steps: correlation computation, displacement estimation, and feature transformation.
Correlation computation. Let us denote the two adjacent input feature maps by and , each of which is 3D tensors of size . The spatial resolution is and the dimensional features on spatial position by . A correlation score of position with respect to displacement is defined as
where denotes dot product. For efficiency, we compute the correlation scores of position only in its neighborhood of size by restricting a maximum displacement: . For frame, a resultant correlation tensor is of size . The cost of computing the correlation tensor is equivalent to that of convolutions with kernels; the correlation computation can be implemented as 2D convolutions on feature map using feature map as kernels. The total FLOPs in a single video amounts to . We apply a convolution layer before computing correlations, which learns to weight informative feature channels for learning visual correspondences. In practice, we set the neighborhood given the spatial resolution and apply an layer with channels. For correlation computation, we adopt C++/Cuda implemented version of correlation layer in FlowNet .
Displacement estimation. From the correlation tensor , we estimate a displacement field for motion information. A straightforward but non-differentiable method would be to take the best matching displacement for position by . To make the operation differentiable, we can use a weighted average of displacements using softmax, called soft-argmax [13, 19], which is defined as
This method, however, is sensitive to noisy outliers in the correlation tensor since it is influenced by all correlation values. We thus use thekernel-soft-argmax  that suppresses such outliers by masking a 2D Gaussian kernel on the correlation values; the kernel is centered on each target position so that the estimation is more influenced by closer neighbors. Our kernel-soft-argmax for displacement estimation is defined as
is the Gaussian kernel and we empirically set the standard deviationto 5. is a temperature factor adjusting the softmax distribution; as decreases, softmax approaches argmax. We set in our experiments.
In addition to the estimated displacement map, we use a confidence map of correlation as auxiliary motion information, which is obtained by pooling the highest correlation on each position :
The confidence map may be useful for identifying displacement outliers and learning informative motion features.
We concatenate the (2-channel) displacement map and the (1-channel) confidence map into a displacement tensor of size for the next step of motion feature transformation. An example of them is visualized in Figure 4.
Feature transformation. We convert the displacement tensor to an effective motion feature that is readily incorporated into downstream layers. The tensor is fed to four depth-wise separable convolution  layers, one layer followed by three layers, and transformed into a motion feature with the same number of channels as that of the original input . The depth-wise separable convolution approximates 2D convolution with a significantly less computational cost [4, 29, 36]
. Note that all depth-wise and point-wise convolution layers are followed by batch normalization
and ReLU. As in the temporal stream layers of , this feature transformation process is designed to learn task-specific motion features with convolution layers by interpreting the semantics of displacement and confidence. As illustrated in Figure 2, the MS module generates motion feature using two adjacent appearance features and and then add it to the input of the next layer. Given
frames, we simply pad the final motion featurewith by setting .
3.2 MotionSqueeze network (MSNet).
The MS module can be inserted into any video understanding architecture to improve the performance by motion feature modeling. In this work, we introduce standard convolutional neural networks (CNNs) with the MS module, dubbed MSNet
, for video classification. We adopt the ImageNet-pretrained ResNet as the CNN backbone and insert TSM  for each residual block of the ResNet. TSM enables 2D convolution to obtain the effect of 3D convolution by shifting a part of input feature channels along the temporal axis before the convolution operation. Following the default setting in , we shift of the input features channels forward and another of the channels backward in each TSM.
The overall architecture of the proposed model is shown in Figure 2; a single MS module is inserted after the third stage of the ResNet. We fuse the motion feature into the appearance feature by element-wise addition:
In section 4.5, we extensively evaluate different fusion methods, e.g., concatenation and multiplication, and show that additive fusion is better than the others. After fusing both features, the combined feature is passed through the next downstream layers. The network outputs over
frames are temporally averaged to produce a final output and the cross-entropy with softmax is used as a loss function for training. By default setting, MSNet learns both appearance and motion features jointly in a single network at the cost of only 2.5% and 1.2% increase in FLOPs and the number of parameters, respectively.
Something-Something V1&V2  are trimmed video datasets for human action classification. Both datasets consist of 174 classes with 108,499 and 220,847 videos in total, respectively. Each video contains one action and the duration spans from 2 to 6 seconds. Something-Something V1&V2 are motion-oriented datasets where temporal relationships are more salient than in others.
Kinetics  is a popular large-scale video dataset, consisting of 400 classes with over 250,000 videos. Each video lasts around 10 seconds with a single action.
HMDB51  contains 51 classes with 6,766 videos. Kinetics and HMDB-51 focus more on appearance information rather than motion.
4.2 Implementation details
Clip sampling. In both training and testing, instead of an entire video, we use a clip of frames that are sampled from the video. We use the segment-based sampling method  for the Something-Something V1&V2 while adopting the dense frame sampling method  for Kinetics and HMDB-51.
Training. For each video, we sample a clip of or frames, resize them into images, and crop images from the resized images 
. The minibatch SGD with Nestrov momentum is used for optimization, and the batch size is set to 48. We use scale jittering for data augmentation. For the Something-Something V1&V2, we set the training epochs to 40 and the initial learning rate to 0.01; the learning rate is decayed by 1/10 afterand epochs. For Kinetics, we set the training epochs to 80 and the initial learning rate to 0.01; the learning rate is decayed by 1/10 after 40 and 60 epochs. In training our model on HMDB-51, we fine-tune the Kinetics-pretrained model as in [37, 21]. We set the training epochs to 35 and the initial learning rate to 0.001; the learning rate is decayed by 1/10 after and epochs.
Inference. Given a video, we sample a clip and test its center crop. For Something-Something V1&V2, we evaluate both the single clip prediction and the average prediction of 10 randomly-sampled clips. For Kinetics and HMDB-51, we evaluate the average prediction of uniformly-sampled 10 clips from each video.
|TRN Two-stream ||✓||8+8||16GN/A||18.3M||42.0||-||55.5||-|
|ECO Two-stream ||✓||92+92||N/A1||300M||49.5||-||-||-|
|I3D from ||32||153G2||28.0M||41.6||72.2||-||-|
|NL-I3D from ||32||168G2||35.3M||44.4||76.0||-||-|
|TSM Two-stream ||✓||16+16||129G1||48.6M||52.6||81.9||65.0||89.4|
|TSM Two-stream ||✓||16+16||129G6||48.6M||-||-||66.0||90.5|
4.3 Comparison with state-of-the-art methods
Table 1 summarizes the results on Something-Something V1&V2. Each section of the table contains results of 2D CNN methods [20, 22, 40, 47], 3D CNN methods [16, 23, 42, 43, 48], ResNet with TSM (TSM ResNet) , and the proposed method, respectively. Most of the results are copied from the corresponding papers, except for TSM ResNet; we evaluate the official pre-trained model of TSM ResNet using a single center-cropped clip per video in terms of top-1 and top-5 accuracies. Our method, which uses TSM ResNet as a backbone, achieves 50.9% and 63.0% on Something-Something V1 and V2 at top-1 accuracy, respectively, which outperforms most of 2D CNN and 3D CNN methods, while using a single clip with 8 input frames only. Compared to the TSM ResNet baseline, our method obtains a significant gain of about 5.3% points and 4.2% points at top-1 accuracy at the cost of only 2.5% and 1.2% increase in FLOPs and parameters, respectively. When using 16 frames, our method further improves achieving 52.1% and 64.7% at top-1 accuracy, respectively. Following the evaluation procedure of two-stream networks, we evaluate the ensemble model (MSNet-R50) by averaging prediction scores of the 8-frame and 16-frame models. With the same number of clips for evaluation, it achieves top-1 accuracy 1.8% points and 1.6% points higher than TSM two-stream networks with 22% less computation, even no optical flow needed. Our 10-clip model achieves 55.1% and 67.1% at top-1 accuracy on Something-Something V1 and V2, respectively, which is the state-of-the-art on both of the datasets. As shown in Figure 1, our model provides the best trade-off in terms of accuracy, FLOPs, and the number of parameters.
|ResNet-50 from ||32||132G25||22.8||61.3||59.4|
|MFNet from ||10||80G10||-||-||56.8|
|Rep-flow (ResNet-50) ||32||132G25||3.7||68.5||76.4|
|Rep-flow (R(2+1)D) ||32||152G25||2.0||75.5||77.1|
|ResNet-50 Two-stream from ||✓||32+32||264G25||0.2||64.5||66.6|
|R(2+1)D Two-stream ||✓||32+32||304G115||0.2||73.9||78.7|
|OFF(RGB+Flow+RGB Diff) ||✓||1+5+5||N/A25||-||-||74.2|
4.4 Comparison with other motion representation methods
Table 2 summarizes comparative results with other motion representation methods [20, 34, 7, 27, 16] based on RGB frames. The comparison is done on Kinetics and HMDB51 since the previous methods commonly report their results on them. Each section of the table contains results of conventional 2D and 3D CNNs, motion representation methods [20, 34, 7, 27, 16], two-stream CNNs with optical flows [12, 37], and the proposed method, respectively. OFF, MFNet, and STM [20, 34, 16] use a sub-network or lightweight modules to calculate temporal gradients of frame-wise feature maps. TVNet  and Rep-flow  internalize iterative TV-L1 flow operations in their networks. As shown in Table 2, the proposed model using 16 frames outperforms all the other conventional CNNs and the motion representation methods [12, 2, 37, 20, 34, 7, 27, 16], while being competitive with the R(2+1)D two-stream  that uses pre-computed optical flows. Furthermore, our model is highly efficient than all the other methods in terms of FLOPs, clips, and the number of frames.
Run-time. We also evaluate in Table 2 the inference speeds of several models to demonstrate the efficiency of our method. All the run-times reported are measured on a single GTX Titan Xp GPU, ignoring the time of data loading. For this experiment, we set the spatial size of the input to and the batch size to 1. The official codes are used for ResNet, TSM ResNet, and Rep-flow [12, 21, 27] except for R(2+1)D we implemented. In evaluating Rep-flow , we use 20 iterations for optimization as in the original paper. The speed of the two-stream networks [31, 37] includes computation time for TV-L1 method on the GPU. The run-time results clearly show the cost of iterative optimizations used in two-stream networks and Rep-flow. In contrast, our model using 16 frames is about 160 faster than the two-stream networks. Compared to Rep-flow ResNet-50, our method performs about 4 faster due to the absence of the iterative optimization process in Rep-flow.
4.5 Ablation studies
We conduct ablation studies of the proposed method on Something-Something V1  dataset. We use ImageNet pre-trained TSM ResNet-18 as a default backbone and use 8 input frames for all experiments in this section.
Displacement estimation in MS module. In Table 5, we experiment with different variants of the displacement tensor in the MS module. We first compare soft-argmax (‘S’) and kernel-soft-argmax (‘KS’) for displacement estimation. As shown in the upper part of Table 5. the kernel-soft-argmax outperforms the soft-argmax, showing the noise reduction effect of Gaussian kernel. In the lower part of Table 5, we evaluate the effect of additional features: confidence maps (‘CM’) and backward displacement tensor (‘BD’). The backward displacement tensor is estimated from to . We concatenate the forward and backward displacement tensors, and then pass them to the feature transformation layers. We obtain 0.9% points gain by appending the confidence map to the displacement tensor. Furthermore, by adding backward displacement we obtain another 0.5% points gain at top-1 accuracy, indicating that forward and backward displacement maps complement each other to enrich motion information. We use the kernel-soft-argmax with the confidence map (‘KS + CM’) as a default method for all other experiments.
|KS + CM||14.9G||45.5||76.5|
|KS + CM + BD||15.1G||46.0||76.7|
Size of matching region. In Figure 4, we evaluate performance varying the spatial size of matching regions of the MS module. Even with a small matching region , it provides a noticeable performance gain of over points to the baseline. The performance tends to increase as the matching region becomes larger due to the larger displacement it can handle between frames. The performance is saturated after .
Position of MS module. In Table 5, we evaluate different positions of the MS module. We denote that by the -th stage of the ResNet. For each stage, it is inserted right after its final residual block. The result shows that while the MS module is beneficial in most cases, both accuracy and efficiency gains depend on the position of the module. It performs the best at ; appearance features from are not strong enough for accurate feature matching while spatial resolutions of appearance features from and are not high enough. The position of the module also affects FLOPs; the computational cost quadratically increases with spatial resolution due to convolution layers of the feature transformation. When inserting multiple MS modules () at the backbone, it marginally improves top-1 accuracy as 0.2% points. Multiple modules appear to generate similar motion information even in different levels of features.
Fusing strategy of MS module. In Table 5, we evaluate different fusion strategies for the MS module; ‘MS only’, ‘multiply’, ‘concat’, and ‘add’. In the case of ‘MS only’, we only pass into downstream layers without . We apply element-wise multiplication and element-wise addition, respectively, for ‘multiply’ and ‘add’. In the case of ‘concat’, we concatenate and , whose channel size is transformed to via an convolution layer. ‘MS only’ is less accurate than the baseline because visual semantic information is discarded. While both ‘multiply’ and ‘concat’ clearly improve the accuracy, ‘add’ achieves the best performance with 45.5% at top-1 accuracy. We find that additive fusion is the most effective and stable in amplifying appearance features of moving objects.
Effect of MS module on different backbones. In Figure 5, we also evaluate the effect of the MS module on ResNet-18, MobileNet-V2, and I3D. We insert one MS module where the spatial resolution of the feature map remains the same. For ResNet-18 and MobileNet-V2, we finetune models pre-trained on ImageNet. We train I3D from scratch. Our MS module benefits both 2D CNNs and 3D CNNs to obtain higher accuracy. The module significantly improves ResNet-18 and MobileNet-V2 by 21.3% and 19.2% points, respectively, in top-1 accuracy. Since 2D CNNs do not use any spatio-temporal features, it obtains significantly higher gain from the MS module. The MS module also improves I3D and TSM ResNet-18 by 2.4% and 4.0% points, respectively, in top-1 accuracy. The gain on 3D CNNs, although relatively small, verifies that the motion features by the MS module are complementary even to the spatio-temporal features; the MS module learns explicit motion information across adjacent frames whereas TSM covers long-term temporal length using (pseudo-)temporal convolutions.
Comparison with two-stream networks. In Table 4.5, we compare the proposed method with variants of TSM two-stream networks  that use TV-L1 optical flows . We denote the two-stream networks by Two-stream where , and indicate the number of frames, optical flows, and their stacking size, respectively. For each frame, the two-stream networks use stacked optical flows, which are extracted using the subsequent frames in the original video. Note that those frames for optical flow extraction are not used in our method (MSNet). The second row of Table 4.5, Two-stream, shows the performance of standard TSM two-stream networks that use 5 stacked optical flows for the temporal stream. Using the multiple optical flows for each frame outperforms our model in terms of accuracy but requires substantially larger FLOPs as well as an additional computation for calculating optical flows. For a fair comparison, we report the performance of the two-stream networks, Two-stream, that do not stack multiple optical flows. Our model outperforms the two-stream networks by 0.8% points at top-1 accuracy, with about two times fewer FLOPs. Note that both Two-stream and Two-stream use optical flows obtained from the original video with a higher frame rate than the input video clip (sampled frames); our method (MSNet) observes the input video clip only. We thus evaluate other two-stream networks, Two-stream, that uses low-fps optical flows as input; we sample a sequence of frames in 3 fps from the original video and extract TV-L1 optical flows using the sequence. As shown in Table 4.5, the top-1 accuracy gap between ours and the two-stream network increases to 1.4% points. The result implies that given low-fps videos, our method may further improve over the two-stream networks.
In Figure 6, we present visualization results on Something-Something V1 and Kinetics datasets. They show that our MS module effectively learns to estimate motion without any direct supervision used in training. The first row of each subfigure shows 6 uniformly sampled frames from a video. The second and third rows show color-coded displacement maps  and confidence maps, respectively; we apply min-max normalization on the confidence map. The resolution of all the displacement and confidence maps is set to 5656 for better visualization. As shown in the figures, the MS module captures reliable displacements in most cases: horizontal and vertical movements (Figure 5(a), 5(c), 5(d)), rotational movements (Figure 5(b)), and non-severe deformation (Figure 5(a), 5(d)). See the supplementary material for additional details and results. We will make our code and data available online.
We have presented an efficient yet effective motion feature block, the MS module, that learns to generate motion features on the fly for video understanding. The MS module can be readily inserted into any existing video architectures and trained by backpropagation. The ablation studies on the module demonstrate the effectiveness of the proposed method in terms of accuracy, computational cost, and model size. Our method outperforms existing state-of-the-art methods on Something-Something-V1&V2 for video classification with only a small amount of additional cost.
Acknowledgements. This work is supported by Samsung Advanced Institute of Technology (SAIT), and also by Basic Science Research Program (NRF-2017R1E1A1A010 77999, NRF-2018R1C1B6001223) and Next-Generation Information Computing Development Program (NRF-2017M3C4A7069369) through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT.
A database and evaluation methodology for optical flow.
International Journal of Computer Vision (IJCV)92 (1), pp. 1–31. Cited by: §4.6, §4.
Quo vadis, action recognition? a new model and the kinetics dataset.
Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: Figure 1, §1, §2, §3, §3, 6(a), 6(b), §4.2, §4.4.
-  (2018) Multi-fiber networks for video recognition. In Proc. European Conference on Computer Vision (ECCV), Cited by: §2.
Xception: deep learning with depthwise separable convolutions. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §3.1.
-  (2015) Long-term recurrent convolutional networks for visual recognition and description. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1.
-  (2015) Flownet: learning optical flow with convolutional networks. In Proc. IEEE International Conference on Computer Vision (ICCV), Cited by: §2, §3.1.
-  (2018) End-to-end learning of motion representation for video understanding. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1, §2, §4.4, Table 2.
-  (2016) Spatiotemporal residual networks for video action recognition. In Proc. Neural Information Processing Systems (NeurIPS), Cited by: §2.
-  (2016) Convolutional two-stream network fusion for video action recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.
-  (2017) The” something something” video database for learning and evaluating visual common sense.. In Proc. IEEE International Conference on Computer Vision (ICCV), Cited by: Figure 1, §1, §2, Figure 9, §4.1, §4.5, §4.
-  (2017) Scnet: learning semantic correspondence. In Proc. IEEE International Conference on Computer Vision (ICCV), Cited by: §2.
-  (2016) Deep residual learning for image recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §3.2, §3, §3, §4.4, §4.4.
Improving landmark localization with semi-supervised learning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §3.1.
-  (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861. Cited by: §3.1, Figure 8.
-  (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167. Cited by: §3.1, Figure 7.
-  (2019) STM: spatiotemporal and motion encoding for action recognition. In Proc. IEEE International Conference on Computer Vision (ICCV), Cited by: §2, §4.3, §4.4, Table 1, Table 2.
-  (2017) The kinetics human action video dataset. arXiv preprint arXiv:1705.06950. Cited by: Figure 10, §4.1, §4.
-  (2011) HMDB: a large video database for human motion recognition. In Proc. IEEE International Conference on Computer Vision (ICCV), Cited by: §4.1.
-  (2019) SFNet: learning object-aware semantic correspondence. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2, §3.1.
-  (2018) Motion feature network: fixed motion filter for action recognition. In Proc. European Conference on Computer Vision (ECCV), Cited by: §1, §2, §4.3, §4.4, Table 1.
-  (2019) Tsm: temporal shift module for efficient video understanding. In Proc. IEEE International Conference on Computer Vision (ICCV), Cited by: Figure 1, §1, §2, §3.2, §3, §3, §4.2, §4.3, §4.4, Table 1.
-  (2019) Learning video representations from correspondence proposals. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: Table 8, §2, §2, §2, §2, §4.3, Table 1.
-  (2019) Action recognition with spatial-temporal discriminative filter banks. In Proc. IEEE International Conference on Computer Vision (ICCV), Cited by: §4.3, Table 1.
-  (2019) Hyperpixel flow: semantic correspondence with multi-layer neural features. In Proc. IEEE International Conference on Computer Vision (ICCV), Cited by: §2.
Rectified linear units improve restricted boltzmann machines.
Proc. International Conference on Machine Learning (ICML), Cited by: §3.1.
-  (2018) Actionflownet: learning motion representation for action recognition. In Proc. Winter Conference on Applications of Computer Vision (WACV), Cited by: §2.
-  (2019) Representation flow for action recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1, §2, §4.4, §4.4, Table 2.
-  (2017) Convolutional neural network architecture for geometric matching. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.
-  (2018) Mobilenetv2: inverted residuals and linear bottlenecks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §3.1, §3, §3, Figure 8.
-  (2018) On the integration of optical flow and action recognition. In Proc. German Conference on Pattern Recognition (GCPR), Cited by: §2.
-  (2014) Two-stream convolutional networks for action recognition in videos. In Proc. Neural Information Processing Systems (NeurIPS), Cited by: §1, §2, §2, §2, §3.1, §4.4, §4.5.
-  (2020) D3d: distilled 3d networks for video action recognition. In Proc. Winter Conference on Applications of Computer Vision (WACV), Cited by: §2.
-  (2018) PWC-net: cnns for optical flow using pyramid, warping, and cost volume. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.
-  (2018) Optical flow guided feature: a fast and robust motion representation for video action recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1, §2, §4.4, Table 2.
-  (2015) Learning spatiotemporal features with 3d convolutional networks. In Proc. IEEE International Conference on Computer Vision (ICCV), Cited by: §1, §2.
-  (2019) Video classification with channel-separated convolutional networks. In Proc. IEEE International Conference on Computer Vision (ICCV), Cited by: §2, §3.1.
-  (2018) A closer look at spatiotemporal convolutions for action recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1, §2, §4.2, §4.4, §4.4, Table 2.
-  (2011) Action recognition by dense trajectories. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1.
-  (2015) Action recognition with trajectory-pooled deep-convolutional descriptors. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1.
-  (2016) Temporal segment networks: towards good practices for deep action recognition. In Proc. European Conference on Computer Vision (ECCV), Cited by: §1, §2, §4.2, §4.3, Table 1.
-  (2018) Non-local neural networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: Figure 1, §2.
-  (2018) Videos as space-time region graphs. In Proc. European Conference on Computer Vision (ECCV), pp. 399–417. Cited by: Figure 1, §4.3, Table 1.
-  (2018) Rethinking spatiotemporal feature learning: speed-accuracy trade-offs in video classification. In Proc. European Conference on Computer Vision (ECCV), Cited by: §1, §2, §4.3, Table 1.
-  (2007) A duality based approach for realtime tv-l 1 optical flow. Pattern Recognition, pp. 214–223. Cited by: §2, §4.5.
-  (2018) Recognize actions by disentangling components of dynamics. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.
-  (2018) Trajectory convolution for action recognition. In Proc. Neural Information Processing Systems (NeurIPS), Cited by: §1.
-  (2018) Temporal relational reasoning in videos. In Proc. European Conference on Computer Vision (ECCV), Cited by: Figure 1, §4.3, Table 1.
-  (2018) Eco: efficient convolutional network for online video understanding. In Proc. European Conference on Computer Vision (ECCV), Cited by: Figure 1, §2, §4.2, §4.3, Table 1.
1 Effects of depth-wise separable (DWS) convolutions
We use DWS convolutions rather than standard convolutions to build the feature transformation (FT) layers deeper and wider while saving computational cost. Table 7 shows the results of different forms of FT layers on Something-Something V1  . The accuracy increases as the FT layers become deeper and have wider receptive fields, and the DWS convolutions show the best accuracy-FLOPs tradeoff.
2 Comparison with the CP module 
As we mentioned in the main paper, the CP module is the one of the most relevant work to our method in the sense that it leverage correspondences between input video frames. Here we provide more detailed comparisons to it.
Difference in motivation and design. Unlike our MS module, which focuses on extracting effective motion features across consecutive frames, the CP module  is designed to capture long-term spatio-temporal relationship within an input video  by computing a non-local correlation tensor across all frames. The CP module selects most likely corresponding features in the correlation tensor with an ‘ top-’ operation, and the operation thus makes the correlation tensor non-differentiable.
Performance comparison. We have already shown in Table 1 of the main paper that the result of our method is better than that of the CP module (from the original paper ) on Something-Something V2 
. The comparison, however, may not be totally fair in the sense that the backbone and the other experimental settings are not the same. For an apples-to-apples comparison between the MS module and the CP module, we conduct an additional experiment using the same backbone and setup. We re-implement the CP module in Pytorch based on the official Tensorflow code111https://github.com/xingyul/cpnet. As a baseline network, we use ImageNet pre-trained TSM ResNet-18 using 8 input frames. Either MS or CP module is inserted after the third stage of the network. Table 8 summarizes the comparative results of the MS module and the CP module on Something-Something V1 . The CP module is effective for improving accuracy while consuming almost 6G FLOPs more than the baseline; the computational cost of the non-local correlation tensor is quadratic to the number of input frames. In contrast, the MS module performs 0.9% points and 0.8% points higher at top-1 and top-5 accuracy, respectively, while consuming 26% less FLOPs, compared to the CP module.
3 Backbone architectures in experiments
In our main paper, we evaluate the effect of the MS module on different backbone architectures: ResNet , TSM ResNet , MobileNet-V2  and I3D . We provide details of the backbone architectures here.
I3D. Figure 6(a), 6(b) show the architecture of I3D  used in our experiment; we reduce the first convolution kernel from 777 to 177 as we only use a sampled clip of frames. The MS module is inserted after of Figure 6(b).
4 Additional examples of visualization
We present more results of visualization on Something-Something V1  in Figure 9 and Kinetics-400  in Figure 10. From the top of each figure, RGB frames, color-coded displacement maps , and confidence maps are illustrated. We visualize examples of horizontal, vertical movements (Figure 8(a), 8(b), 8(c), 9(a), 9(b), 9(c)), rotations (Figure 8(d), 9(d)), scale changes (Figure 8(e), 9(e)), and deformations (Figure 8(f), 9(f)). We also report some failure cases in the last row of figures (Figure 8(g), 8(h), 9(g), 9(h)); estimated displacement maps around regions of occlusion or severe deformation are often inaccurate.
|Layers||ResNet-18||TSM ResNet-18||ResNet-50||TSM ResNet-50||Output size|
7, 64, stride 1,2,2
3 max pool, stride 2
|global average pool, FC||# of classes|
|177, 32, stride 1,2,2||T112112|
|111, 1280, stride 1,1,1|
|global average pool, FC||# of classes|