Few-Shot Video Classification via Temporal Alignment

06/27/2019 ∙ by Kaidi Cao, et al. ∙ Stanford University 5

There is a growing interest in learning a model which could recognize novel classes with only a few labeled examples. In this paper, we propose Temporal Alignment Module (TAM), a novel few-shot learning framework that can learn to classify a previous unseen video. While most previous works neglect long-term temporal ordering information, our proposed model explicitly leverages the temporal ordering information in video data through temporal alignment. This leads to strong data-efficiency for few-shot learning. In concrete, TAM calculates the distance value of query video with respect to novel class proxies by averaging the per frame distances along its alignment path. We introduce continuous relaxation to TAM so the model can be learned in an end-to-end fashion to directly optimize the few-shot learning objective. We evaluate TAM on two challenging real-world datasets, Kinetics and Something-Something-V2, and show that our model leads to significant improvement of few-shot video classification over a wide range of competitive baselines.



There are no comments yet.


page 1

page 3

page 7

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Figure 1: Our few-shot video classification setting. Pairs of semantically matched frames are connected with a blue dashed line. The arrows show the direction of the temporal alignment path.

The emergence of deep learning has greatly advanced the frontiers of action recognition

[41, 4]. The main focus tends to center around learning effective video representations for classification using large amounts of labeled data. In order to recognize novel classes that a pretrained network has not seen before, typically we need to manually collect hundreds of video samples for knowledge transferring. But such a procedure is rather tedious and labor intensive especially for videos, where the difficulty and cost of labeling is much higher compared to images.

There is a growing interest in learning models capable of effectively adapting themselves to recognize novel classes with only a few labeled examples. This is known as the few-shot learning task [10, 6]. Under the setup of meta-learning based few-shot learning, the model is explicitly trained to deal with scarce training data for previously unseen classes across different episodes. While the majority of recent few-shot learning works focus on image classification, adapting it to video data is not a trivial extension.

Videos are much more complicated than images, as recognizing some specific actions, such as opening the door, usually requires a complete modeling of temporal information. In the previous literature of video classification, 3D convolution and optical flow are two of the most popular methods to model temporal relations. The direct output of neural network encoders is always a temporal sequence of deep encoded features. State-of-the-art approaches commonly apply a temporal pooling module (usually mean pooling) in order to make final prediction. As observed before, averaging the deep features only captures the co-occurrence rather than the temporal ordering of patterns, which will unavoidably result in information loss.

Loss of information is even more severe for few-shot learning. It is hard to learn the local temporal patterns which are useful for few-shot classification with limited amount of data. Utilizing the long-term temporal ordering information, which is often neglected in previous works on video classification, might potentially help with few-shot learning. For example, if the model could verify that there is a procedure of pouring water before a close-up view of a just made tea, as shown in Fig. 1, the model will then become quite confident about predicting the class of this query video to be making tea, rather than some other potential predictions like boiling water or serving tea. In addition, Fig. 1 also shows that for two videos in the same class, even though they both contain a procedure of pouring water followed by closed-up view of tea, the exact temporal duration of each atomic step can vary dramatically. This non-linear temporal variations of videos pose great challenges for few-shot video learning.

With these insights, we thus propose Temporal Alignment Module (TAM) for few-shot video classification, a novel temporal-alignment based approach that learns to estimate temporal alignment score of a query video with corresponding proxies in the support set. To be specific, we compute temporal alignment score for each potential query-support pair by averaging per-frame distances along a temporal alignment path, which enforces the score we use to make prediction to preserve temporal ordering. Furthermore, TAM is fully differentiable so that the model can be trained end-to-end and optimize the few-shot objective directly. This in turn helps the model to better utilize long-term temporal information to make few-shot learning predictions. This module allows us to better model the temporal evolution of videos, while enabling stronger data efficiency in the few-shot setting.

We evaluate our model for few-shot video classification task on two action recognition datasets: Kinetics [17] and Something-Something V2 [12]. We show that when there is only a single example available, our method outperforms the mean pooling baseline which does not consider temporal ordering information by approximately 8% in top-1 accuracy. We also show qualitatively that the proposed framework is able to learn meaningful alignment path in an end-to-end manner.

In summary, our main contributions are: (i) We are the first to explicitly address the non-linear temporal variations issue in the few-shot video classification setting. (ii) We propose Temporal Alignment Module (TAM), a data-efficient few-shot learning framework that can dynamically align two video sequences while preserving the temporal ordering, which is often neglected in previous works. (iii) We use continuous relaxation to make our model fully differentiable and show that it outperforms previous state-of-the-art methods by a large margin on two challenging datasets.

2 Related Work

Few-Shot Learning. To address few-shot learning, a direct approach is to train a model on the training set and fine-tune with the few data in the novel classes. Since the data in novel classes are not enough to fine-tune the model with general learning techniques, methods are proposed to learn a good initialization model [9, 26, 32] or develop a novel optimizer [30, 25]

. These works aim to relieve the difficulty of fine-tuning the model with limited samples. However, such methods suffer from overfitting when the training data in novel classes are scarce but the variance is large. Another branch of works, which learns a common metric for both seen and novel classes, can avoid overfitting to some extent. Convolutional Siamese Net

[20] trains a Siamese network to compare two samples. Latent Embedding Optimization [39] employs attention kernel to measure the distance. Prototypical Network [35] utilizes the Euclidean distance to the class center. Graph Neural Networks [10] constructs a weighted graph to represent all the data and measure the similarity between data. Other methods use data augmentation, which learns to augment labeled data in unseen classes for supervised training [13, 44]. However, video generation is still an under-explored problem at least generating videos condition on a typical category. Thus, in this paper, we employ the metric learning approach and designs a temporal-aligned video metric for few-shot video classification.

There are works exploring few-shot recognition. OSS-Metric Learning [19] proposes a novel OSS-Metric Learning to measure the similarity of video pairs to enable one-shot video classification. [23] introduces a zero-shot method which learns a mapping function from an attribute to a class center. It has an extension to few-shot learning by integrating labeled data on unseen classes. CMN [47] is the most related work to ours. They introduce a multi-saliency embedding algorithm to encode video into a fixed-size matrix representation. Then they propose a compound memory network (CMN) to compress and store the representation and classify videos by matching and ranking. However, previous works collapse the order of frames at representation [19, 23, 47]. Thus, the learned model is sub-optimal for video datasets where sequence order is important. In this paper, we preserve the frame order in video representation and estimate distance with temporal alignment, which utilizes video sequence order to solve few-shot video classification.

Figure 2: Overview of our method. We first extract per-frame deep features using the embedding network. We then compute the distance matrices between the query video and videos in the support set. Next, an alignment score is computed out of the matrix representation. Finally we apply softmax operator over the alignment score of each novel class.

Video Classification. A significant amount of research has tackled the problem of video classification. State-of-the-art video classification methods have evolved from hand-crafted representation learning [18, 33, 40] to deep-learning based models. C3D [37] utilizes 3D spatial-temporal convolutional filters to extract deep features from sequences of RGB frames. TSN [41] and I3D [4] uses two-stream 2D or 3D CNNs with larger size on both RGB and optical flow sequences. By factorizing 3D convolutional filters into separate spatial and temporal components, P3D [29] and R(2+1)D [38] yield models with comparable or superior classification accuracy but smaller in size. An issue of these video representation learning methods is their dependence on large-scale video datasets for training. Models with an excessive amount of learnable parameters tend to fail when only a small number of training samples are available.

Another concern of video representation learning is the lack of temporal relational reasoning. Classification on the videos sensitive to temporal ordering poses a more significant challenge to the above networks which are tailored to capture short-term temporal features. Non-local neural networks [42] introduce self-attention to aggregate temporal information in the long-term. Wang [43] further employ space-time region graphs to model the spatial-temporal reasoning. Recently, TRN [46]

proposes a temporal relational module to achieve superior performance. Still, these networks inevitably pool/fuse features from different frames in the last layers to extract a single feature vector representing the whole video. In contrast, our model is able to learn video representation without loss of temporal ordering in order to generate more accurate final predictions.

Sequence Alignment. Sequence alignment is of great importance in the field of bioinformatics, which describes the way of arrangement of DNA/RNA or protein sequences, in order to identify the regions of similarity among them [2]. In the vision community, researchers have growing interests in tackling the sequence alignment problem with high dimensional multi-modal data, such as finding the alignment between untrimmed video sequence and the corresponding textual action sequence [5, 8, 31]. The main technique that has been applied to this line of work is dynamic programming. While dynamic programming is guaranteed to find the optimal alignment between two sequences given a prescribed distance function, the discrete operations used in dynamic programming are non-differentiable and hence prevent learning distance functions with gradient-based methods. Our work is closely related to recent progress on using continuous relaxation of discrete operations to tackle sequence alignment problem [5] and hence allow us to train our entire model end-to-end.

3 Methods

Our goal is to learn a model which can classify novel classes of videos with only a few labeled examples. The wide range of intra-class spatial-temporal variations of videos poses great challenges for few-shot video classifications. We address this challenge by proposing a few-shot learning framework with Temporal Alignment Module (TAM), which is to our best knowledge the first model that can explicitly learn a distance measure independent of non-linear temporal variations in videos. The use of TAM sets our approach apart from previous works that fail to preserve temporal ordering and relation during meta training and meta testing. Fig.2 shows the outline of our model.

In the following, we will first provide a problem formulation of few-shot video classification task, and then define our model and show how it can be used at training and test time.

3.1 Problem Formulation

In the few-shot video classification setting, we split the classes we have annotation of into : the base classes that have sufficient data for representation learning and : the novel or unseen classes that have only a few labeled data during testing stage. The goal of few-shot learning is then to train a network that can generalize well to new episodes over novel classes. In a -way, -shot problem, for each episode the support set will contain novel classes, and each class will have a very small amount of samples ( in our setting). The algorithm will have to classify videos from query set to one of the novel classes in support set. Episodes are randomly drawn from a larger collection of data, which we hereby denote them as meta set. In our setting, we introduce 3 splits over classes as meta training , meta validation and meta testing sets.

We formulate the few-shot learning as a representation learning problem through a distance function , where and are two samples drawn from and is an embedding function that maps samples to their representations. The difference between our problem formulation with the majority of previous few-shot learning researches lies in the fact that we are now dealing with higher dimensional inputs, i.e. (2+1)D volumes instead of 2D images. The addition of the time dimension in few-shot setting demands the model to be able to learn temporal ordering and relation with limited data in order to generalize to novel classes, which pose challenges that have not been properly addressed by previous works.

3.2 Model

With the above problem formulation, our goal is to learn a video distance function by minimizing the few-shot learning objective. Our key insight is that we want to explicitly learn a distance function independent of non-linear temporal variations by aligning the frames of two videos. Unlike previous works which use weighted average or mean pooling along the time dimension [41, 37, 42, 45, 4, 47], our model is able to infer temporal ordering and relationship during meta training or meta testing in an explicit and data efficient manner. In this subsection, we will breakdown our model following the pipeline as illustrated in Fig. 2.

Embedding Module: The purpose of the embedding module is to generate a compact representation of a trimmed action video that encapsulates its visual content. A raw video usually consists of hundreds of frames, whose information could be redundant were to perform per frame inference. Thus frame sampling is usually adopted as a preproccessing stage for video inputs. Existing frame sampling schemes can be mainly divided into two categories: dense sampling (randomly cropping out consecutive frames from the original full-length video and then dropping every other frame) [37, 4, 42, 45] and sparse sampling (samples distribute uniformly along the temporal dimension) [41, 48, 46, 21]. We follow the sparse sampling protocol first described in TSN [41], which divides the video sequence into segments and extracts a short snippets in each segment. The sparse sampling scheme allows each video sequence to be represented by a fix number of snippets. The sampled snippets span the whole video, which enables long-term temporal modeling.

Given a input sequence , we will apply a CNN backbone network to process each of the snippets individually. After the encoding, raw video snippets will then turn into a sequence of feature vectors . It is worth noticing that for the embedding of each video , its dimension is , rather than for image embedding, which is usually chosen as the activation before final fully-connected layer of a CNN network.

Distance Measure with Temporal Alignment Module (TAM):

Given two videos and their embedded features , we can calculate the frame-level distance matrix as


where is the frame-level distance value between the th frame of video and the th frame of video .

Figure 3: Methods for calculating alignment score. Each subplot shows a distance matrix. The darker of the color of an entry, the smaller the distance value is of a pair of relevant frames. The entries with green border denotes the entries contributing to the final alignment score.

We further define to be the set of possible binary alignment matrices, where , if the th frame of video is aligned to the th frame of video . Our goal is to find the best alignment


which minimizes the inner product between the alignment matrix and the frame-level distance matrix defined in Eq. (1). The video distance measure is thus given by


We propose to use a variant of Dynamic Time Warping (DTW) algorithm [24] to solve Eq. (2). This is achieved by solving for a cumulative distance function


In this setting of plain DTW, an alignment path is a contiguous set of matrix elements which defines a mapping between two sequences that satisfies the following conditions: boundary conditions, continuity and monotonicity. The boundary condition poses constraints on alignment matrix such that and

must be true in all possible alignment paths. In our alignment formulation, though the videos are trimmed, the action in the query video does not have to match exactly about its start and end action with the proxy. For example, consider the action of making coffee, there might be a snippet of stirring coffee at the end of action and it might not. To address this issue, we propose to relax this the boundary condition. Instead of having a path aligning the two videos from start to end, we allow the algorithm to find a path with flexible starting and ending points, while maintaining continuity and monotonicity. To work through this, we pad two column of

s at the start and end of the distance matrix so that it functions as enabling the alignment process to start and end at arbitrary position. So for our method, instead of computing the alignment score on a matrix, we work with the padded matrix of size . We further denote the indexes of the first dimension as , and indexes of the second dimension as , for simplicity. The compute of cumulative distance function is then changed into function


Note that if we follow the Eq. 5 to compute the alignment score, the score by itself is naturally normalized. Since at each time step except and , the alignment function forces a path from to , the final alignment score will be a summation of exactly scores. In this light, alignment scores computed from different pairs of query videos and support videos are normalized, which means that the scale will not be affected by the path chosen.

Differentiable TAM with Continuous Relaxation: While the above formulation is straightforward, the key technical challenge is that is not differentiable with respect to the distance function . Following the recent works on continuous relaxation of discrete operation and its application in video temporal segmentation [5, 22], we introduce a continuous relaxation to our Temporal Alignment Module (TAM). We use log-sum-exp with a smoothing parameter to approximate the non-differentiable minimum operator in Eq. (5)


While the use of continuous relaxation in Eq. (6

) does not convexify the the objective function, it helps the optimization process and allows gradients to be backpropagated through TAM.

Training and Inference: We have shown how to compute the cumulative distance function and use continuous relaxation to make it differentiable given a pair of input videos . The video distance measure is given by


In training time, given ground-truth video pair and support set

, we train our entire model end-to-end by directly minimizing the loss function


At test time we are given an unseen query video and its support set , our goal is to find the video that minimize the video distance function


4 Experiments

In this work, our task is few-shot video classification, where the objective is to classify novel classes with only a few examples from the support set. We divide our experiments into the following sections.

4.1 Datasets

As pointed out by [45, 46], existing action recognition datasets can be roughly classified into two groups: Youtube type videos: UCF101 [36], Sports 1M [16], Kinetics [17], and crowd-sourced videos: Jester[1], Charades [34], Something-Something V1&V2 [12], in which the videos are collected by asking the crowd-source workers to record themselves performing instructed activities. Crowd-sourced videos usually focus more on modeling the temporal relationships, since visual contents among different classes are more similar than those of Youtube type videos. To demonstrate the effectiveness of our approach on these two groups of video data, we base our few-shot evaluation on two action recognition datasets, Kinetics [17] and Something-Something V2 [12].

Kinetics [17] and Something-Something V2 [12] are constructed to serve as action recognition datasets so we have to construct their few-shot versions. For Kinetics dataset, we follow the same split as CMN [47] and sample 64 classes for meta training, 12 classes for validation and 24 classes for meta testing. Since there is no existing split for few-shot classification on Something-Something V2, we construct a few-shot dataset following the same rule as CMN [47]. We randomly selected 100 classes from the whole dataset. The 100 classes are then split into 64, 12 and 24 classes as the meta-training, meta-validation and meta-testing set, respectively.

4.2 Implementation Details

For a -way, -shot test setting, we randomly sample classes with each class containing examples as the support set. We construct the query set to have examples, where each unlabeled sample in the query set belongs to one of the classes in the support set. Thus each episode has a total of examples. We report the mean accuracy by randomly sampling 10,000 episodes in the following experiments.

We follow the video preprocessing procedure introduced in TSN [41]. During training we first resize each frame in the video to and then randomly crop a region from the video clip. For inference we change the random crop to center crop. For Kinetics dataset we randomly apply horizontal flip during training. Since the label in Something-Something V2 dataset incorporates an assumption of left and right, e.g. pulling something from left to right and pulling something from right to left, so we do not use horizontal flip for this dataset.

Following the experiment setting of CMN, we use ResNet-50 [14]

as the backbone network for TSN. We initialize network using pre-trained models on ImageNet

[7]. We optimized our model with SGD [3], with a starting learning rate of

and decaying every 30 epochs by

. We use meta-validation set to tune the parameters, and stop the training process when the accuracy on the meta-validation set is about to decrease. We implemented the whole framework with PyTorch

[27] framework. The whole model takes 4 TITAN Xp GPUs to train for hours.

4.3 Evaluating Few-Shot Learning

Kinetics Something V2
Method 1-shot 5-shot 1-shot 5-shot
Matching Net [47] 53.3 74.6 - -
MAML [47] 54.2 75.3 - -
CMN [47] 60.5 78.9 - -
TSN++ 64.5 77.9 33.6 43.0
CMN++ 65.4 78.8 34.4 43.8
TRN++ 68.4 82.0 38.6 48.9
TAM (ours) 73.0 85.8 42.8 52.3
Table 1: Few-shot video classification results. We report 5-way video classification accuracy on meta-testing set.

We now evaluate the representations we learned after optimizing few-shot learning objective. We compare our method with the two following categories of baselines:

4.3.1 Train from ImageNet Pretrained Features

For baselines that use ImageNet pretrained features, we follow the same setting as described in CMN. As the fact that previous few-shot learning algorithms are all designed to deal with images, they usually take image-level feature encoded by some backbone network as input. To circumvent this discrepancy, we first feed frames of a video to a ResNet-50 network pretrained on ImageNet, and then average frame-level features to obtain a video-level feature. The averaged video-level feature is then served as the input of few-shot algorithms.

Matching Net [39] We use an FCE classification layer in the original paper without fine-tuning in all experiments. The FCE module uses a bidirectional-LSTM and each training example could be viewed as an embedding of all the other examples.

MAML [9] Given the video-level feature as the input, we train the model following the default hyper-parameter and other settings described in [9].

CMN [47] As CMN is specially designed for few-shot video classification, it could handle video feature inputs directly. The encoded feature sequence is first fed into a multi-saliency embedding function to get a video-level feature. Final few-shot prediction is done by a compound memory structure similar to [15].

For the experiment results using ImageNet pretrained backbones, we directly take the numbers from CMN [47] to ensure fair comparison.

Figure 4: Visualization of our learning results. Comparison of our matched with CMN’s matched results in an episode. Although the averaged score is quite high given the false matching and the query image, our algorithm is able to find the correct alignment path the minimize the alignment score, which ultimately results in the correct prediction.

4.3.2 Finetune from Backbone on Meta Training Set

As raised by [6, 11, 28], using cosine distances between the input feature and the trainable proxy for each class could explicitly reduce intra-class variations among features during training. The rigorous experiments in [6] has shown that the Baseline++ model is competitive or even surpass when compared with other few-shot learning methods. So in finetuned settings we adapt several previous approaches with the structure of Baseline++ to serve as strong baselines.


For TSN++ baseline, we also use episode-based training to simulate the few-shot setting at meta-train stage to directly optimize for generalization to unseen novel classes. In order to get a video-level representation, we average over the temporal dimension of extracted per frame features for both query sets and support sets. The video level feature from support set could then serve as proxies for each novel class. We can then obtain the prediction probability for each class by normalizing these cosine distance values with a softmax function. For inference during meta-testing stage, we first forward each video in the support set to get proxies for each class. Given the proxies we can then make prediction for videos in query set.


We follow the setting of CMN and reimplement this method by ourselves. The only difference about CMN++ and CMN is that we replace the ImageNet pretrained feature with the feature extracted by TSN++ mentioned above.

TRN++ We also compare our approach against methods that attempt to learn a compact video-level representation given a sequence of image-level features. TRN [46]

proposes a temporal relation module, which uses multilayer perceptrons (MLP) to fuse features of different frames. We refer TRN++ to one of baselines by replacing average consensus module in TSN++ with temporal relation module.

By default we conduct 5-way few-shot classification if there is no further clarification. The 1-shot and 5-shot video classification results on both the Kinetics and Something-Something V2 datasets are listed in Table 1. It can be concluded that our approach significantly outperforms all the baselines on both datasets. In CMN paper, the experimental observations show that fine-tuning the backbone module on the meta-training set does not improve the few-shot video classification performance. In contradiction, we find that with proper data augmentation and training strategy, a model could be trained to generalize better on unseen classes in a new domain given the meta-training set. By comparing the results of TSN++ and TRN++, we could conclude that considering temporal relation explicitly helps with model generalization on unseen classes. Compare to TSN++, the improvement brought by CMN++ is not as large as the gap on ImageNet pretrained features reported in the original paper. This may be due to the reason that we are now using a more suitable distance function (cosine distance) during meta-training so that the frame-level feature is more discriminative among unseen classes. This in turn makes it harder to improve the final prediction given those strong features as the input. Finally it is worthwhile to note that TAM outperforms all the finetuned baselines by a large margin. This demonstrates the importance of taking temporal ordering information into consideration while dealing with few-shot video classification problem.

4.4 Qualitative Results and Visualizations

We show qualitative results comparing CMN and TAM in Fig. 4. In particular, we observe that CMN has difficulty in differentiating two actions from different classes with very similar visual clues among all the frames, e.g., backgrounds. As can be seen from the distance matrices in Fig. 4, though our method cannot alter the fact that the two visually similar action clips will have an averagely lower frame-wise distance value, it is able to find a temporal alignment that minimize the cumulative distance score between the query action video and the true support class video while the per-frame visual clue is not evident enough. Though the mean score of TAM is lower than the match of CMN, TAM succeeds in making the right prediction via calculating a lower alignment score out of the distance matrix.

4.5 Ablation Study

Kinetics Something V2
matching type 1-shot 5-shot 1-shot 5-shot
Min 52.4 71.6 29.7 38.5
Mean 67.8 78.9 35.2 45.3
Diagonal 66.2 79.3 38.3 48.7
Plain DTW 69.2 80.6 39.6 49.0
TAM(Ours) 73.0 85.8 42.8 52.3
Table 2: Temporal matching ablation study. We compare our method to temporal-agnostic and temporal-aware baselines.

Here we perform ablation experiments to demonstrate the effectiveness of our selections of the final model. We have shown in Section 4.3 that explicitly modeling the temporal ordering plays an important role for generalization to unseen classes. We now analyze the effect of different temporal alignment approaches.

While having the cosine distance matrix , there are several choices we could adopt to extract the alignment score out of the matrix, as visualized in Fig. 3

. In addition to our proposed method, we consider several heuristics for generating the scores. The first is “Min”, where we use the minimum element in the matrix

to represent the video distance value. The second is “Mean”, for which we average over the cosine distance value of all pairs of frames. These two choices both neglect the temporal ordering. We will then introduce a few potential choices that explicitly consider sequence ordering when computing the temporal alignment score. An immediate scheme is to take an average over the diagonal of the distance matrix. The assumption made behind this approach is that the query video sequence shall be perfectly aligned with its corresponding support proxy of the same class, which could be somewhat ideal in read world applications. To allow for more adaptive alignment strategy, we introduce Plain DTW and our method. Here the Plain DTW in Table. 2 means that there is no padding so that and are assumed to be in the alignment path, and for each time step during computing alignment score we allow a possible movement choice among , and .

The results are shown in Table. 2. It can be observed that we are able to improve the few-shot learning by considering temporal ordering explicitly. There are some slight differences in performance between method Diagonal and Mean regarding to the two datasets here. There are less visual clues in each frame of Something-Something V2 than that of Kinetics, so the improvement of using Diagonal with regard to using Mean is prominent for Something-Something V2, while the gap is closed for Kinetics. However, we see that through adaptive temporal alignment, our method consistently improve the baselines on two datasets by more than 3% accross 1-shot and 5-shots. This shows that by reinforcing the model to learn an adaptive alignment path across query videos and proxies, the final model could learn to encode better representations for the video, as well as a more accurate alignment score which could in turn help with few-shot classification.

The next ablation study is on the sensitivity of smoothing parameter . Previous works [5, 22] have shown that using empirically helps optimization in many tasks. Intuitively, a smaller functions more like the min operation and a larger means a heavier smoothing effect over the values in nearby positions. We experimented on within the value set of .

Figure 5: Smoothing factor sensitivity. We compare the effect of using different smoothing factors.

The results are shown in Fig. 5. In general, the performance is stable across values of . We observe that in practice ranges from 0.05 to 0.1 works relatively good under the setting of both two datasets. Thus we notice that a suitable is essential for the representation learning. When is too small, though it is able to function most similarly as the real min operator, the gradient is too imbalanced so that some pairs of frames are not adequately trained. On the contrary, a large might be too smooth so that the difference among all kinds of alignments are not notable enough.

5 Conclusion

We propose Temporal Alignment Module (TAM), a novel few-shot framework that can explicitly learn distance measure and representation independent of non-linear temporal variations in videos using very few data. In contrast to previous works, TAM dynamically aligns two video sequences while preserving the temporal ordering and it further uses continuous relaxation to directly optimize for the few-shot learning objective in an end-to-end fashion. Our results and ablations show that our model significantly outperforms a wide range of competitive baselines and achieves state-of-the-art results on two challenging real-world datasets.


This work has been partially supported by JD.com American Technologies Corporation (”JD”) under the SAILJD AI Research Initiative. This article solely reflects the opinions and conclusions of its authors and not JD or any entity associated with JD.com.


  • [1] The 20bn-jester dataset v1. https://20bn.com/datasets/jester.
  • [2] S. F. Altschul, T. L. Madden, A. A. Schäffer, J. Zhang, Z. Zhang, W. Miller, and D. J. Lipman. Gapped blast and psi-blast: a new generation of protein database search programs. Nucleic acids research, 25(17):3389–3402, 1997.
  • [3] L. Bottou.

    Large-scale machine learning with stochastic gradient descent.

    In Proceedings of COMPSTAT’2010, pages 177–186. Springer, 2010.
  • [4] J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In

    proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    , pages 6299–6308, 2017.
  • [5] C.-Y. Chang, D.-A. Huang, Y. Sui, L. Fei-Fei, and J. C. Niebles. : Discriminative differentiable dynamic time warping for weakly supervised action alignment and segmentation. arXiv preprint arXiv:1901.02598, 2019.
  • [6] W.-Y. Chen, Y.-C. Liu, Z. Kira, Y.-C. Wang, and J.-B. Huang. A closer look at few-shot classification. In International Conference on Learning Representations, 2019.
  • [7] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009.
  • [8] P. Dogan, B. Li, L. Sigal, and M. Gross. A neural multi-sequence alignment technique (neumatch). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8749–8758, 2018.
  • [9] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1126–1135. JMLR. org, 2017.
  • [10] V. Garcia and J. Bruna. Few-shot learning with graph neural networks. In ICLR, 2017.
  • [11] S. Gidaris and N. Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4367–4375, 2018.
  • [12] R. Goyal, S. E. Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, et al. The” something something” video database for learning and evaluating visual common sense. In ICCV, volume 2, page 8, 2017.
  • [13] B. Hariharan and R. Girshick. Low-shot visual recognition by shrinking and hallucinating features. In Proceedings of the IEEE International Conference on Computer Vision, pages 3018–3027, 2017.
  • [14] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  • [15] Ł. Kaiser, O. Nachum, A. Roy, and S. Bengio. Learning to remember rare events. arXiv preprint arXiv:1703.03129, 2017.
  • [16] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei.

    Large-scale video classification with convolutional neural networks.

    In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1725–1732, 2014.
  • [17] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017.
  • [18] A. Klaser, M. Marszałek, and C. Schmid. A spatio-temporal descriptor based on 3d-gradients. In BMVC 2008-19th British Machine Vision Conference, pages 275–1. British Machine Vision Association, 2008.
  • [19] O. Kliper-Gross, T. Hassner, and L. Wolf. One shot similarity metric learning for action recognition. In International Workshop on Similarity-Based Pattern Recognition, pages 31–45. Springer, 2011.
  • [20] G. Koch, R. Zemel, and R. Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML Deep Learning Workshop, volume 2, 2015.
  • [21] J. Lin, C. Gan, and S. Han. Temporal shift module for efficient video understanding. arXiv preprint arXiv:1811.08383, 2018.
  • [22] A. Mensch and M. Blondel. Differentiable dynamic programming for structured prediction and attention. ICML, 2018.
  • [23] A. Mishra, V. K. Verma, M. S. K. Reddy, S. Arulkumar, P. Rai, and A. Mittal. A generative approach to zero-shot and few-shot action recognition. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 372–380. IEEE, 2018.
  • [24] M. Müller. Dynamic time warping. Information retrieval for music and motion, pages 69–84, 2007.
  • [25] T. Munkhdalai and H. Yu. Meta networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 2554–2563. JMLR. org, 2017.
  • [26] A. Nichol and J. Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999, 2018.
  • [27] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017.
  • [28] H. Qi, M. Brown, and D. G. Lowe. Low-shot learning with imprinted weights. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5822–5830, 2018.
  • [29] Z. Qiu, T. Yao, and T. Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In proceedings of the IEEE International Conference on Computer Vision, pages 5533–5541, 2017.
  • [30] S. Ravi and H. Larochelle. Optimization as a model for few-shot learning. 2016.
  • [31] A. Richard, H. Kuehne, A. Iqbal, and J. Gall. Neuralnetwork-viterbi: A framework for weakly supervised video learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7386–7395, 2018.
  • [32] A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960, 2018.
  • [33] P. Scovanner, S. Ali, and M. Shah. A 3-dimensional sift descriptor and its application to action recognition. In Proceedings of the 15th ACM international conference on Multimedia, pages 357–360. ACM, 2007.
  • [34] G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev, and A. Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European Conference on Computer Vision, pages 510–526. Springer, 2016.
  • [35] J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pages 4077–4087, 2017.
  • [36] K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.
  • [37] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 4489–4497, 2015.
  • [38] D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450–6459, 2018.
  • [39] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pages 3630–3638, 2016.
  • [40] H. Wang and C. Schmid. Action recognition with improved trajectories. In Proceedings of the IEEE international conference on computer vision, pages 3551–3558, 2013.
  • [41] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pages 20–36. Springer, 2016.
  • [42] X. Wang, R. Girshick, A. Gupta, and K. He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7794–7803, 2018.
  • [43] X. Wang and A. Gupta. Videos as space-time region graphs. In Proceedings of the European Conference on Computer Vision (ECCV), pages 399–417, 2018.
  • [44] Y.-X. Wang, R. Girshick, M. Hebert, and B. Hariharan. Low-shot learning from imaginary data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7278–7286, 2018.
  • [45] S. Xie, C. Sun, J. Huang, Z. Tu, and K. Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European Conference on Computer Vision (ECCV), pages 305–321, 2018.
  • [46] B. Zhou, A. Andonian, A. Oliva, and A. Torralba. Temporal relational reasoning in videos. In Proceedings of the European Conference on Computer Vision (ECCV), pages 803–818, 2018.
  • [47] L. Zhu and Y. Yang. Compound memory networks for few-shot video classification. In Proceedings of the European Conference on Computer Vision (ECCV), pages 751–766, 2018.
  • [48] M. Zolfaghari, K. Singh, and T. Brox. Eco: Efficient convolutional network for online video understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pages 695–712, 2018.