Detecting the Moment of Completion: Temporal Models for Localising Action Completion

10/06/2017 ∙ by Farnoosh Heidarivincheh, et al. ∙ University of Bristol 0

Action completion detection is the problem of modelling the action's progression towards localising the moment of completion - when the action's goal is confidently considered achieved. In this work, we assess the ability of two temporal models, namely Hidden Markov Models (HMM) and Long-Short Term Memory (LSTM), to localise completion for six object interactions: switch, plug, open, pull, pick and drink. We use a supervised approach, where annotations of pre-completion and post-completion frames are available per action, and fine-tuned CNN features are used to train temporal models. Tested on the Action-Completion-2016 dataset, we detect completion within 10 frames of annotations for 75 Results show that fine-tuned CNN features outperform hand-crafted features for localisation, and that observing incomplete instances is necessary when incomplete sequences are also present in the test set.



There are no comments yet.


page 2

page 4

page 10

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

An action is defined, based on the Oxford dictionary, as the fact or process of doing something, typically to achieve an aim. Previous works on action recognition from visual data have overlooked assessing whether the action’s aim has actually been achieved, rather than merely attempted. Similarly, action localisation approaches detect the start and the end of an action’s attempt, without assessing whether the aim has been successfully achieved. The notion of assessing completion was introduced in [1]

where the performance of a variety of hand-crafted RGB-D features to distinguish between complete and incomplete sequences was reported, using a binary classifier which was learnt per action. However, the approach in 

[1] considered the sequence as a whole and classified it as either complete or incomplete. In this work, we closely examine the progression of the action and attempt to locate the moment in time when the action can indeed be considered completed. Temporal localisation of the moment of completion is a step towards an online approach that can detect completion once the action’s goal has been achieved. Completion could prompt reminders or guidance for the next action. For example, when preparing a food recipe, the user could be notified to wash the vegetables once these have been picked up.

Thus, we define the problem of ‘detecting the moment of completion’ as detecting the frame that separates pre-completion from post-completion per sequence, when present. Note that the ‘moment of completion’ is different from the typical ‘start’/‘end’ frames in action localisation. The former focuses on the action’s goal, while the latter focuses on the start and conclusion of any motion of relevance to the action. Consider the action ‘drink’; the action is completed when the subject consumes some of the beverage from a cup or glass. Traditionally, localisation approaches locate the first movement of the hand as it lifts the cup up to the mouth, all the way to putting the cup down. Even if drinking does not actually take place, the start and end of the hand motion is localised. In contrast, the moment of completion is as soon as the subject actually drinks the beverage. The moment of completion cannot be localised when the action’s goal has not been completed.

Figure 1: Complete and incomplete sequences for action switch are performed similarly up to the completion moment, after which an observer would confidently confirm completion. In this example, completion occurs as soon as the light is off. The pre- and post-completion frames are shown in orange and purple bordering, respectively.

In detecting the moment of completion, we take a supervised approach, where we observe complete and incomplete sequences for the same action. Up to the moment of completion, both complete and incomplete sequences would be similar, and an observer cannot confidently say if the action is going to be completed. The sequences differ after the completion takes place. Fig. 1 illustrates this notion using two sequences of action switch from the dataset in [1]. In this example, our approach detects the moment the light is observed to be switched off, as the moment of completion - whereas in [1] the aim was to label the entire sequence as complete/incomplete. Completing an action makes changes in the scene which can be considered as visual cues for detecting completion. We propose an approach for completion detection in which these learnt signatures are trained using Convolutional Neural Network (CNN) features along with a temporal model to encode the action’s evolution, similar to [2, 3, 4]. This approach is particularly useful for action/activity detection tasks, where the classification per frame depends not only on the observations of the current frame but also on the information passed from previous frames. Hidden Markov Models (HMMs) and Long-Short Temporal Memory models (LSTMs) are two temporal models, suitable for dealing with sequential input data and capable of handling frame by frame analysis of an action sequence. In this work, we compare HMM and LSTM for completion detection on six action classes.

The remainder of this paper is organised as follows: related work in Sec. 2, the proposed method and features used in Sec. 3, results in Sec. 4 and conclusion and future work in Sec. 5.

2 Related Work

We now present a brief review of related works in action recognition/detection and discuss which methods could be applied for completion detection.

Remarkable results for action recognition/detection have been achieved through extracting robust features, encoding them, then feeding them into classification methods [5]. For example,  [6, 7, 8, 9, 10, 11, 12, 13] extract space-time features from interest points or dense trajectories, encode them using BoWs or FVs, and then use SVM to recognise the actions. Such approaches do not model the temporal dynamics of actions explicitly, thus cannot provide per-frame detection. Recently, CNN features of spatial and temporal information have been used for action recognition/detection using dual streams [14, 15] or 3D convolutions [16, 17]. A seminal work of Wang et al. [13] encodes the transformations an action introduces to the scene from precondition into effect. Although their approach is similar to ours, focussing on the changes that actions make into the scene, they perform action recognition without an explicit temporal encoding, which makes the approach unsuitable for frame level analysis or detecting the moment of completion.

Many recent works on action recognition/detection, such as [2, 3, 4, 18, 19], take advantage of CNN models to extract features from their last fully-connected layers in order to have a good representation of spatial information per frame. Then, they also train a temporal model on top of the extracted features to encode the dynamics of the action. This approach suits our completion detection task in two ways. First, extracting features from the CNN helps to encode the appearance changes in the scene. Second, it uses a temporal model and can provide per-frame decisions, while considering the context. Two temporal models widely applied for action recognition/detection are HMMs and LSTMs. HMM and its variants have been widely used in action recognition/detection to model temporal dynamics of actions [20, 21, 22, 23, 24, 25]. For this purpose, an HMM model is usually trained per desired class. Then, the likelihood of a test sequence to belong to each model is evaluated and classification is performed by specifying the model which produces the maximum likelihood for the sequence. To detect completion, we similarly learn a single supervised HMM model per action, with two hidden states pre-completion/post-completion, then use the Viterbi algorithm to find the most likely label sequence. LSTM, as another temporal model designed to deal with sequential information, is able to capture long-term dependencies in the input data, using the memory inside each cell. In many recent works, LSTMs have achieved promising results for the task of action recognition/detection, such as [26, 18, 27, 28, 2, 3].

The closest work to ours, [1], extracts skeletal features from depth data which are temporally encoded by Fourier temporal pyramid for recognising sequence-level completion. As they use a sequence-level representation, the method is not suitable for per-frame decisions.

In this work, we go beyond sequence-level completion to, (i) achieve frame-level detection of the moment of completion, (ii) show fine-tuned appearance features outperform pre-trained and previously used skeletal features for completion detection, (iii) report results on six actions from the previously introduced action completion dataset [1], and (iv) compare per-action HMM and LSTM temporal models. Using fine-tuned appearance features, we detect the moment of completion for six different actions - within 10 frames - for 75.1% and 74.6% of the sequences by HMM and LSTM models, respectively.

Figure 2: Overall scheme of the proposed method for completion detection (of action plug):: CNN features are extracted per frame and fed into the temporal model (LSTM or HMM). The outputs pre and post represent pre- and post-completion labels, respectively.
Figure 3: An HMM model for completion detection encodes that the model starts at pre- state, and cannot transiting back to pre- state once at post- state.

3 Proposed Method for Detecting Completion

To present our proposal for detecting action completion, we first consider the underlying assumptions of the model, and then discuss how we use per-frame features and train two temporal models, HMMs and LSTMs. We make three reasonable assumptions, namely;

  • Temporally Segmented Sequences: We assume sequences include a single instance of the action that has not been completed at the beginning of the sequence, and the action is not repeated within the sequence. Accordingly, we only need to detect a single moment of completion per sequence, if at all. The sequences do not include cases where the subject undertakes steps to undo the completion, e.g. a person places the cup down after picking it up.

  • Momentary Completion: we assume completion could be detected using a single frame in the sequence. This is typically the first frame where a human observer would be sufficiently confident that the goal has been achieved. All frames prior to the completion localisation are pre-completion and all frames from the completion moment onwards indicate post-completion. For an incomplete sequence, all the frames would be labelled as pre-completion.

  • Uniform Prior for Completion:

    in a Bayesian setting one might wish to consider the prior probability of an action being completed. Consider that the prior probability for a person to forget to switch off the light in their home is 0.01. A method that thus assumes completion for every sequence will have a high accuracy but is nevertheless not suitable for completion detection. Thus, we assume a uniform prior (i.e. 50-50 chance) for an action to be completed.

Fig. 2 illustrates the overall scheme of the proposed method, which relies on per-frame features, optimised to classify pre-completion and post-completion frames. We test hand-crafted, pre-trained and fine-tuned CNN features for per-frame representations. For hand-crafted features, we evaluate the best skeletal feature from [1], namely Joint-Velocities (JV), as well as the best raw-depth feature, namely Local Occupancy Patterns (LOP) [29]. We also test hand-crafted RGB features, namely Histogram of Oriented Gradients (HOG). For CNN features, we use the “very deep VGG-16-layer” model [30] trained on UCF101 action dataset [14], as pre-trained features. We also fine-tune these models per action for two classes: pre-completion and post-completion, using leave-one-subject-out. Features are then extracted from the fc7 layer for both the pre-trained and fine-tuned CNNs.

Completion detection is then trained as a temporal model and evaluated using two temporal approaches:

HMM: A supervised HMM model with two hidden states is trained per action. Fig. 3

illustrates this model. The parameters of this HMM, i.e. initial and transition probabilities, in addition to covariance matrices and mean vectors of two multivariate Gaussians (for the two states of

pre- and post-completion) are computed from the training data. Note that, due to the assumptions made for completion detection, in this supervised HMM model, the initial probabilities for pre- and post- completion states are learned from training data to be 1 and 0, respectively. Also, the probability of transitioning from post-completion state to pre-completion state is learned to be 0, as the sequence remains in post-completion state once it transitions to it.

LSTM: The LSTM model is also trained per action. For this purpose, the frames’ CNN features, along with the frames’ labels (pre- or post-completion

), are fed into an LSTM network. The LSTM is optimised using Stochastic Gradient Descent with back-propagation. Also, the labels per frame are predicted using a

softmax layer on top of the LSTM hidden layer. The details of the LSTM architecture are presented in Sec 4.

In presenting results, we also evaluate the need for training using incomplete sequences per action. For these results, we only train on labeled complete sequences. This aims to assess when completion detection requires, or benefits from, observing incompletion.

4 Experimental Results

Dataset and Completion Annotation: We use the RGBD-AC dataset from [1], which contains both complete and incomplete sequences for the same action. It contains 414 complete and incomplete sequences of six actions: switch, plug, open, pull, pick and drink. However, the annotations provided by the dataset are sequence-level annotations. Thus, we re-annotate each sequence in the dataset for the moment of completion. An annotation corresponding to the completion moment is assigned to every complete sequence by one human observer.

Implementation Details: As noted earlier, we use a pre-trained spatial stream CNN from [14] which uses the VGG-16 architecture [30], fine-tuned on UCF101 action dataset [31]

. We perform Leave-One-Person-Out cross validation on the 8 subjects in the dataset. For fine-tuning, 15 epochs are performed, and the learning rate is started at

, divided by 10 at epochs 3 and 5, and fixed thereafter. All the other hyper-parameters are set similar to [15]. The 4096-dimension features are extracted per frame.

We test the full dimension as well as applying dimensionality reduction via PCA, as this proves essential for training the HMM. Without applying PCA, the HMM remains in the pre-completion state and never goes to post-completion

. We keep the number of components that encapsulates at least 90% of the variance. LSTM however, does not require dimensionality reduction, and indeed we see a slight drop in performance when PCA is applied. Hence, for all results reported below, we use PCA-HMM or LSTM for the temporal models. The LSTM model has one hidden layer with 128 units. After a random initialisation, 25 epochs are performed for training. The learning rate is

for the first epoch and is fixed at for the next epochs. For each test sequence, the output of HMM’s Viterbi algorithm and LSTM are validated against ground truth labels.


precision 78.0 99.0 99.8 99.7 99.8
recall 86.0 36.8 95.1 100 100
-score 81.8 53.7 97.4 99.9 99.9
plug precision 43.0 23.8 100 82.4 99.5
recall 76.4 56.3 9.3 21.0 94.5
-score 55.0 33.4 17.1 33.4 97.0
open precision 56.9 NA NA 80.9 99.9
recall 85.1 0 0 9.2 36.2
-score 68.2 NA NA 16.6 53.2
pull precision 57.6 77.8 100 100 99.9
recall 69.6 11.5 44.1 14.5 94.6
-score 63.0 20.0 61.2 25.3 97.2
pick precision 62.9 93.7 100 83.1 88.7
recall 82.3 65.7 3.8 56.3 97.4
-score 71.3 77.3 7.3 67.1 92.9
drink precision 66.2 96.1 95.6 68.2 84.7
recall 94.9 43.4 13.7 32.4 90.8
-score 78.0 59.8 24.0 43.9 87.6
Total precision 61.5 63.2 99.1 84.3 92.4
recall 85.1 36.1 30.2 39.8 89.0
-score 71.4 46.0 46.3 54.1 90.6

Table 1: Comparing features for completion detection using PCA-HMM.

Evaluation Metrics: Due to the imbalance in the number of pre- and post-completion

labels, we present precision and recall results, where the

post-completion is considered as the positive class. For comparison, -score percentage is also reported.

While precision, recall and

-score are suitable for comparing per-frame decisions, we wish to evaluate the accuracy with which the moment of completion is detected. Thus, we introduce a cumulative metric for evaluating completion detection that captures the difference between the labelled and estimated moment of completion,


where and correspond to the first post-completion label in ground truth and predicted labels, respectively, is the shift between the predicted and true completion frames, is a boolean function, and is the number of sequences. We do not use the absolute difference in frames, so that for complete sequences, we can capture whether detection takes place prematurely, or is overdue per action. When completion is not detected, . Also, for incomplete sequences .

We report these results as a cumulative graph of vs , for complete and incomplete sequences of each action, independently. The optimal plot for complete sequences is one that shows a step function from 0 to 1 when i = 0. The optimal graph for incomplete sequences would thus be a flat line at 0.

Feature evaluation: We compare fine-tuned CNN features (FT-CNN) features to the other hand-crafted and pre-trained features using PCA-HMM in Table 1. As can be seen, FT-CNN has the highest -score in total which shows its ability to represent the subtle changes in the scene observed after completion.

Temporal Model evaluation: The results of comparing PCA-HMM with LSTM per action (using CNN features), are summarised in Table 2 and shown in Fig. 4. Also, the combined results for all six actions in the dataset are shown in Fig. 5. The figure shows that while LSTM detects completion more frequently - in fact completion is detected for 96.7% of the complete sequences using LSTM, compared to 88.5% using PCA-HMM, LSTM is more likely to detect completion in incomplete sequences as well. The slightly delayed detection by PCA-HMM could be because of its low transition probability (less than 1%) to go from pre- to post-completion state. Also, it should be noted that, since LSTM does not learn any definite constraint on its transitions, it can transit back from post-completion to pre-completion, while PCA-HMM is forced to remain in post-completion. The results also show that action switch is the nearest to the ground truth for both temporal models, which is due to the obvious change it introduces to the scene. In contrast, action open has the worst performance, probably due to the subtle change it introduces to the environment.


precision 100 99.8
recall 99.7 100
-score 99.9 99.9

precision 92.2 99.5
recall 96.0 94.5
-score 94.0 97.0

precision 84.8 99.9
recall 69.1 36.2
-score 76.2 53.2

precision 96.0 99.9
recall 96.5 94.6
-score 96.2 97.2

precision 75.6 88.7
recall 96.1 97.4
-score 84.7 92.9

precision 82.1 84.7
recall 91.9 90.8
-score 86.7 87.6

precision 87.7 92.4
recall 92.7 89.0
-score 90.1 90.6

Table 2: Comparing LSTM and PCA-HMM using FT-CNN features.
Figure 4: LSTM vs PCA-HMM for complete (top) and incomplete (bottom) sequences.
Figure 5: Results for complete (left) and incomplete (right) sequences across the six actions

The need for observing incomplete sequences: In theory, completion detection could use any dataset (which only contains complete sequences), as both pre- and post-completion labels are present in complete sequences. In this experiment, we train PCA-HMM on only complete sequences, but test on both complete and incomplete sequences. The results are shown in Fig 6. As seen, while performance is slightly improved for detecting completion in complete sequences, particularly for actions open and pull, the performance on incomplete sequences clearly deteriorates. The method tends to label completion when the action is not completed, (especially for actions switch and drink). This shows that training only on complete sequences may cause the CNN learn incorrect features for completion detection and supports the need for learning from both complete and incomplete sequences.

In Fig. 7, we present some qualitative success and failure results for detecting completion.

Figure 6: The PCA-HMM model is trained only on complete sequences, then tested on complete (top) and incomplete (bottom) sequences.

Figure 7: sample sequences of success and failure for completion detection for both PCA-HMM and LSTM: (a) PCA-HMM has an early detection by 1 frame, (b) LSTM has a late detection by 1 frame, (c) PCA-HMM is delayed by 7 frames (6 frames omitted for space), while LSTM is delayed by 1 frame, (d) both PCA-HMM and LSTM fail to detect completion.

5 Conclusion and Future Work

This paper presents action completion detection as the task of localising the moment in time when a human observer believes an action’s goal has been achieved, for example an object has been picked up, or the light has been switched off. The approach goes beyond sequence-level recognition of completion towards frame level fine-grained perception of completion.

We proposed a supervised approach for detecting completion per action. Fine-tuned CNN features were extracted per-frame and used to train temporal models. Two widely used temporal models, HMM and LSTM, were applied and their results were validated against the ground truth pre- and post-completion labels. We showed that, for detecting completion, fine-tuned CNN features outperform pre-trained and hand-crafted features, as they capture the subtle changes in the scene when the action is completed. We also introduced a metric for completion detection to assess when completion is prematurely detected or when the decision is overdue. We showed that both temporal models (LSTM and PCA-HMM) can detect completion within 10 frames in 75% of complete sequences. However, we found that PCA-HMM outperforms LSTM on incomplete sequences.

We aim to pursue two directions for future work. First, we aim to search for novel temporal models that particularly aim at completion detection, and can work in untrimmed videos. Second, we aim to test the method on other datasets including a variety of actions for a wider understanding of action completion detection.


  • [1] Heidarivincheh, F., Mirmehdi, M., Damen, D.: Beyond action recognition: Action completion in rgb-d data. In: British Machine Vision Conference (BMVC). (2016)
  • [2] Ma, S., Sigal, L., Sclaroff, S.: Learning activity progression in lstms for activity detection and early detection.

    In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2016)

  • [3] Yeung, S., Russakovsky, O., Jin, N., Andriluka, M., Mori, G., Li, F.F.: Every moment counts: Dense detailed labeling of actions in complex videos. arXiv preprint arXiv:1507.05738 (2015)
  • [4] Lei, J., Li, G., Zhang, J., Guo, Q., Tu, D.: Continuous action segmentation and recognition using hybrid convolutional neural network-hidden markov model model. IET Computer Vision (2016)
  • [5] Kang, S.M., Wildes, R.P.: Review of action recognition and detection methods. arXiv preprint arXiv:1610.06906 (2016)
  • [6] Zhang, H., Zhou, W., Reardon, C., Parker, L.E.: Simplex-based 3d spatio-temporal feature description for action recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2014)
  • [7] Laptev, I., Marszalek, M., Schmid, C., Rozenfeld, B.: Learning realistic human actions from movies. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2008)
  • [8] Niebles, J.C., Wang, H., Fei-Fei, L.: Unsupervised learning of human action categories using spatial-temporal words. International Journal of Computer Vision (2008)
  • [9] Gilbert, A., Illingworth, J., Bowden, R.: Scale invariant action recognition using compound features mined from dense spatio-temporal corners. In: European Conference on Computer Vision (ECCV). (2008)
  • [10] Matikainen, P.K., Hebert, M., Sukthankar, R.: Trajectons: Action recognition through the motion analysis of tracked features. In: ICCV Workshop on Video-Oriented Object and Event Classification. (2009)
  • [11] Wang, H., Kläser, A., Schmid, C., Liu, C.L.: Dense trajectories and motion boundary descriptors for action recognition. International Journal of Computer Vision (2013)
  • [12] Wang, H., Schmid, C.: Action Recognition with Improved Trajectories. In: IEEE International Conference on Computer Vision (ICCV). (2013)
  • [13] Wang, X., Farhadi, A., Gupta, A.: Actions ~ transformations. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2016)
  • [14] Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Advances in Neural Information Processing Systems (NIPS). (2014)
  • [15] Feichtenhofer, C., Pinz, A., Zisserman, A.: Convolutional two-stream network fusion for video action recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2016)
  • [16] Ji, S., Xu, W., Yang, M., Yu, K.: 3d convolutional neural networks for human action recognition. IEEE Transaction on Pattern Analysis and Machine Intelligence (2013)
  • [17] Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3d convolutional networks. In: IEEE International Conference on Computer Vision (ICCV). (2015)
  • [18] Donahue, J., Hendricks, L.A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., Darrell, T.: Long-term recurrent convolutional networks for visual recognition and description. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2015)
  • [19] Yue-Hei Ng, J., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., Toderici, G.: Beyond short snippets: Deep networks for video classification. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2015)
  • [20] Brand, M., Oliver, N., Pentland, A.: Coupled hidden markov models for complex action recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (1997)
  • [21] Oliver, N.M., Rosario, B., Pentland, A.P.: A bayesian computer vision system for modeling human interactions. IEEE Transaction on Pattern Analysis and Machine Intelligence (2000)
  • [22] Ramanan, D., Forsyth, D.A.: Automatic annotation of everyday movements. In: International Conference on Neural Information Processing Systems (NIPS). (2003)
  • [23] Weinland, D., Boyer, E., Ronfard, R.: Action recognition from arbitrary views using 3d exemplars. In: IEEE International Conference on Computer Vision (ICCV). (2007)
  • [24] Yu, E., Aggarwal, J.K.: Human action recognition with extremities as semantic posture representation. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). (2009)
  • [25] Kellokumpu, V., Zhao, G., Pietikäinen, M.: Recognition of human actions using texture descriptors. Machine Vision and Applications (2011)
  • [26] Baccouche, M., Mamalet, F., Wolf, C., Garcia, C., Baskurt, A.:

    Action classification in soccer videos with long short-term memory recurrent neural networks.

    In: International Conference on Artificial Neural Networks. (2010)
  • [27] Grushin, A., Monner, D., Reggia, J.A., Mishra, A.: Robust human action recognition via long short-term memory. In: International Joint Conference on Neural Networks (IJCNN). (2013)
  • [28] Veeriah, V., Zhuang, N., Qi, G.J.: Differential recurrent neural networks for action recognition. In: IEEE International Conference on Computer Vision (ICCV). (2015)
  • [29] Wang, J., Liu, Z., Wu, Y., Yuan, J.: Mining actionlet ensemble for action recognition with depth cameras. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2012)
  • [30] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR (2014)
  • [31] Soomro, K., Zamir, A.R., Shah, M.: A dataset of 101 human actions classes from videos in the wild. CoRR (2012)