In this work we introduce a fully end-to-end approach for action detection in videos that learns to directly predict the temporal bounds of actions. Our intuition is that the process of detecting actions is naturally one of observation and refinement: observing moments in video, and refining hypotheses about when an action is occurring. Based on this insight, we formulate our model as a recurrent neural network-based agent that interacts with a video over time. The agent observes video frames and decides both where to look next and when to emit a prediction. Since backpropagation is not adequate in this non-differentiable setting, we use REINFORCE to learn the agent's decision policy. Our model achieves state-of-the-art results on the THUMOS'14 and ActivityNet datasets while observing only a fraction (2 frames.READ FULL TEXT VIEW PDF
Action detection in long, real-world videos is a challenging problem in computer vision. Algorithms must reason not only about whether an action occurs somewhere in a video, but also on the temporal extent ofwhen it occurs. Most existing work [22, 39, 13, 46]
take the approach of building frame-level classifiers, running them exhaustively over a video at multiple temporal scales, and applying post-processing such as duration priors and non-maximum suppression. However, this indirect modeling of action localization is unsatisfying in terms of both accuracy as well as computational efficiency.
In this work, we introduce an end-to-end approach to action detection that reasons directly on the temporal bounds of actions. Our key intuition (Fig. 1) is that the process of detecting an action is one of continuous, iterative observation and refinement. Given a single or a few frame observations, a human can already formulate hypotheses about when an action may occur. We can then skip ahead or back some frames to verify, and quickly narrow down the action location (e.g. swinging a baseball bat in Fig. 1). We are able to sequentially decide where to look and how to refine our hypotheses to obtain precise localization of the action with far less exhaustive search compared to existing algorithms.
Based on this intuition, we present a single coherent model that takes a long video as input, and outputs the temporal bounds of detected action instances. Our model is formulated as an agent that learns a policy for sequentially forming and refining hypotheses about action instances. Casting this into a recurrent neural network-based architecture, we train the model in a fully end-to-end fashion using a combination of backpropagation and REINFORCE .
Our model draws inspiration from works that have used REINFORCE to learn spatial glimpse policies for image classification and captioning [19, 1, 30, 43]. However, action detection presents the additional challenge of how to handle a variable-sized set of structured detection outputs. To address this, we present a model that decides both which frame to observe next as well as when to emit a prediction, and we introduce a reward mechanism that enables learning this policy. To the best of our knowledge, this is the first end-to-end approach for learning to detect actions in video.
We show that our model is able to reason effectively on the temporal bounds of actions, and achieve state-of-the-art performance on the THUMOS’14  and ActivityNet  datasets. Moreover, because it learns policies for which frames to observe, or temporally glimpse, it is able to do so while observing only a fraction (2% or less) of the frames.
There is a long history of work in video analysis and activity recognition [20, 49, 2, 31, 17, 8, 10, 12, 50]. For a survey we refer to Poppe  and Weinland et al. . Here we review recent work relevant to temporal action detection.
Temporal action detection. Canonical work in this vein is Ke et al. . Rohrbach et al.  and Ni et al.  use hand-centric and object-centric features, respectively, to detect fine-grained cooking actions in a fixed-camera kitchen setting. More related to our work is the unconstrained and untrimmed setting of the THUMOS’14 action detection dataset. Oneata et al. , Wang et al. , Karaman et al. , and Yuan et al.  use fusions of dense trajectories, frame-level CNN features, and/or sound features in a sliding window framework to perform temporal action detection. Sun et al.  uses web images as a prior to improve detection performance. Pirsiavash and Ramanan  build grammars over complex actions and additionally detect sub-components in time.
Methods for spatio-temporal action detection have also been developed. Within the context of “unconstrained” internet videos, this includes a body of work on spatio-temporal action proposals [44, 16, 36, 9, 7, 45, 41]. Analysis of broader surveillance scenes for action detection is also an active area of research. Shu et al.  reason about groups of people, Loy et al.  across multi-camera setups, and Kwak et al.  based on quadratic programming-based instantiations of rules. Common among these works is reasoning on spatio-temporal action proposals or human tracks, typically using sliding window-based approaches in the temporal dimension. Furthermore, these works are in the context of trimmed or constrained-setting video clips. In contrast, we address the task of temporal action detection in untrimmed, unconstrained videos, with an efficient method for determining which frames to examine.
End-to-end detection. Our goal of directly reasoning on the temporal bounds of actions shares philosophy with work in object detection that has regressed from full images to object bounds [29, 35, 5, 6, 26, 25]. In contrast, existing action detection methods typically use exhaustive sliding-window approaches and post-processing to produce action instances [22, 39, 13, 46]. To the best of our knowledge, our work is the first to address learning of temporal action detection in an end-to-end framework.
Learning task-specific policies. We draw inspiration from recent approaches that have used REINFORCE  to learn task-specific policies. Mnih et al. , Ba et al. , and Sermanet et al.  learn spatial attention policies for image classification, and Xu et al.  for image caption generation. In a non-visual task, Zaremba et al. 
Our goal is to take a long sequence of video and output any instances of a given action. Fig. 2 shows our model structure. The model is formulated as a reinforcement learning agent that interacts with a video over time. The agent receives a sequence of video frames as input, and can observe a fixed proportion of the frames. It must learn to effectively utilize these observations, or frame glimpses, to reason on the temporal bounds of actions.
The model consists of two main components: an observation network (Sec. 3.1.1), and a recurrent network (Sec. 3.1.2). The observation network encodes visual representations of video frames. The recurrent network sequentially processes these observations and decides both which frame to observe next and when to emit a prediction. We now describe each of these in more detail. Later in Sec. 3.2, we explain how we use a combination of backpropagation and REINFORCE to train the model in end-to-end fashion.
As shown in Fig. 2, the observation network , parameterized by
, observes a single video frame at each timestep. It encodes the frame into a feature vectorand provides this as input to the recurrent network.
Importantly, encodes information about both where in the video an observation was taken as well as what was seen. The inputs to the observation network therefore consist of the normalized temporal location of the observation, , and the corresponding video frame .
The architecture of the observation network is inspired by the spatial glimpse network of . Both and are mapped to a hidden space and then combined with a fully connected layer. is typically mapped with a sequence of convolutional, pooling, and fully connected layers; in our experiments we extract fc7 features from a fine-tuned VGG-16 network  and use .
The recurrent network , parameterized by , forms the core of the learning agent. As can be seen in Fig. 2, the input to the network at each timestep is observation feature vector . The network’s hidden state , a function of both and the previous hidden state , models temporal hypotheses about action instances.
As the agent reasons on a video, three outputs are produced at each timestep: candidate detection , binary indicator signaling whether to emit as a prediction, and temporal location indicating the frame to observe next. We now describe each of these in more detail.
Candidate detection. A candidate detection is produced using the function , where is a fully connected layer. is a tuple , where and are the normalized start and end locations of the detection, and is the confidence level of the detection. This candidate detection represents the agent’s hypothesis surrounding a current action instance. However, it is not emitted as a prediction at each timestep, which would lead to a large amount of noise and many false positives. Instead, the agent uses a separate prediction indicator output to signal when a candidate detection should be emitted as a prediction.
Prediction indicator. The binary prediction indicator signals whether corresponding candidate detection should be emitted as a prediction. , where
is a fully connected layer followed by a sigmoid nonlinearity. At training time,
is used to parameterize a Bernoulli distribution from which
is sampled; at test time, the maximum a posteriori estimate is used.
The combination of the candidate detection and prediction indicator is crucial for the detection problem, where positive instances may occur anywhere or not at all. It enables the network to indicate when it has identified a unique action instance to add to the prediction set, and essentially folds non-maximum suppression in as a learnable component of our end-to-end framework.
Location of next observation. The temporal location indicates the video frame that the agent chooses to observe next. This location is not constrained, and the agent may skip both forwards and backwards around a video.
The location is computed as , where is a fully connected layer, such that the agent’s decision is a function of its past observations and their temporal locations. At training time,
is sampled from a Gaussian distribution with a mean of
Fig. 2 further illustrates the roles of these outputs and their interaction with an example of a forward pass through the network.
Our end goal is to learn to output a set of detected actions. To achieve this, we need to train the three outputs at each step of the agent’s recurrent network: candidate detection , prediction indicator , and next observation location . Given supervision from temporal action annotations in long videos, training these involves challenges of designing suitable loss and reward functions, and handling non-differentiable model components. We now explain how we address these challenges. We use standard backpropagation to train , and REINFORCE to train and .
Candidate detections are trained using backpropagation to maximize the correctness of each candidate. We wish to maximize correctness regardless of whether a candidate is ultimately emitted, since the candidates encode the agent’s hypotheses about actions. This requires matching each candidate with a ground truth instance during training. We use the insight that at each timestep, the agent should form a hypothesis around the action instance (if any) nearest its current location in the video. This enables us to design a simple yet effective matching function.
Matching to ground truth. Given a set of candidate detections produced by a recurrent network of timesteps, and given ground truth action instances , each candidate is matched to one ground truth instance, or none if .
We define matching function
In other words, candidate is matched to ground truth if the agent’s temporal location at timestep is closer to than any other ground truth instance. Defining as the start and end location of a ground truth instance, is simply .
Once candidate detections have been matched to ground truth instances, we optimize a multi-task classification and localization loss function over the set:
Here the classification term is a standard cross-entropy loss on the detection confidence , encouraging the confidence to be closer to 1 if detection is matched to a ground truth instance, and 0 otherwise.
If the detection is matched to a ground truth (i.e. ), the localization term is an -regression loss that further encourages minimizing the distance between the two segments.
We optimize this loss function using backpropagation.
The observation location and prediction indicator outputs are non-differentiable components of our model that cannnot be trained with standard backpropagation. However, REINFORCE  is a powerful approach that enables learning in non-differentiable settings. We first briefly describe REINFORCE below. We then introduce a reward function that we use with REINFORCE to learn effective policies for observation and prediction emission sequences.
REINFORCE. Given , a space of action sequences, and , a distribution over and parameterized by , the REINFORCE objective can be expressed as
Here is a reward assigned to each possible action sequence, and is the expected reward under the distribution of possible action sequences. In our case we wish to learn network parameters that maximize the expected reward of a sequence of location and prediction indicator outputs.
The gradient of the objective is
This leads to a non-trivial optimization problem due to the high-dimensional space of possible action sequences. REINFORCE addresses this by learning network parameters using Monte Carlo sampling and an approximation to the gradient equation:
|Karaman et al. ||0.9||1.4||2.1||3.4||4.6|
|Wang et al. ||8.3||11.7||14.0||17.0||18.2|
|Oneata et al. ||14.4||20.8||27.0||33.6||36.6|
|Ours w/o w/o||8.6||14.6||20.0||27.1||33.3|
|CNN with NMS||6.4||9.6||12.8||16.7||18.5|
|LSTM with NMS||5.6||7.8||10.3||13.9||15.7|
Given an agent interacting with an environment, in our case a video, is the agent’s policy. This is a learned distribution over actions conditioned on the interaction sequence thus far. At timestep , is the policy’s current action (e.g. location or prediction indicator ), is the history of past states including the current, and is the history of past actions. is the cumulative future reward obtained from the current timestep onward, for a sequence of timesteps. The approximate gradient is computed by running an agent’s current policy in its environment to obtain interaction sequences.
To reduce the variance of the gradient estimate, a baseline reward is often estimated, e.g. via a separate network, and subtracted so that the gradient equation becomes:
REINFORCE learns model parameters according to this approximate gradient. The log-probabilityof actions leading to high future reward are increased, and those leading to low reward are decreased. Model parameters can then be updated using backpropagation.
Reward function. Training with REINFORCE requires designing an appropriate reward function. Our goal is to learn policies for the location and prediction indicator outputs that lead to action detection with both high recall and high precision. We therefore introduce a reward function that seeks to maximize true positive detections while minimizing false positives:
All reward is provided at the th (final) timestep, and is 0 for , since we want to learn policies that jointly lead to high overall detection performance. is the number of ground truth action instances, and is the number of predictions emitted by the agent. is the number of true positive predictions, is the number of false positive predictions, and and are positive and negative rewards contributed by each of these predictions, respectively. A prediction is considered correct if its overlap with a ground truth is both greater than a threshold and higher than that of any other prediction. In order to encourage the agent not to be overly conservative, a negative reward is provided if the the video contains ground truth instances () but the model did not emit any predictions ().
We use this function with REINFORCE to train the location and prediction indicator outputs, and learn observation and emission policies optimized for action detection.
We evaluate our model on two datasets - THUMOS’14  and ActivityNet . We show that our end-to-end approach enables the model to outperform state-of-the-art results by a large margin on both datasets. Furthermore, the learned policy of frame observations is both effective and efficient; the model achieves these results while observing in total only 2% or less of the video frames.
We learn a 1-vs-all model for each action class. In the observation network, we use a VGG-16 network  fine-tuned on the dataset to extract visual features from observed video frames. Fc7-layer features are extracted and embedded with the temporal location of the frame into a 1024-dimensional observation vector.
For the recurrent network, we use a 3-layer LSTM network with 1024 hidden units in each layer. Videos are downsampled to 5fps in THUMOS’14 and 1fps in ActivityNet, and processed in sequences of 50 frames. The agent is given a fixed number of observations for each sequence, typically 6 in our experiments. All temporal locations are normalized to
in a video sequence. Any predictions overlapping or crossing sequence bounds are merged with a simple union rule. We learn with mini-batches of 256 sequences, and use RMSProp
to modulate the per-parameter learning rate during optimization. Other hyperparameters are learned through cross-validation. The ratio of sequences containing positive examples in each mini-batch is an important hyperparameter to prevent the model from being overly conservative. Approximately one-third to one-half positive examples is typically used.
The action detection task of THUMOS’14  consists of 20 classes of sports, and Table 1 shows results on this dataset. Since the task comprises only 20 of the 101 action classes in the dataset, we first coarsely filter the full set of test videos for these classes, using video-level average pooling over class probabilities that are computed every 300 frames (0.1 fps). We report mAP for different IOU thresholds , and compare with the top 3 performers on the THUMOS’14 challenge leaderboard . All these methods compute dense trajectories  and/or CNN features over temporal windows, and use a sliding window approach with non-maximum suppression to obtain predictions.  uses dense trajectories only,  uses temporal windows of combined dense trajectories and CNN features, and  uses temporal windows of dense trajectories with video-level CNN classification predictions.
Our model outperforms existing methods at all values of . The relative margin increases as we decrease , indicating that our model more frequently predicts actions near ground truth instances even when not precisely localized. Our model achieves these results while processing only 2% of videos frames using its learned observation policy.
Ablation experiments. Table 1 also shows results for ablation experiments analyzing the contributions of different model components. The ablation models are as follows:
Ours w/o removes the prediction indicator output. The candidate detection at every timestep is emitted, and merged with non-maximum suppression.
Ours w/o removes the location output indicating where to observe next. Observations are instead determined by uniform sampling with the same total number of observations.
Ours w/o w/o removes both the prediction indicator and location observation outputs.
Ours w/o removes localization regression. All emitted detections are of median length from the training set, and centered on the currently observed frame.
CNN with NMS removes direct prediction of temporal action bounds. Per-frame class probabilities from the VGG-16 Network  used in our observation network are densely obtained at multiple temporal scales and aggregated with non-maximum suppression, similar to existing work [13, 39, 22].
|Baseball Pitch||8.6||14.6||Hamm. Throw||34.7||28.9|
|Basket. Dunk||1.0||6.3||High Jump||17.6||33.3|
|Clean and Jerk||13.3||42.8||Long Jump||47.6||39.0|
|Cliff Diving||17.7||15.6||Pole Vault||19.6||16.3|
|Cricket Shot||2.6||3.5||Soccer Penalty||8.7||8.3|
|Frisbee Catch||1.2||10.4||Throw Discus||36.2||29.5|
|Golf Swing||22.6||13.8||Volley. Spike||1.4||5.2|
Ours w/o obtains lower performance compared to the full model, due to many false positives. Ours w/o also lowers performance since uniform sampling does not provide sufficient resolution to localize action boundaries (Fig. 3). Interestingly, removing cripples the model more than removing , highlighting the importance of the observation policy. As expected, removing both outputs in Ours w/o w/o decreases performance further. Ours w/o is the poorest performing model at , even below the CNN, showing the importance of temporal regression. The relative difference with the CNN decreases and then flips when we decrease , indicating that the model still detects the rough location of actions but suffers from imprecise localization. Finally, the CNN with NMS achieves significantly lower performance than all ablation models except the Ours w/o model, quantifying the contribution of our end-to-end framework. Its performance is also in the range of but lower than , which uses dense trajectories 
and Imagenet-pretrained CNN features. This suggests that additionally incorporating motion-based features would further improve the performance of our model.
As an additional baseline, we perform NMS on top of an LSTM, a standard temporal network which produces frame-level smoothing and consistency . The LSTM with NMS achieves lower performance than the CNN with NMS, despite adding greater temporal consistency. The main reason appears to be that increasing the temporal smoothness of frame-level class probabilities is actually harmful, not beneficial, to the task of action instance detection, where precise localization of temporal boundaries is required.
Finally, we experimented with different numbers of observations per video sequence, e.g. 4, 8, and 10. Detection performance was not substantially different across this range. This is consistent with other work on CNNs for action recognition using max-pooling, highlighting the importance of learning effective frame observation policies.
Per-class breakdown. Table 2 shows the per-class AP breakdown of our model, and comparison with the top performer  on the THUMOS’14 leaderboard. Our model outperforms  on 12 out of 20 classes. Notably, it shows significant improvement on some of the most challenging classes in the dataset such as basketball dunk, diving, and frisbee catch. Fig. 4 shows examples of our model’s predictions, including several from these challenging classes. The model’s ability to reason holistically on action extents enables it to infer temporal boundaries even when frame appearance is challenging: e.g. similar pose and environment, or abrupt scene change in the second diving example.
Observation policy analysis. Fig. 5 shows examples of observation policies that our model learns, as well as accompanying predictions. For reference, we also show frame-level CNN probabilities from the VGGNet used in our observation network, to provide insight into frame-level signal for the action. In the top example of a javelin throw, the model begins to take more frequent observations once the person begins running. Near the end boundary of the action, it takes a step backwards to refine its hypothesis and then emits a prediction before moving on.
The lower example of diving is a challenging case where two action instances occur in quick succession. While the strength of the frame-level CNN probabilities over the sequence would be difficult for standard sliding-window approaches to handle, our model is able to discern the two separate instances. The model once again takes steps backwards to refine its prediction, including once (frame 93) when motion blur makes it difficult to discern much from the frame. However, the predictions are also somewhat longer than the ground truth, and upon observing its first frame of the second instance (frame 101), the model immediately emits a prediction of comparable but slightly longer duration than the first. This suggests that the model may have learned duration priors that, while generally beneficial, were overly strong in this case.
The ActivityNet action detection dataset  consists of 68.8 hours of temporal annotations in 849 hours of untrimmed, unconstrained video. There are 1.41 action instances per video and 193 instances per class. Tables 3 and 4 show per-class and mAP performance on the “Playing sports” and “Work, main job” subsets of ActivityNet, respectively. Evaluation uses the ActivityNet validation set, and hyperparameters are cross-validated on the training set.
Our model outperforms existing work 
, which is based on a combination of dense trajectories, SIFT, and ImageNet-pretrained CNN features, by significant margins. It outperforms in 13 out of 21 classes on the Sports subset and in 10 out of 15 classes on the Work subset. The improvement is particularly large on the Work subset. This is partially attributable to the fact that work activities are generally less well-defined and have less discriminative movements. In the example sequence of the Organizing Boxes action in Fig. 6, this is evident in the weaker, more diffuse frame-level CNN probabilities for the action. While this creates a challenge for approaches that rely on post-processing, our model’s direct reasoning on action extents enables it to still produce strong predictions.
|Discus Throw||26.2||17.6||Pole Vault||47.7||35.5|
|Ham. Throw||22.2||13.7||Start Fire||25.3||38.4|
|High Jump||41.3||21.9||Triple Jump||36.4||16.1|
|Search Security||24.5||33.9||Pumping Gas||54.7||34.0|
|Buy Fast Food||34.4||45.8||Setup Comp.||37.4||30.3|
|Clean Laptop Fan||26.0||35.8||Sharp. Knife||36.3||35.2|
|Making Copies||18.2||41.7||Sort Books||29.3||16.7|
|Organizing Boxes||29.6||19.1||Using Comp.||37.4||50.2|
|Organiz. Cabin.||19.0||43.7||Using ATM||29.5||64.9|
In conclusion, we have introduced an end-to-end approach for action detection in videos that learns to directly predict the temporal bounds of actions. Our model achieves state-of-the-art results on the THUMOS’14 and ActivityNet action detection datasets while observing only a fraction of frames. A direction for future work is to extend our framework to learn joint spatio-temporal observation policies.
We would like to thank Juan Carlos Niebles and Fabian Caba Heilbron for help with ActivityNet baselines.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961–970, 2015.
Efficient feature extraction, encoding, and classification for action recognition.In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 2593–2600. IEEE, 2014.