Tube-CNN: Modeling temporal evolution of appearance for object detection in video

by   Tuan-Hung Vu, et al.

Object detection in video is crucial for many applications. Compared to images, video provides additional cues which can help to disambiguate the detection problem. Our goal in this paper is to learn discriminative models for the temporal evolution of object appearance and to use such models for object detection. To model temporal evolution, we introduce space-time tubes corresponding to temporal sequences of bounding boxes. We propose two CNN architectures for generating and classifying tubes, respectively. Our tube proposal network (TPN) first generates a large number of spatio-temporal tube proposals maximizing object recall. The Tube-CNN then implements a tube-level object detector in the video. Our method improves state of the art on two large-scale datasets for object detection in video: HollywoodHeads and ImageNet VID. Tube models show particular advantages in difficult dynamic scenes.



There are no comments yet.


page 1

page 5

page 10

page 11


Plug Play Convolutional Regression Tracker for Video Object Detection

Video object detection targets to simultaneously localize the bounding b...

G-CNN: an Iterative Grid Based Object Detector

We introduce G-CNN, an object detection technique based on CNNs which wo...

Object Detection in Videos with Tubelet Proposal Networks

Object detection in videos has drawn increasing attention recently with ...

Online Generative-Discriminative Model for Object Detection in Video: An Unsupervised Learning Framework

Traditional single-view object detection methods often perform worse und...

Accident Forecasting in CCTV Traffic Camera Videos

This paper presents a novel dataset for traffic accidents analysis.Our g...

Spatio-activity based object detection

We present the SAMMI lightweight object detection method which has a hig...

Geometry-Aware Video Object Detection for Static Cameras

In this paper we propose a geometry-aware model for video object detecti...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Object detection has been recently advanced by the large progress of convolutional neural networks (CNNs). In the last few years, the performance of object detectors has doubled on some benchmarks 

[11, 10, 9] and the technology became mature for deployment in real applications such as automatic car driving. Despite this success, accurate object detection remains a difficult challenge in situations with partial occlusions, unusual viewpoints, motion blur and difficult lighting.

Figure 1: Space-time object tubes used in our work to capture evolution of object appearance over short sequences of video frames.

Static images may sometimes lack information to identify the presence of an object. Video, on the other hand, provides additional cues which can help to disambiguate the detection problem. For example, sequences with objects undergoing dynamic occlusions and rotations are structured and contain space-time patterns with object-specific properties, see Figure 1. In this paper, we argue that dynamic changes of appearance provide discriminative information that can boost object detection in difficult situations.

Object tracking in video can compensate some failures of object detectors by imposing temporal smoothness. Tracking, however, often assumes the temporal continuity of appearance and can be misleading when such assumptions break due to strong occlusions or fast motions. In contrast to tracking, our goal in this paper is to learn discriminative models for the temporal evolution of object appearance and to use such models for object detection.

To model the temporal evolution of appearance, we introduce space-time tubes corresponding to sequences of object bounding boxes in consecutive frames. We propose two CNN architectures for generating and classifying tubes, respectively. Our tube proposal network (TPN) first generates a large number of spatio-temporal tube proposals maximizing the objectness score and guaranteeing high recall for the ground truth object tubes. The Tube-CNN implements a tube-level object detector in the video using tube proposals as input. Examples of tubes are shown in Figure 1.

People are among the most frequent and difficult objects in images and video. Changes in clothing, poses and hair-style as well as frequent occlusions and dynamic variations of appearance make person detection particularly challenging. In this work, we pay close attention to this object class. To span the variety of dynamic scenes and activities, we consider people in Hollywood movies and address detection of person heads. To train our models we use the recent HollywoodHeads dataset [36] and extend it with the annotation of head tracks in hours of video from different movies. We test our method on two datasets and report significant improvements compared to the state of the art. We also evaluate and demonstrate advantages of our method in difficult scenes with strong dynamic occlusions.

In addition, the new dataset from ImageNet challenge of object detection in video (VID) [30] allows us to train and test tube models on other object classes. The dataset contains short video snippets fully annotated with object bounding-box tracklets. Our tube models show notable improvement in comparison with frame-level detector baselines. We achieve comparable results with the winner of ImageNet VID 2015 challenge [16]. On the Youtube-Objects (YTO) dataset [27], a weakly annotated dataset closely related to ImageNet VID dataset, our framework surpasses state-of-the-art localization performance by a large margin.

2 Related works

Object detection in still images is one of the most active subjects in computer vision. Since the success of R-CNN 

[11], many works have advanced the region-based CNN model. SPP-net [13] and Fast R-CNN [10] speed up R-CNN by a large magnitude with the idea of performing regional-pooling on a feature map. Ren et al. [28]

propose the Region Proposal Network (RPN) removing the need for ad-hoc bottom-up object proposals, and increasing the speed of object detectors. DeepMask 

[24] and SharpMask [25] are examples of methods aiming to generate class-agnostic segmentation mask proposals. Although using segmentation proposals could be beneficial, those methods heavily depend on large scale segmentation datasets like COCO [20] to achieve satisfactory result. Motivated by previous works, we propose two models that can generate and classify spatio-temporal tube proposals for object detection in video.

Object detection in video is closely related to object detection in images. One can directly use object detector on every single frames. Video provides additional cues, such as temporal continuity. A common practice for video object detection is to post-process detections produced by object detector on single image with trackers [2, 23, 5, 26]. The tracker first links high-confident detections on consecutive frames into tracks. As object detector often produces unstable scores along an object track, the linking step helps finding good detections by filling in the gaps. An additional smoothing step could be done to stabilize object position on the track. Recently, Kang et al. [17] use this practice to generate object tracks. In their work, the tracker output is named tube proposal

. It then becomes input for the next post-processing steps including box-perturbation, max pooling and rescoring. To avoid possible confusion, we stress that the term

tube proposal is used in our work with a different meaning. Unlike other video detection methods  [26, 33], which separate appearance and motion processing phases, our models treat video signal as an entire combination of the two factors.

Although there are many works on object proposals in still images (such as SelectiveSearch [35]), choice of methods for video tube proposals is limited. Most of existing tube proposal methods [22, 8] focus on action localization rather than object detection. The parallel work of Kang et al. [15]

is probably the closest to us. In this work, the authors propose a Tubelet Proposal Networks for generating tubelet proposals, which are then scored by an encoder-decoder CNN-LSTM. Regression is done on every frame to adjust box location along the tubelet. The authors show that their proposal network performs best with temporal window size of

frames. In our work, we argue that in such short temporal intervals, a linear movement assumption is good enough to produce well-localized object tube proposals for detection. Our best spatio-temporal tube proposals linearly span on temporal frames. Although the proposal network in [15] share the same spirit with our TPN model, our model is much simpler while being efficient. Compared to [15], we show significant improvements on ImageNet VID and Youtube-Objects datasets.

Object detection in video and tracking-by-detection have a tight relation. Graph-based methods [26, 33] formulate object tracking as a global optimization problem on association graph. The association graph is constructed by measuring appearance similarity between objects. In [19], the authors propose a Siamese CNN to learn such appearance similarity efficiently. In this work, we consider tracking-by-detection methods as useful tools to improve object detection performance. We focus on object detection and do not aim at improving tracking performance. Our model design and parameter choices are in favor of this goal.

3 Tube-CNN for object detection

Figure 2: Architecture of Tube-CNN for object detection. The input is a chunk of images and a set of tube proposals. The model starts with the CNN feature extractor to get a spatio-temporal feature volume. Afterwards the model splits into the three branches: tube classification (starts with Tube-of-Interest, TOI, pooling), regression on the start and end frames, which begins with the frame-level Region-of-Interest pooling, ROI.

We now describe the Tube-CNN model for detecting objects in video. Our model operates on short chunks of consecutive video frames. Instead of rectangular image regions, our elementary units are spatio-temporal tubes spanning several frames. In this section, we assume that a set of candidate tubes (tube proposals) is provided and describe our Tube-CNN that classifies and refines them.

The number of all possible tubes is huge and does not allow to consider them exhaustively. In this paper, we consider only linear tubes, i.e., tubes corresponding to uniform linear motion in the video. Such an approximation is reasonable only locally for small chunks of consecutive frames. In what follows, we always represent a linear tube by the two rectangles on its first and last frames. We study the effect of different tube lengths in Section 5.5.

3.1 Architecture

Tube-CNN is a convolutional neural network operating on chunks of consecutive frames and the corresponding sets of tube proposals. For each proposal, the model outputs the class scores and the refined proposal position.

Tube-CNN is an end-to-end model consisting of three main blocks: CNN feature extraction, tube classification and tube regression. The overall network architecture is shown in Figure 


CNN feature extractor.

The first block of the network extracts a feature map independently for every frame in the input chunk. Extracted features are stacked along the temporal dimension to form a spatio-temporal feature volume. To achieve good performance, we reuse the idea of sharing computations from first CNN layers among all proposals [13, 10]. For the architecture on this block, we can reuse most of single frame CNNs (we try CaffeNet111 and ResNet [14]).

Tube classification.

Classification starts with Tube-of-Interest (TOI) pooling on the feature volume. Within the TOI-pooling layer, we first map the coordinates of tube proposals to subvolumes of features. Next, we max-pool within each frame to obtain a fixed-size feature map. The last stage consists in the temporal aggregation of the maps coming from all the frames. We have tried max-pooling, average pooling, and 1-dimensional convolution. Our conclusion is that temporal max-pooling works best (without significant difference although) and we use it in what follows. Note, that if temporal aggregation is done by max or average pooling, the number of parameters in Tube-CNN does not depend on the chunk length , and the learned model can be readily applied to chunks of arbitrary length.

After TOI-pooling, we obtain a feature map of a fixed size for each tube proposal. This map is fed into the final part of the classification network. For example, in the case of ResNet-101 pipeline [14], it is the 4-th block. The tube classification branch ends with the cross-entropy loss.

Tube regression.

Tube regression adjusts spatial positions of tube proposals to better localize the object. Tube regression consists of two networks for bounding box regression [10] at the two ends of the tube. Both branches of tube regression start with ROI pooling [10] on the corresponding frame (the start and end frames of the chunk). Tube regression branches end with the smooth L1 loss [10].

3.2 Supervision

To train the Tube-CNN model, one needs a dataset with annotated object tracks. Details of the datasets used for our training are given in Section 5.1.

At the training time, Tube-CNN model takes a chunk of consecutive frames and a set of tube proposals as inputs. Given the chunk, we find a set of ground-truth tracks passing through it. Each ground-truth track is approximated within the chunk by a linear ground-truth tube. We use these tubes to assign labels to tube proposals in this chunk.

Label assignment for a tube proposal is based on the tube overlap between the tube proposal and the ground-truth tubes. We define the tube overlap between two tubes as minimum of the spatial Intersection-over-Union (IoU) overlaps at their ends. Each sampled tube proposal of a training chunk is assigned with a label: positive (class label of an object) and negative (background). Tube proposals having tube overlap with the best matched ground-truth are marked as positives. Tube proposals with are marked as negatives. Additional details on the training setup are given in Section 5.2 and in the Appendix A.

4 Generating tube proposals

The Tube-CNN model uses tube proposals as input. Tube proposals are defined as the sequences of region proposals on consecutive video frames that hypothesize spatio-temporal positions of objects. To cut down the overall number of possible proposals, we consider only the ones corresponding to uniform linear motion.

In this section, we introduce a CNN model that generates tube proposals given an input chunk of frames: Tube Proposal Network. We also propose a baseline tracking-based approach to build tube proposals on top of single-frame region proposals. Evaluation of different tube proposal methods is reported in Section 5.3.

4.1 Tube proposal network

We now present our Tube Proposal Network (TPN). This is a CNN model that can generate a small set of high quality class-agnostic tube proposals used for object detection.

Figure 3: Architecture of Tube Proposal Network (TPN). Tube anchors are plotted in red, yellow and green.


Tube Proposal Network is a fully convolutional network [21], taking a chunk of consecutive frames as input and producing a set of tube proposals. Each tube proposal has an objectness score and a refined region location. Figure 3 illustrates the TPN model.

Similar to the Tube-CNN model, TPN passes input chunks of consecutive frames through several CNN layers. The output feature maps are stacked along the temporal dimension and passed through a volumetric convolutional layer (conv3D) [34] to form a feature volume.

We refer to all spatial positions of the feature volume as seed locations and the corresponding features as seed features

. Each seed feature vector is associated with a set of

 reference tubes, which we call tube anchors. Tube anchors sharing the same center axis in the feature volume correspond to the same seed location as well as the same seed feature vector.

Each seed feature vector is then passed through an anchor network to produce two outputs per each of the  tube anchors: the objectness score and parameters for the tube regression. In our setting, the anchor network consists of two fully-connected layers. Finally, we obtain the tube proposals by collecting the regressed locations of all the tube anchors. Scores of the tube proposals directly come from the objectness scores of the anchors.

To prune tube proposals, we apply Non Maximum Suppression (NMS) based on the objectness score. We modify the standard NMS to the case of tubes by replacing the spatial IoU overlap ratio with the tube overlap ratio. We provide more details in the Appendix B..


TPN and Tube-CNN have many common properties, thus, their training procedures are similar. Compared to Tube-CNN operating with tube proposals, TPN is trained on mini-batches of tube anchors.

Each seed feature vector is connected to tube anchors. Tube anchors have the same center axis with the receptive field of the seed features. In the related work [28], anchors for region proposals correspond to several scales and aspect ratios. Adopting the setup to tubes, we have tried to add anchors with motion between the start and end regions, but it has not improved the recall while significantly increasing the number of anchors, thus complexity at the testing stage. We therefore use tube anchors, with spatial location fixed, but with varying scales and aspect ratios. Figure 4 shows examples of initial tube anchors and the corresponding final TPN proposals. We show that TPN can regress an anchor without motion to a tube proposal with motion, because the receptive field of the anchor is typically much bigger than the anchor itself and, thus, TPN can get enough information about the spatio-temporal neighborhood.

To train, we label each tube anchor as either positive (object) or negative (background). The label assignment procedure is similar to one of the Tube-CNN training procedure. The only difference is that the supervision of TPN is class-agnostic. Positive anchors have tube overlap ratio with a ground-truth tube of any object class. Anchors having with all ground-truth tubes are labeled as negatives.

Figure 4: Examples of TPN proposal. Examples of TPN proposals. Tube anchors and final proposal are plotted by green and magenta respectively.

4.2 Tube proposals by tracking box proposals

We introduce a tracking-based method to produce tube proposals from region proposals (which we refer to as box proposals) of single frames. Given a chunk of consecutive frames, we generate box proposals of the start frame by the Selective Search method [35]. For each of those box proposals, we hypothesize its corresponding positions on the end frame by using a tracker. Specifically, we use the Kanade-Lucas-Tomasi (KLT) tracker [31] to obtain point tracks between the start and the end frames of the chunk. Each box proposal is associated with a set of interior point tracks, corresponding to a set of movement directions. Those directions are then clustered into directional bins. We apply RANSAC on most populated bins to construct candidate temporal paths in the interval of frames222We use and in our experiments. Tube proposals are then constructed by linearly moving the box along all hypothesized temporal paths.

The above process could be built upon the output of any box proposal method. To have a rough comparison with [16], we generate tube proposals from high-scoring detections produced by our Fast R-CNN baseline333This baseline is described in details in Section 5.4.

5 Experiments

In this section, we present and analyze experimental results of our method. We first introduce video datasets for object detection in Section 5.1. We then clarify our experimental setup in Section 5.2. Section 5.3 evaluates the quality of tube proposals. Section 5.4 presents our main results for object detection and compares them to the state of the art. In Section 5.5 we present an ablation study analyzing parameters and design choices of our method.

5.1 Datasets

HollywoodHeads dataset.

HollywoodHeads444 [36] is a recent large-scale dataset for person head detection with video clips from 21 movies. The original dataset provides head annotation for all the frames. To train tube models in this work, we extend the annotation by combining detections to form tracks. Our extended HollywoodHeads dataset contains video frames with complete annotation of head tracks, split into ,    and frames for training, validation and test respectively. For evaluation and comparison to the previous work, we keep the test set of [36] with annotated frames555We have removed test frames from the original test set, because they belonged to short video clips of length less than frames..

Occlusions are among the most challenging factors for object detection. To study the detection performance under occlusions, we select a subset of video clips, named HollywoodHeads-Hard, with partially visible heads. HollywoodHeads-Hard is automatically composed from clips with multiple annotated heads, where ground-truth bounding boxes have significant overlap on the horizontal axis. In total, we obtain difficult clips and select one frame for testing from each of them.

Casablanca dataset.

In addition to HollywoodHeads, we evaluate head detection on the Casablanca dataset. This dataset was introduced in [29] and later extended in [36] with the annotation of missing ground-truth heads.

ImageNet VID dataset.

We also evaluate our method on the multi-class object detection benchmark. ImageNet VID [30] is a large-scale dataset for object detection providing complete annotation of 30 object classes in million video frames. The dataset contains short videos split into videos for training, videos for validation and videos for testing. The object classes of the ImageNet VID dataset are a subset of classes used in ImageNet DET challenge. As the ground-truth for the test set is not published, we evaluate and compare our results on the validation set with frames.

Youtube-Objects dataset.

Youtube-Objects (YTO) is a video dataset for object localization providing annotation for object classes on a sparse set of key frames [27]. The object classes in YTO form a subset of 30 object classes in ImageNet VID. We use YTO for additional evaluation of the models pretrained on ImageNet VID.

5.2 Training details

Base networks.

In our experiments, Tube-CNN is initialized either from CaffeNet [10] or from Resnet-101 [14] models pre-trained on ImageNet [4]. The TPN model is based on the CaffeNet architecture.

Training parameters.

We optimize Tube-CNN and TPN with the stochastic gradient descent (SGD) algorithm with momentum 0.9 and weight decay 0.0005 on mini-batches 

[18]. The classification and regression are trained with the cross-entropy loss and smooth L1 loss [10], respectively. We fix tube length for all tube models. This design choice is analyzed in Section 5.5.

Hard negative mining.

Hard negative mining enables training on difficult samples and has been proven effective for object detection with HOG and DPM models [3, 12] and more recently with CNN methods [1, 32]. In our experiments, we find that a few iterations of hard negative mining improves detection performance for all of our models. More details are provided in the Appendix C. Section 5.5 analyzes the effect of hard negative mining on the detection performance.

5.3 Evaluation of tube proposals

In this section, we evaluate the quality of tube proposals produced by different methods on HollywoodHeads and ImageNet VID datasets.

Tube proposal baselines.

Tube proposals have been previously proposed for the task of action detection in video. We here evaluate and compare two state-of-the-art methods for action proposals [8, 22]. For the 3D-Proposal method [22], we extract tube proposals for every shots of frames. For APT [8], we use longer shots due to the length constraint imposed by the method.

Evaluation of recall.

Object proposals should maximize the coverage of ground-truth object bounding boxes. We evaluate the quality of tube proposals by examining their recall on both frame level and video level. We refer to the two measures as box-recall and tube-recall, respectively. Tube-recall is the percentage of ground-truth tubes having tube overlap-ratio more than with at least one tube proposal (tube overlap-ratio is defined in Section 3.2). To compute box-recall, we first split tube proposals into per-frame box proposals. Box-recall is then computed as the percentage of still images ground-truth bounding boxes having IoU overlap-ratio more than with at least one box proposal.

       HollywoodHeads   ImageNet VID
      Number of proposals    Number of proposals
Figure 5: Box-recall and tube-recall vs. average number of tube proposals per chunk on HollywoodHeads and ImageNET VID datasets.


Avg.# TP Tube Rec. Box Rec.
3D Proposals [22] 10K 55.3 61.3
APT Proposals [8] 4.2K 48.7 62.1
SS-Track (ours) 6.7K 92.8 96.7
Det-Track (ours) 6K 94.8 97.6
TPN (ours) 100 91.1 91.9
500 93.2 96.2
1.6K 94.8 98.3
5.2K 95.3 99.6

Method Avg.# TP Tube Rec. Box Rec. SS-Track (ours) 17K 96.8 98.1 Det-Track (ours) 17K 97.5 98.5 TPN (ours) 100 81.3 83.2 1K 93.3 96.5 5K 95.3 98.8 10K 95.5 99.2

(a) HollywoodHeads
(b) ImageNet VID
Table 1: Evaluation of tube proposals in terms of tube-recall and box-recall on HollywoodHeads and VID datasets. Avg.#TP stands for the average number of tube proposals per chunk of consecutive frames.

We evaluate the recall on HollywoodHeads and ImageNet VID datasets in Table 1. For each dataset we select a validation set of about frame chunks. We first observe that action proposals do not achieve satisfactory level of recall for object detection. Due the low performance, we omit action proposals from further evaluation. We next evaluate the two track-based tube proposals introduced in Section 4.2. The proposals based on Selective Search and single-frame object detections are denoted by SS-Track and Det-Track respectively. Both methods obtain high recall at the cost of the large number of tube proposals.

For TPN we experiment with different numbers of top-scoring tube proposals after tube-NMS. For HollywoodHeads TPN outperforms all other methods in both Tube recall and Box recall. For ImageNet VID TPN achieves slightly worse recall compared to SS-Track and Det-Track. This might be due to the large variation of object aspect ratio in ImageNet VID. Figure 5 illustrates recall depending on the number of tube proposals. In all cases TPN achieves a significant reduction in the number of proposals by the cost of a minor drop of recall. This property can be used to significantly reduce computational complexity of detection at test time.

5.4 Detection results

In this section we report results for object detection. Following the PASCAL VOC detection challenge [6], we use Average Precision (AP) measure to evaluate object detection on the level of individual frames. Given results of a tube object detector, we decompose tubes into per-frame object detections. On each frame we aggregate detections from multiple tubes using the standard NMS procedure. Resulting detections are evaluated against ground truth object bounding boxes on test frames.

Model HH Casa VID
Context-aware CNN [36] 72.7 72.7 -
Kang [15] - - 68.4
Box-CNN 82.4 81.9 68.7
Box-CNN + track 84.7 82.1 68.9
Tube-CNN 86.8 84.0 72.7
Table 2: Detection performance mAP () on the HollywoodHeads, Casablanca and ImageNet VID datasets.
Kang [15] 84.6 78.1 72.0 67.2 68.0 80.1 54.7 61.2 61.6 78.9 71.6 83.2 78.1 91.5 66.8 21.6

79.1 73.7 79 66.3 68 80.3 56.8 59.1 70.8 75.6 71.7 82.5 76.9 84.1 62.9 25.1
Tube-CNN 81.1 77 79.9 72.5 72.5 84.2 56.2 60.3 77.8 77.4 73.5 83.9 82 94.9 71.6 40.5
Model mAP
Kang [15] 74.4 36.6 76.3 51.4 70.6 64.2 61.2 42.3 84.8 78.1 77.2 61.5 66.9 88.5 68.4
Box-CNN 78.3 44 76.9 58.1 53.6 56.8 72.6 52.2 88.5 78.9 77.6 64 60 87.3 68.7
Tube-CNN 78.8 50 83.7 55.5 56.7 58.5 75.6 62.0 90.4 81.4 80.3 70.7 65.6 87.4 72.7
Table 3: Object detection performance AP () on ImageNet VID validation set.


We compare our method to the single-frame Fast R-CNN object detectors [10] which we re-train on our data. We refer to such detectors as Box-CNN to contrast with the tube-level models. Box-CNNs are trained on single frames of training videos and frame-level annotations.

Single-frame object detectors often produce noisy detections along object tracks [17]. A common practice to improve object detection in video is to link detections on consecutive frames with tracking-by-detection methods [5, 26]. We here follow the procedure in [5] and link object detections across frames based on the consensus of KLT point tracks. This procedure enables to overcome some failures of object detectors by bridging the gaps in object tracks along time and by discarding short tracks. We apply tracking-by-detection to the Box-CNN method and denote this strong baseline as ”Box-CNN + track”.

Head detection result.

In the first and second columns of Table 2 we compare our method to the state-of-the-art [36] as well as to Box-CNN and Box-CNN + track baselines on HollywoodHeads and Casablanca datasets. Our Tube-CNN significantly outperforms [36] by more than 10% AP on both datasets. As will be detailed in Section 5.5 this improvement originates both from the Tube-CNN object detectors and from the more powerful ResNet-101 base network. Our best Tube-CNN model is trained on SS-Track tube proposals. Tube-CNN achieves consistent improvement over other baseline methods using comparable base CNN architectures.

Figure 6: Detection results on HollywoodHeads-Hard.

Figure 6 illustrates results for the HollywoodHeads-Hard test set with occluded heads. Compared to the full test set, the performance on HollywoodHeads-Hard is significantly lower for all tested methods. Compared to the Box-CNN, Tube-CNN shows improvement of almost 10% AP, confirming the advantage of our method for particularly difficult scenes. Qualitative results of Tube-CNN for difficult examples of occluded heads are illustrated in Figure 7.

Method Avg.
Kang [17] 94.1 69.7 88.2 79.3 76.6 18.6 89.6 89.0 87.3 75.3 76.8
Galteri [7] 87.8 94.8 81.7 95.1 84.3 97.5 78.0 61.0 94.8 76.8 85.2
Kang [15] 91.2 99.4 93.1 94.8 94.3 99.3 90.2 87.8 89.7 84.2 92.4
Tube-CNN 98.2 100 92.4 97.5 96.5 99.3 92.1 96.1 95.8 89.9 95.8
Table 4: Localization performance CorLoc () on the YTO dataset.

Detection results on ImageNet VID and YTO.

We next evaluate Tube-CNN for the more general case of multi-class object detection. Table 3 presents per-class results for the validation set of the ImageNet VID benchmark. Similar to experiments on head detection in HollywoodHeads dataset, we observe the consistent improvement of the proposed Tube-CNN over Box-CNN for out of object categories. We, hence, conclude that our method generalizes well to a large number of object classes. The VID column of Table 2 compares results of Tube-CNN to baselines on the ImageNet VID validation set. As in the case of head detection, our method improves all tested baselines. In Figure 8 we visualize some qualitative results comparing Box-CNN and Tube-CNN.

On YTO we evaluate results using the standard Correct Localization (CorLoc) measure used for this dataset in [27, 17, 15, 7]. Given an object class, CorLoc is defined as the percentage of positive images correctly localized according to the PASCAL criterion. Table 4 compares CorLoc results of the Tube-CNN method with the best results reported in the literature. Our method provides best performance on YTO even if it was not trained nor fine-tuned on this dataset.

5.5 Ablation study

In this section, we analyze design choices and parameters of our method. We also investigate and demonstrate advantages of TPN in terms of computational complexity.

Base architectures and hard negative mining.

Box-CNN and Tube-CNN methods depend on the training schemes, base network architectures as well as the type and the amount of used tube proposals. Here we analyze the detection performance by varying each of these factors. Table 5 reports performance of our models on HollywoodHeads and ImageNet VID datasets under different settings.

With respect to the training scheme, we achieve consistent improvement with hard negative mining (HN) for all models. Regarding the choice of the base network, ResNet architecture (RN) provides significant improvements compared to the more shallow CaffeNet (CN) network. Our strongest models use Resnet-101 base network. For the ImageNet VID dataset we only report results of models based on Resnet-101.

We train and test Tube-CNN models using tube proposals generated by our TPN model and by tracking box proposals (i.e. SS-Track). In Table 5, we denote these two Tube-CNN models as ”Tube-CNN + SS-Track” and ”Tube-CNN + TPN” respectively. In both cases, Tube-CNN models outperform Box-CNN and other baselines. The main advantage of TPN is in test time, which grows linearly with the number of proposals. Using a smaller set of proposals, Tube-CNN + TPN is able to achieve similar (or even better) results compared to Tube-CNN + SS-Track. This together with the analysis of recall in Section 5.3 confirms the advantage the proposed TPN scheme.

Tube-CNN performs slightly better on SS-Track proposals compared to TPN. We believe that Tube-CNN + TPN detection results could be further improved if using deeper networks for the TPN model.




Box-CNN 2K 71.2 76.4 75.5 82.4
Box-CNN + track 2K 73.1 76.9 78.0 84.7
Tube-CNN + SS-Track 6K 76.9 78.3 83.3 86.8
Tube-CNN + TPN 100 74.9 78.6 81.7 83.9
300 75.8 78.7 82.3 85.0
2K 73.7 79.1 82.7 86.2

(a) HollywoodHeads



Box-CNN 2K 66.1 68.7
Box-CNN + track 2K 67.2 68.9
Tube-CNN + SS-Track 17K - 72.7
Tube-CNN + TPN 300 69.5 70.6
1K 70 70.9
2K 69.8 70.9

(b) ImageNet VID
Table 5: Detection performance mAP () on HollywoodHeads and ImageNet VID dataset. CN and RN and HN stands for CaffeNet, Resnet and hard negative mining respectively.

Effect of the tube length.

We restrict our method to tubes with linear motion to enable a tractable approach for generating tube proposals. We believe linear tubes might be sufficient to represent short time intervals, which is the main target in this work. In Table 6 we report results comparing performance of Tube-CNN on the HollywoodHeads dataset for different values of tube length (all models are based on CaffeNet without hard negative mining). We observe the best performance obtained for .

Tube length T=1 T=5 T=10 T=15 T=20
AP (%) 71.2 76.6 76.9 73.4 70.1
Table 6: Tube-CNN performance for different values of T.

Advantage of feature-level aggregation.

In this experiment, we show the advantages of early feature aggregation with TOI pooling and learning features for tube classification. We compare our Tube-CNN framework with baselines of temporal aggregation of single-frame image features. In particular, we use our best CaffeNet-based single frame Box-CNN detector to extract FC7 features on all the frames of training tubes and train a linear SVM classifier based on the temporally aggregated features. SVM parameters for each setup are chosen with 5-fold cross-validation. In Table 7 we report results for max and average aggregation baselines. The improvement of Tube-CNN over baselines confirms the advantage of jointly learning features and their temporal aggregation as proposed in this work.

Box-CNN fc7 max fc7 avg Tube-CNN
76.4 77.0 77.3 79.1
Table 7: Aggregation of Box-CNN and Tube-CNN.

Running time.

Table 8 compares running times of object detectors (in frames per second) for Tube-CNN on SS-Track and TPN proposals. To obtain results for SS-Track, we accumulate the time required for per-frame Selective Search proposals, KLT point tracking and Tube-CNN evaluation. Results for TPN are obtained by summing times for the evaluation of the TPN network, applying tube-NMS and evaluating Tube-CNN. When using ResNet-based Tube-CNN object detectors, our method with 100 TPN is about times faster compared to Tube-CNN + SS-Track. The improvements of running time come from the efficient generation of TPN proposals and from the reduced number of proposals required for the Tube-CNN evaluation.

Method SS-Track RPN-Track TPN
Num.TP 6K 1.2K 100 300 500 1K
CN 0.3 2.3 4.3 3.7 3.0 2.3
RN 0.06 0.7 1.7 1.1 1.0 0.7
Table 8: Running times (frames per second) for Tube-CNN object detectors using SS-Track proposals and different numbers of TPN proposals.
Box-CNN Tube-CNN
Box-CNN Tube-CNN Box-CNN Tube-CNN

Figure 7: Qualitative results for the Tube-CNN and Box-CNN models on Casablanca dataset and the HollywoodHeads-Hard set. On each frame we illustrate results of the Box-CNN model and the Tube-CNN model. For both methods we choose thresholds such that precision equals recall on the corresponding dataset. All detections with scores above threshold are plotted. Yellow boxes: correct detections, red boxes: false detections.
Box-CNN Tube-CNN Box-CNN Tube-CNN
Figure 8: Qualitative results for the Tube-CNN and Box-CNN models for the ImageNet VID dataset. We use the same structure as one in Figure 7.

6 Conclusion

We have addressed the task of object detection in video. To this end, we proposed two CNN models for generating and classifying object tube proposals respectively. Our tube classification model achieves state-of-the-art results on all four tested datasets for object detection in video. The unified framework of the two models is both accurate and computationally efficient at the running time. Our method particularly improves object detection in difficult situations with dynamic occlusions. We therefore believe it could be highly valuable when used as input for object tracking and other challenging scenes.


This work was supported in part by the ERC grant ACTIVIA and the MSR-Inria joint lab.

Appendix A Training Tube-CNN

Training procedure.

We optimize Tube-CNN with the stochastic gradient descent (SGD) algorithm using momentum and weight decay on mini-batches. Mini-batch sampling is done hierarchically. We first sample chunks of consecutive frames. Then, on each sampled chunk, we sample a set of tube proposals. The training batch therefore contains tube proposals in total. Positive samples contribute at most of the training batch. We initialize the learning rate at , and decrease it several times by a factor of after the validation error reaches saturation.

Network architectures.

We use two ImageNet pre-trained networks to initialize Tube-CNN model: CaffeNet [10] and Resnet-101 [14]. Our training settings for these two architectures are descrived below.


Tube-CNN initialized from CaffeNet has a CNN feature extractor block composed of the first convolutional layers. The last fully-connected (FC) layers are used for tube classification and tube regression. TOI pooling used for tube classification, and ROI pooling used for tube regression are applied to the feature volume created by stacking feature maps.

As described in Section 3.1, our TOI pooling consists of: spatial max-pooling and temporal max-pooling. Similarly to [10], the spatial max-pooling produces fixed-size output features. On the other hand, temporal max-pooling collapses all values along temporal dimension into one max value. Therefore, for every input tube proposal, TOI pooling layer outputs a -channels feature map with a fixed spatio-temporal extent of . Each feature map is flattened into a -dimentional feature vector, which is then passed through FC layers to compute the classification score.

We use the same spatial hyper-parameters of for ROI pooling. Given an input tube proposal, the ROI pooling layer outputs two -dimensional feature vectors corresponding to the two ends of the tube proposal. We pass the two feature vectors through FC layers to produce regression parameters for the beginning and the end of the tube.


TOI pooling and ROI pooling layers are inserted between the third and the fourth blocks of the network, i.e. conv_4x and conv_5x of Table 1 in [14]. In detail, convolutional layers before and including conv_4x belong to CNN feature extractor block of Tube-CNN. All layers after and including conv_5x construct tube classification and tube regression blocks.

The spatial extent of both TOI and ROI pooling layers has a fixed-size of . The temporal max-pooling is done similarly as for CaffeNet. Without being flattened, pooled feature maps are passed into followed by an average pooling layer. In our setting, output size of is different from the one of the original network. We adapt to this change by adjusting kernel size of the average pooling layer to . The last FC layers of tube classification and tube regression blocks take the average pooling output to produce final object score and position.

Appendix B Training TPN

Training batch.

TPN model is trained on mini-batches of tube anchors. Similar to the Tube-CNN, mini-batches are hierarchically sampled. The training batch contains tube anchors in total. Different to Tube-CNN, TPN supervision is class-agnostic, i.e. each tube anchor is labeled as either an object (positive) or background (negative). In our setting, at most of the training batch are positive samples.

We optimize the TPN model with an SGD algorithm. The model is only initialized with the CaffeNet [10] based network. The learning rate of SGD is set to , momentum – to , weight decay – to . We have reduced the learning rate by a factor of after saturation of the validation error.

Tube anchor design.

Our tube models are trained on two datasets: HollywoodHeads [36] and ImageNet VID [30]. While Tube-CNN hyper-parameters are the same on the two datasets, the ones of TPN are adapted to each dataset, i.e. different tube anchors are used.

As mentioned in Section 4.1, we use tube anchors with spatial location fixed, but with varying scales and aspect ratios. On both datasets, tube anchors have fixed scales. On HollywoodHeads dataset, because human heads are mostly square, we impose only one aspect ratio 1:1 to all tube anchors. On ImageNet VID dataset, as objects have more diverse shapes, the TPN achieves best object recall with aspect ratios 1:4, 1:2, 1:1, 2:1 and 4:1.

Appendix C Hard negative mining

As discussed in Section 5.2 and Section 5.3, our best Box-CNN and Tube-CNN models are trained with a few passes of hard negative mining. We here describe the procedure in an abstract manner. The term proposal stands for box proposal in case of Box-CNN, and tube proposal in case of Tube-CNN. The overlap ratio between proposal and ground-truth is defined separately for each model.

An iteration of hard negative mining consists of three steps: (1) a forward pass on a subset of input proposals to find hard negatives; (2) composition of several training batches with hard negatives and random positives; (3) training the network on the constructed batches. We define hard negatives as high-scoring false positives with no overlap to any ground-truth. The hard negative training batch contains positives and hard negatives at most. The rest of the batch is composed of negatives having overlap ratio to ground-truth in the range .


  • [1] R. Arandjelović, P. Gronat, A. Torii, T. Pajdla, and J. Sivic. Netvlad: Cnn architecture for weakly supervised place recognition. arXiv preprint arXiv:1511.07247, 2015.
  • [2] P. Bojanowski, F. Bach, I. Laptev, J. Ponce, C. Schmid, and J. Sivic. Finding actors and actions in movies. In ICCV, 2013.
  • [3] N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In CVPR, 2005.
  • [4] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009.
  • [5] M. Everingham, J. Sivic, and A. Zisserman. Hello! my name is… Buffy”–automatic naming of characters in tv video. In BMVC, 2006.
  • [6] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The Pascal visual object classes (VOC) challenge. IJCV, 2010.
  • [7] L. Galteri, L. Seidenari, M. Bertini, and A. Del Bimbo. Spatio-temporal closed-loop object detection. TIP, 2017.
  • [8] J. Gemert, M. Jain, E. Gati, C. G. Snoek, et al. APT: Action localization Proposals from dense Trajectories. BMVA Press, 2015.
  • [9] S. Gidaris and N. Komodakis. Locnet: Improving localization accuracy for object detection. In CVPR, 2016.
  • [10] R. Girshick. Fast R-CNN. In ICCV, pages 1440–1448, 2015.
  • [11] R. B. Girshick, J. Donahue, T. Darrell, and J. Malik. Region-based convolutional networks for accurate object detection and segmentation. TPAMI, 2016.
  • [12] R. B. Girshick, P. F. Felzenszwalb, and D. McAllester. Discriminatively trained deformable part models, release 5.
  • [13] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. In ECCV, 2014.
  • [14] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. CVPR, 2016.
  • [15] K. Kang, H. Li, T. Xiao, W. Ouyang, J. Yan, X. Liu, and X. Wang. Object detection in videos with tubelet proposal networks. In CVPR, 2017.
  • [16] K. Kang, H. Li, J. Yan, X. Zeng, B. Yang, T. Xiao, C. Zhang, Z. Wang, R. Wang, X. Wang, and W. Ouyang. T-CNN: Tubelets with convolutional neural networks for object detection from videos. arXiv preprint, 2016.
  • [17] K. Kang, W. Ouyang, H. Li, and X. Wang. Object detection from video tubelets with convolutional neural networks. In CVPR, 2016.
  • [18] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
  • [19] L. Leal-Taixé, C. Canton-Ferrer, and K. Schindler. Learning by tracking: Siamese cnn for robust target association. In CVPR Workshops, 2016.
  • [20] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
  • [21] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CPVR, 2015.
  • [22] D. Oneata, J. Revaud, J. Verbeek, and C. Schmid. Spatio-temporal object detection proposals. In ECCV, 2014.
  • [23] V. Ordonez, G. Kulkarni, and T. L. Berg. Im2text: Describing images using 1 million captioned photographs. In NIPS, 2011.
  • [24] P. O. Pinheiro, R. Collobert, and P. Dollár. Learning to segment object candidates. In NIPS, 2015.
  • [25] P. O. Pinheiro, T.-Y. Lin, R. Collobert, and P. Dollár. Learning to refine object segments. In ECCV, 2016.
  • [26] H. Pirsiavash, D. Ramanan, and C. C. Fowlkes. Globally-optimal greedy algorithms for tracking a variable number of objects. In CVPR, 2011.
  • [27] A. Prest, C. Leistner, J. Civera, C. Schmid, and V. Ferrari. Learning object class detectors from weakly annotated video. In CVPR, 2012.
  • [28] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, 2015.
  • [29] X. Ren. Finding people in archive films through tracking. In CVPR, 2008.
  • [30] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet large scale visual recognition challenge. IJCV, 2015.
  • [31] J. Shi and C. Tomasi. Good features to track. In CVPR, 1994.
  • [32] A. Shrivastava, A. Gupta, and R. Girshick. Training region-based object detectors with online hard example mining. arXiv preprint arXiv:1604.03540, 2016.
  • [33] S. Tang, B. Andres, M. Andriluka, and B. Schiele. Subgraph decomposition for multi-target tracking. In CVPR, 2015.
  • [34] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3D convolutional networks. In ICCV, 2015.
  • [35] J. R. Uijlings, K. E. van de Sande, T. Gevers, and A. W. Smeulders. Selective search for object recognition. IJCV, 2013.
  • [36] T.-H. Vu, A. Osokin, and I. Laptev. Context-aware CNNs for person head detection. In ICCV, 2015.