In data association based MOT, the tracking performance is heavily affected by the detection results. We implement our detector based on Faster R-CNN 
. In our implementation, the CNN model is fine-tuned from the VGG-16 on ImageNet. The additional training data includes ETHZ pedestrian dataset, Caltech pedestrian dataset  and the self-collected surveillance dataset (365653 boxes in 47556 frames). We adopt the multi-scale training strategy by randomly sampling a pyramid scale for each time. However, we only use a single scale and a single model during test. Moreover, we also use skip pooling  and multi-region  strategies to combine features at different scales and levels.
In considering the definition of MOTA in MOT16 , the sum of false negatives (FN) and false positives (FP) poses a large impact on the value of MOTA. In Table 1, we show that our detection optimization strategies lead to the significant decrease in the sum of FP and FN333We use detection score threshold 0.3 for Faster R-CNN and -1 for DPMv5 , labeling the ID of detection box with incremental integer, and evaluate FP and FN with MOT16 devkit..
|Faster R-CNN baseline||5384||47343||52727|
|Faster R-CNN + skip pooling||5410||46399||51809|
|Faster R-CNN + multi-region||4476||46738||51214|
|Faster R-CNN + both||8722||37865||46587|
2 Appearance Feature
The distance between appearance features is used for computing the affinity value in data association. The affinity value based on the ideal appearance feature should be large for persons of the same identity, and be small for persons of different identities. In our implementation, we extract the appearance feature using a network which is similar to GoogLeNet . The input size of our network is , and the kernel size of layer is instead of . The output layer is a fully connected layer which outputs the 128 dimensional feature. In the tracking phase, patches are first cropped according to the detection responses, and then resized to
for feature extraction. The cosine distance is used for measuring the appearance affinity.
For training, we collect a dataset which contains nearly 119 K patches from 19835 identities. Such a dataset consists of multiple person re-id datasets, including PRW , Market-1501 , VIPeR  and CUHK03 . We use the softmax and triplet loss jointly during training. The softmax loss guarantees the discriminative ability of the appearance feature, while the triplet loss ensures the cosine distance of the appearance features of the same identity to be small.
3 Online Tracker
We implement a simple online tracker, which uses Kalman filter for motion prediction and Kuhn-Munkres algorithm  for data association. The overall tracking procedure is described in Algorithm 1.
In the following, we introduce the affinity matrix construction, data association method, threshold value setting and tracking quality metric.
Affinity Matrix Construction. To construct an affinity matrix for the Kuhn-Munkres algorithm, we calculate the affinity between tracklets and detections. We combine motion, shape and appearance affinity as the final affinity. Specifically, the appearance affinity is calculated based on the appearance feature described in Section 2. Details of the affinity calculation are given below:
, and indicate appearance, motion and shape affinity between the detection and tracklet, respectively. We combine these affinities with weights and as the final affinity.
Data Association. The tracklets and new detections are associated using the Kuhn-Munkres algorithm. Since the Kuhn-Munkres algorithm attempts to yield the global optimal result, it may fail when some detections are missing. To this end, we use a two-stage matching strategy, which divides into high tracking quality set and low quality set . The matching is first performed between and , and then performed between and .
Threshold Value Setting. On line 2 of Algorithm 1, we introduce to divide into high and low tracking quality set. The strategy is intuitive: we mark a tracklet with high flag whose tracking quality is higher than , other tracklets will be mark as low. On line 4, we use to mark the association to be success or fail based on the affinity value. On line 7, we use as a threshold to drop a tracklet which is lost for more than frames.
Tracking Quality Metric. Tracking quality is designed to measure whether a object is tracking well or not. We use following formula to define tracking quality:
where , with the form , is a set that contains every success association couple in history.
4 Offline Tracker
Our offine tracker an improved version of HT  while based on K-Dense Neighbors . It is more robust and efficient than HT in handling the complex tracking scenarios. The overall procedure of the tracker is described in Algorithm 2.
We make the following improvements over HT .
Appearance Representation. To construct the affinity matrix for the dense neighbors (DN) search, we need to calculate three affinities, _meaning:NTF . i.e _catcode:NTF a i.e. i.e., appearance, motion, and smoothness affinity. Among these three affinities, the appearance affinity is the most important one and we use the CNN based feature described in Section. 2, instead of the hand-crafted feature in .
Big Target. A scenario that HT  does not work well is the mixture of small and big targets. The reason is that the motion and smoothness affinities are unreliable for the big targets. Such unreliability is caused by the unsteady detection responses of the big targets. We introduce two thresholds, and , regarding the object scale to deal with this challenge, _meaning:NTF . i.e _catcode:NTF a i.e. i.e., for preventing associating detection responses from very different scale, and for determining whether to reduce the weights of motion and smoothness affinity. Specifically, if the ratio of the detection response scale and the target scale is less than , such a detection response will not be associated with the target. If the ratio of the detection response height and the image height is greater than , such a detection response will not be associated with the target. Both and are set as 0.5.
Algorithm Efficiency. HT is slow in handling the long tracking sequence where there exist plenty of targets. Among the steps in the algorithm, the step of DN search is the most time-consuming. To be more specific, the larger an affinity matrix, the longer time it will take to perform the DN search. Thus, we abandon the high-order information  when constructing the affinity matrix, which significantly reduces the matrix dimensions and improves the algorithm efficiency.
Our online and offline tracker are not learning based algorithm. We only tuning detection score threshold on train set and apply it to its similar scene from test set. For evaluation and submission, 0.1 is set for MOT16-03 and MOT16-04 due to high precision of detection result (03 and 04 are both surveillance scene, which is quite easy while our detector have been trained by self-collected surveillance dataset), and 0.3 is set for other sequences.
For both online555we use 0.5 for , 1.5 for , 1.2 for ,0.5 for , 0.4 for and 100 frames for . and offline tracker, we compare our detector with the official detector, and compare our feature with default CNN feature. The comparison results on MOT16  train set are listed in Table 2 and Table 3, respectively. Note that our detector leads to much better results in MT, ML, FP and FN, and our feature helps reduce both IDS and FM.
|Det. and Feat.||MT||ML||FP||FN||IDS||FM||MOTA||MOTP|
|DPMv5 + Our Feat.||7.54%||52.42%||6197||70952||784||2697||29.4||77.2|
|Our Det. + GoogLeNet Feat.||31.72%||16.25%||3207||35472||1541||2235||63.6||82.6|
|Our Det. and Feat.||37.33%||14.70%||3497||34241||716||1973||65.2||82.4|
|Det. and Feat.||MT||ML||FP||FN||IDS||FM||MOTA||MOTP|
|DPMv5 + Our Feat.||10.64%||52.80%||27238||63443||1540||1853||16.5||77.4|
|Our Det. + GoogLeNet Feat.||13.93%||60.93%||1258||58213||1350||2196||44.9||85.0|
|Our Det. and Feat.||37.52%||17.60%||2762||33327||462||717||66.9||83.3|
6 ECCV 2016 Challenge Results.
|Our Online Tracker||33.99%||20.82%||5061||55914||805||3093||66.1||79.5|
|Our Offline Tracker||40.97%||18.97%||11479||45605||933||1093||68.2||79.4|
Our ECCV 2016 Challenge results are listed in Table 4
. Obviously, both our online and offline trackers outperform the state-of-the-art approaches by a large margin. Note that our offline tracker achieves the best performance in FN. However, its performance in FP is moderate, due to the interpolation module.
In this submission, we take many efforts to get high performance detection and deep learning based appearance feature. We show that they lead to the state-of-the-art multiple object tracking results, even with very simple online tracker. One observation is that with high performance detection and appearance feature, the state-of-the-art offline tracker does not have expected advantages over the much simpler online one. This observation is not reported in many current MOT papers, which often use detections that are not good enough. We make our detections and deep learning based re-ID features on MOT2016 publicly available, and hope that they can help more sophisticated trackers to get better performance.
Bell, S., Zitnick, C.L., Bala, K., Girshick, R.B.: Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks. CoRR (2015)
-  Dollár, P., Wojek, C., Schiele, B., Perona, P.: Pedestrian detection: A benchmark. In: CVPR (2009)
-  Du, D., Qi, H., Li, W., Wen, L., Huang, Q., Lyu, S.: Online deformable object tracking based on structure-aware hyper-graph. TIP (2016)
-  Ess, A., Leibe, B., Schindler, K., Gool, L.J.V.: A mobile vision system for robust multi-person tracking. In: CVPR (2008)
-  Gidaris, S., Komodakis, N.: Object detection via a multi-region and semantic segmentation-aware CNN model. In: ICCV (2015)
-  Kalman, R.E.: A new approach to linear filtering and prediction problems. Journal of Basic Engineering (1960)
-  Kuhn, H.W.: The hungarian method for the assignment problem. Naval research logistics quarterly (1955)
-  Li, W., Zhao, R., Xiao, T., Wang, X.: Deepreid: Deep filter pairing neural network for person re-identification. In: CVPR (2014)
-  Li, W., Wen, L., Chuah, M.C., Lyu, S.: Category-blind human action recognition: A practical recognition system. In: ICCV (2015)
-  Li, W., Wen, L., Chuah, M.C., Zhang, Y., Lei, Z., Li, S.Z.: Online visual tracking using temporally coherent part cluster. In: WACV (2015)
-  Liu, H., Yang, X., Latecki, L.J., Yan, S.: Dense neighborhoods on affinity graph. IJCV (2012)
-  Milan, A., Leal-Taixé, L., Reid, I.D., Roth, S., Schindler, K.: MOT16: A benchmark for multi-object tracking. CoRR (2016)
Prosser, B., Zheng, W., Gong, S., Xiang, T.: Person re-identification by support vector ranking. In: BMVC (2010)
-  Ren, S., He, K., Girshick, R.B., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: NIPS (2015)
-  Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S.E., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: CVPR (2015)
-  Wen, L., Li, W., Yan, J., Lei, Z., Yi, D., Li, S.Z.: Multiple target tracking based on undirected hierarchical relation hypergraph. In: CVPR (2014)
-  Zamir, A.R., Dehghan, A., Shah, M.: Gmcp-tracker: Global multi-object tracking using generalized minimum clique graphs. In: ECCV (2012)
-  Zheng, L., Zhang, H., Sun, S., Chandraker, M., Tian, Q.: Person re-identification in the wild. CoRR (2016)