Online video repositories like Youtube, Dailymotion etc have been experiencing an explosion of user-generated videos. Such videos are often shot/recorded from the television by users, and uploaded onto these sites. They have very little metadata like dialogue scripts, or a textual summary/representation of the content. When an user searches these repositories by keywords, (s)he is suggested hundreds of videos, out of which (s)he may choose a small number for viewing. This has given rise to the topic of Video Summarization , which aims to provide the user a short but comprehensive summary of the video. However, the current state-of-the-art mostly provides a few keyframes as summary, which may not have much semantic significance. The high-level semantic information of videos that is most important to users is carried by entities- such as persons or other objects. With the recent progress in object detection   in single images and videos, it is now possible to have a high-level representation of videos in terms of such entities. One effective way of summarization is to have a list of entities that appear frequently in a video. Further, an user may want to watch only a part of a video, for example wherever a particular person (or set of persons) appears, which motivate the tasks of Entity Discovery and Entity-driven Summarization  of videos.
The problem of automated discovery of persons from videos along with all their occurrences has attracted a lot of interest  in video analytics. Existing attempts try to leverage meta-data such as scripts  and hence do not apply to videos available on the wild, such as TV-Series episodes uploaded by viewers on Youtube (which have no such meta-data). In this paper, we pose this problem as tracklet clustering, as done in . Our goal is to design algorithms for tracklet clustering which can work on long videos. Tracklets are formed by detections of an entity (say a person) from a short contiguous sequence of 10-20 video frames. They have complex spatio-temporal properties. We should be able to handle any type of entity, not just person. Given a video in the wild it is unlikely that the number of entities will be known, so the method should automatically adapt to unknown number of entities. To this end we advocate a Bayesian non-parametric
clustering approach to Tracklet clustering and study its effectiveness in automated discovery of entities with all their occurrences in long videos. The main challenges are in modeling the spatio-temporal properties. To the best of our knowledge this problem has not been studied either in Machine Learning or in Computer Vision community.
To explain the spatio-temporal properties we introduce some definitions. A track is formed by detecting entities (like people’s faces) in each video frame, and associating detections across a contiguous sequence of frames (typically a few hundreds in a TV series) based on appearance and spatio-temporal locality. Each track corresponds to a particular entity, like a person in a TV series. Forming long tracks is often difficult, especially if there are multiple detections per frame. This can be solved hierarchically, by associating the detections in a short window of frames (typically 10-20) to form tracklets  and then linking the tracklets from successive windows to form tracks. The short-range association of tracklets to form tracks is known as tracking. But in a TV series video, the same person may appear in different (non-contiguous) parts of the video, and so we need to associate tracklets on a long-range basis also (see Figure 1). Moreover the task is complicated by lots of false detections which act as spoilers. Finally, the task becomes more difficult on streaming videos, where only one pass is possible over the sequence.
A major cue for this task comes from a very fundamental property of videos: Temporal Coherence(TC). This property manifests itself at detection-level as well as tracklet-level; at feature-level as well as at semantic-level. At detection-level this property implies that the visual features of the detections (eg. appearance of an entity) are almost unchanged across a tracklet (See Fig. 2). At tracklet-level it implies that spatio-temporally close (but non-overlapping) tracklets are likely to belong to the same entity (Fig. 3). Additionally, overlapping tracklets (that span the same frames), cannot belong to the same entity. A tracklet can be easily represented as all the associated detections are very similar (due to detection-level TC). Such representation is not easy for a long track where the appearances of the detections may gradually change.
Contribution Broadly, this paper has two major contributions: it presents the first Bayesian nonparametric models for TC in videos, and also the first entity-driven approach to video modelling. To these ends, we explore tracklet clustering, an active area of research in Computer Vision, and advocate a Bayesian non-parametric(BNP) approach for it. We apply it to an important open problem: discovering entities (like persons) and all their occurrences from long videos, in absence of any meta-data, e.g. scripts. We use a simple and generic representation leading to representing a video by a matrix, whose columns represent individual tracklets (unlike other works which represent an individual detection by a matrix column, and then try to encode the tracklet membership information). We propose Temporally Coherent-Chinese Restaurant process(TC-CRP), a BNP prior for enforcing coherence on the tracklets. Our method yields a superior clustering of tracklets over several baselines especially on long videos. As an advantage it does not need the number of clusters in advance. It is also able to automatically filter out false detections, and perform the same task on streaming videos, which are impossible for existing methods of tracklet clustering. We extend TC-CRP to the Temporally Coherent Chinese Restaurant Franchise (TC-CRF), that jointly models short video segments and further improves the results. We show that the proposed methods can be applied to entity-driven video summarization, by selecting a few representative segments of the video in terms of the discovered entities.
Ii Problem Definition
In this section, we elaborate on our task of tracklet clustering for entity discovery in videos.
In this work, given a video, we fix beforehand the type of entity (eg. person/face, cars, planes, trees) we are interested in, and choose an appropriate detector like  , which is run on every frame of the input video. The detections in successive frames are then linked based on spatial locality, to obtain tracklets. At most detections from contiguous frames are linked like this. The tracklets of length less than are discarded, hence all tracklets consist of detections. We restrict the length of tracklets so that the appearance of the detections remain almost unchanged (due to detection-level TC), which facilitates tracklet representation. At we work with the individual detections.
We represent a detection by a vector of dimension . This can be done by downscaling a rectangular detection to square and then reshaping it to a -dimensional vector of pixel intensity values (or some other features if deemed appropriate). Each tracklet is a collection of detections . Let the tracklet be represented by . So finally we have vectors (: number of tracklets).
The tracklets can be sorted topologically based on their starting and ending frame indices. Each tracklet has a predecessor tracklet and a successor tracklet . Also each tracklet has a conflicting set of tracklets which span frame(s) that overlap with the frames spanned by . Each detection (and tracklet) is associated with an entity, which are unknown in number, but presumably much less than the number of detections (and tracklets). These entities also are represented by vectors, say . Each tracklet is associated with an entity indexed by , i.e. .
Ii-B Entity Discovery
Let each video be represented as a sequence of -dimensional vectors along with the set . We aim to learn the vectors and the assignment variables . In addition, we have constraints arising out of temporal coherence and other properties of videos. Each tracklet is likely to be associated with the entity that its predecessor or successor is associated with, except at shot/scene changepoints. Moreover, a tracklet cannot share an entity with its conflicting tracklets , as the same entity cannot occur twice in the same frame. This notion is considered in relevant literature  . Mathematically, the constraints are:
Learning a -vector is equivalent to discovering an entity, and its associated tracklets are discovered by learning the set . These constraints give the task a flavour of non-parametric constrained clustering with must-link and don’t-link constraints.
Finally, the video frames can be grouped into short segments, based on the starting frame numbers of the tracklets. Consider two successive tracklets and , with starting frames and . If the gap between frames and is larger than some threshold, then we consider a new temporal segment of the video starting from , and add to a list of changepoints (CP). The beginning of a new temporal segment does not necessarily mean a scene change, the large gap between frames and may be caused by failure of detection or tracklet creation. The segment index of each tracklet is denoted by .
Iii Related Work
Person Discovery in Videos is a task which has recently received attention in Computer Vision. Cast Listing  is aimed to choose a representative subset of the face detections or face tracks in a movie/TV series episode. Another task is to label all the detections in a video, but this requires movie scripts  or labelled training videos having the same characters . Scene segmentation and person discovery are done simultaneously using a generative model in , but once again with the help of scripts. An unsupervised version of this task is considered in , which performs face clustering in presence of spatio-temporal constraints as already discussed. For this purpose they use a Markov Random Field, and encode the constraints as clique potentials. Another recent approach to face clustering is  which incorporates some spatio-temporal constraints into subspace clustering.
Tracklet Association Tracking is a core topic in computer vision, in which a target object is located in each frame based on appearance similarity and spatio- temporal locality. A more advanced task is multi-target tracking , in which several targets are present per frame. A tracking paradigm that is particularly helpful in multi-target tracking is tracking by detection , where object-specific detectors like  are run per frame (or on a subset of frames), and the detection responses are linked to form tracks. From this came the concept of tracklet  which attempts to do the linking hierarchically. This requires pairwise similarity measures between tracklets. Multi-target tracking via tracklets is usually cast as Bipartite Matching, which is solved using Hungarian Algorithm. Tracklet association and face clustering are done simultaneously in  using HMRF. The main difference of face/tracklet clustering and person discovery is that, the number of clusters to be formed is not known in the latter.
Independent of videos, Constrained Clustering is itself a field of research. Constraints are usually must-link and don’t-link, which specify pairs which should be assigned the same cluster, or must not be assigned the same cluster. A detailed survey is found in . The constraints can be hard  or soft/probabilistic 
. Constrained Spectral Clustering has also been studied recently , which allow constrained clustering of datapoints based on arbitrary similarity measures.
All the above methods suffer from a major defect- the number of clusters needs to be known beforehand. A way to avoid this is provided by Dirichlet Process
, which is able to identify the number of clusters from the data. It is a mixture model with infinite number of mixture components, and each datapoint is assigned to one component. A limitation of DP is that it is exchangeable, and cannot capture sequential structure in the data. For this purpose, a Markovian variation was proposed: Hierarchical Dirichlet Process- Hidden Markov Model (HDP-HMM). A variant of this is thesticky HDP-HMM (sHDP-HMM) , which was proposed for temporal coherence in speech data for the task of speaker diarization, based on the observation that successive datapoints are likely to be from the same speaker and so should be assigned to the same component. Another Bayesian nonparametric approach for sequential data is the Distance-Dependent Chinese Restaurant Process (DDCRP) 
, which defines distances between every pair of datapoints, and each point is linked to another with probability proportional to such distances. A BNP model for subset selection is Indian Buffet Process (IBP), a generative process for a sequence of binary vectors. This has been used for selecting a sparse subset of mixture components (topics) in Focussed Topic Modelling  as the Compound Dirichlet Mixture Model.
Finally, Video Summarization has been studied for a few years in the Computer Vision community. The aim is to provide a short but comprehensive summary of videos. This summary is usually in the form of a few keyframes, and sometimes as a short segment of the video around these keyframes. A recent example is  which models a video as a matrix, each frame as a column, and each keyframe as a basis vector, in terms of which the other columns are expressed. A more recent work  considers a kernel matrix to encode similarities between pairs of frames, uses it for Temporal Segmentation of the video, assigns an importance label to each of these segments using an SVM (trained from segmented and labelled videos), and creates the summary with the important segments. However, such summaries are in terms of low-level visual features, rather than high-level semantic features which humans use. An attempt to bridge this gap was made in , which defined movie scenes and summaries in terms of characters. This work used face detections along with movie scripts for semantic segmentation into shots and scenes, which were used for summarization.
Iv Generative Process for Tracklets
We now explain our Bayesian Nonparametric model TC-CRP to handle the spatio-temporal constraints (Eq II-B) for tracklet clustering, and describe a generative process for videos based on tracklets.
Iv-a Bayesian Nonparametric Modelling
In Section II, we discussed the vectors each of which represent an entity. In this paper we consider a Bayesian approach with Gaussian Mixture components to account for the variations in visual features of the detections, say face detections of a person. As already mentioned, number of components is not known beforehand, and must be discovered from the data. That is why we consider nonparametric Bayesian modelling. Also, as we shall see, this route allows us to elegantly model the temporal coherence constraints. In this approach, we shall represent entities as mixture components and tracklets as draws from such mixture components.
Dirichlet Process 
has become an important clustering tool in recent years. Its greatest strength is that unlike K-means, it is able to discover the correct number of clusters. Dirichlet Process is a distribution over distributions over a measurable space. A discrete distributionis said to be distributed as over space if for every finite partition of as , the quantity is distributed as , where is a scalar called concentration parameter, and is a distribution over called Base Distribution. A distribution is a discrete distribution, with infinite support set , which are draws from , called atoms.
Iv-B Modeling Tracklets by Dirichlet Process
We consider to be a multivariate Gaussian with parameters and . Each atom corresponds to an entity (eg. a person). The generative process for the set is then as follows:
Here is an atom. is a tracklet representation corresponding to the entity, and its slight variation from (due to effects like lighting and pose variation) is modelled using .
Using the constructive definition of Dirichlet Process, called the Stick-Breaking Process , the above process can also be written equivalently as
Here is a distribution over integers, and is an integer that indexes the component corresponding to the tracklet . Our aim is to discover the values , which will give us the entities, and also to find the values , which define a clustering of the tracklets. For this purpose we use collapsed Gibbs Sampling, where we integrate out the in Equation 2 or in Equation IV-B. The Gibbs Sampling Equations and are given in . For ,
Here, is the data likelihood term. We focus on the part to model TC.
Iv-C Temporally Coherent Chinese Restaurant Process
In the generative process (Equation IV-B) all the are drawn IID conditioned on . Such models are called Completely Exchangeable. This is, however, often not a good idea for sequential data such as videos. In Markovian Models like sticky HDP-HMM, is drawn conditioned on and . In case of DP, the independence among -s is lost on integrating out . After integration the generative process of Eq IV-B can be redefined as
The predictive distribution for for Dirichlet Process is known as Chinese Restaurant Process (CRP). It is defined as where is the number of times the value is taken in the set .
We now modify CRP to handle the Spatio-temporal cues (Eq II-B) mentioned in the previous section. In the generative process, we define with respect to , similar to the Block Exchangeable Mixture Model as defined in . Here, with each we associate a binary change variable . If then , i.e the tracklet identity is maintained. But if , a new value of is sampled. Note that every tracklet has a temporal predecessor
. However, if this predecessor is spatio-temporally close, then it is more likely to have the same label. So, the probability distribution of change variableshould depend on this closeness. In TC-CRP, we use two values ( and ) for the Bernoulli parameter for the change variables. We put a threshold on the spatio-temporal distance between and , and choose a Bernoulli parameter for based on whether this threshold is exceeded or not. Note that maintaining tracklet identity by setting is equivalent to tracking.
Several datapoints (tracklets) arise due to false detections. We need a way to model these. Since these are very different from the Base mean , we consider a separate component with mean and a very large covariance , which can account for such variations. The Predictive Probability function(PPF) for TC-CRP is defined as follows:
where is the set of values of for the set of tracklets that overlap with , and is the number of points () where and . The first rule ensures that two overlapping tracklets cannot have same value of . The second rule accounts for false tracklets. The third and fourth rules define a CRP restricted to the changepoints where . The final tracklet generative process is as follows:
where is the PPF for TC-CRP, defined in Eq IV-C.
Inference in TC-CRP can be performed easily through Gibbs Sampling. We need to infer , and . As and are coupled, we sample them in a block for each as done in . If and , then we must have and . If and , then , and is sampled from . In case and , then with probability proportional to . If then if , and otherwise. If then is governed by TC-CRP. For sampling
, we make use of the Conjugate Prior formula of Gaussians, to obtain the Gaussian posterior with meanwhere , and
. Finally, we update the hyperparametersand after every iteration, based on the learned values of
, using Maximum Likelihood estimate., can also be updated, but in our implementation we set them to and respectively, based on empirical evaluation on one held-out video. The threshold was also similarly fixed.
V Generative Process for Video Segments
In the previous section, we considered the entire video as a single block, as the TCCRP PPF for any tracklet involves -values from all the previously seen tracklets throughout the video. However, this need not be very accurate, as in a particular part of the video some mixture components (entities) may be more common than anywhere else, and for any , may depend more heavily on the -values in temporally close tracklets than the ones far away. This is because, a TV-series video consists of temporal segments like scenes and shots, each characterized by a subset of persons (encoded by binary vector ). The tracklets attached to a segment cannot be associated with persons not listed by . To capture this notion we propose a new model: Temporally Coherent Chinese Restaurant Franchise (TC-CRF) to model a video temporally segmented by (see Section II).
V-a Temporally Coherent Chinese Restaurant Franchise
Chinese Restaurant Process is the PPF associated with Dirichlet Process. Hierarchical Dirichlet Process (HDP)  aimed at modelling grouped data sharing same mixture components. It assumes a group-specific distribution for every group . The generative process is:
where datapoint belongs to the group . The PPF corresponding to this process is obtained by marginalizing the distributions and , and is called the Chinese Restaurant Franchise process, elaborated in . In our case, we can modify this PPF once again to incorporate TC, analogously to TC-CRP, to have Temporally Coherent Chinese Restaurant Franchise (TC-CRF) Process. In our case, a group corresponds to a temporal segment, and as already mentioned, we want a binary vector , which indicates the components that are active in segment . But HDP assumes that all the components are shared by all the groups, i.e. any particular component can be sampled in any of the groups. We can instead try sparse modelling by incorporating into the model, as done in  for Focussed Topic Models. For this purpose we put an IBP  prior on the variables, where where is the number of times component has been sampled in all scenes before , and . The TC-CRF PPF is then as follows:
where , the index of the temporal segment to which the datapoint belongs. Based on TC-CRF, the generative process of a video, in terms of temporal segments and tracklets, is given below:
where is the PPF for TC-CRF, and is the temporal segment index associated with tracklet .
Inference in TC-CRF can also be performed through Gibbs Sampling. We need to infer the variables , , and the components . In segment , for a datapoint where , a component may be sampled with , which is the number of times has been sampled within the same segment. If has never been sampled within the segment but has been sampled in other segments, , where is the number of segments where has been sampled (Corresponding to according to IBP), and is the CRP parameter for sampling a new component. Finally, a completely new component may be sampled with probability proportional to . Note that .
Vi Relationship with existing models
TC-CRP draws inspirations from several recently proposed Bayesian nonparametric models, but is different from each of them. It has three main characterestics: 1) Changepoint-variables 2) Temporal Coherence and Spatio-temporal cues 3) Separate component for non-face tracklets. The concept of changepoint variable was used in Block-exchangeable Mixture Model , which showed that this significantly speeds up the inference. But in BEMM, the Bernoulli parameter of changepoint variable depends on while in TC-CRP it depends on . Regarding spatio-temporal cues, the concept of providing additional weightage to self-transition was introduced in sticky HDP-HMM , but this model does not consider change-point variables. Moreover, it uses a transition distribution for each mixture component , which increases the model complexity. Like BEMM  we avoid this step, and hence our PPF (Eq IV-C) does not involve . DDCRP  defines distances between every pair of datapoints, and associates a new datapoint with one of the previous ones () based on this distance. Here we consider distances between a point and its predecessor only. On the other hand, DDCRP is unrelated to the original DP-based CRP, as its PPF does not consider : the number of previous datapoints assigned to component . Hence our method is significantly different from DDCRP. Finally, the first two rules of TC-CRP PPF are novel.
TC-CRF is inspired by HDP . However, once again there are three differences mentioned above hold good. In addition, the PPF of TC-CRF itself is different from Chinese Restaurant Franchise as described in . The original CRF is defined in terms of two concepts: tables and dishes, where tables are local to individual restaurants (data groups) while dishes (mixture components) are global, shared across restaurants (groups). Also individual datapoints are assigned mixture components indirectly, through an intermediate assignment of tables. The concept of table, which comes due to marginalization of group-specific mixture distributions, results in complex book-keeping, and the PPF for datapoints is difficult to define. Here we avoid this problem, by skipping tables and directly assigning mixture components to datapoints in Eq V-A. Inspiration of TC-CRF is also drawn from IBP-Compound Dirichlet Process . But the inference process of  is complex, since the convolution of the DP-distributed mixture distribution and the sparse binary vector is difficult to marginalize by integration. We avoid this step by directly defining the PPF (Eq V-A) instead of taking the DP route. This approach of directly defining the PPF was taken for DD-CRP  also.
Vii Experiments on Person Discovery
One particular entity discovery task that has recently received a lot of attention is person discovery from movies/ TV series. We carried out extensive experiments for person discovery on TV series videos of various lengths. We collected three episodes of The Big Bang Theory (Season 1). Each episode is 20-22 minutes long, and has 7-8 characters (occurring in at least 100 frames). We also collected 6 episodes of the famous Indian TV series “The Mahabharata” from Youtube. Each episode of this series is 40-45 minutes long, and have 15-25 prominent characters. So here, each character is an entity. These videos are much longer than those studied in similar works like , and have more characters. Also, these videos are challenging because of the somewhat low quality and motion blur. Transcripts or labeled training sets are unavailable for all these videos. As usual in the literature , we represent the persons with their faces. We obtained face detections by running the OpenCV Face Detector on each frame separately. As described in Section II the face detections were all converted to grayscale, scaled down to , and reshaped to form -dimensional vectors. We considered tracklets of size and discarded smaller ones. The dataset details are given in Table 1.
Vii-a Alternative Methods
A recent method for face clustering using track information is WBSLRR  based on Subspace Clustering. Though in  it is used for clustering detections rather than tracklets, the change can be made easily. Apart from that, we can use Constrained Clustering as a baseline, and we choose a recent method . TC and frame conflicts are encoded as must-link and don’t-link constraints respectively. A big problem is that the number of clusters to be formed is unknown. For this purpose, we note that the tracklet matrix formed by juxtaposing the tracklet vectors should be approximately low-rank because of the similarity of spatio-temporally close tracklet vectors. Such representation of a video as a low-rank matrix has been attempted earlier  . We can find a low-rank representation of the tracklet matrix by any suitable method, and use the rank as the number of clusters to be formed in spectral clustering. We found that, among these the best performance is given by Sparse Bayesian Matrix Recovery (SBMR) . Others are either too slow (BRPCA ), or recover matrices with ranks too low (OPTSPACE ) or too high (RPCA ). Finally, we compare against another well-known sequential BNP method- the sticky HDP-HMM .
Vii-B Performance Measures
The task of entity discovery with all their tracks is novel and complex, and has to be judged by suitable measures. We discard the clusters that have less than 10 assigned tracklets. It turns out that the remaining clusters cover about of all the tracklets. Further, there are some clusters which have mostly ( or more) false (non-entity) tracklets. We discard these from our evaluation. We call the remaining clusters as significant clusters. We say that a cluster is “pure” if at least of the tracklets assigned to it belong to any one person (say Sheldon for a BBT video, or Arjuna for a Mahabharata video). We also declare that the cluster and its corresponding mixture component corresponds to the person . Also, then is considered to be discovered. The threshold of purity was set to because we found this roughly the minimum purity needed to ensure that a component mean is visually recognizable as the entity (after reshaping to ) (See Fig. 4, 5). We measure the Purity: fraction of significant clusters that are pure, i.e. correspond to some entity. We also measure Entity Coverage: the number of persons (entity) with at least 1 cluster (at least 10 tracklets) corresponding to them. Next, we measure Tracklet Coverage: the fraction of tracklets that are assigned to pure clusters. Effectively, these tracklets are discovered, and the ones assigned to impure clusters are lost.
The results on the three measures discussed above are shown in Tables 2,3,4. In terms of the three measures, TC-CRF is usually the most accurate, followed by TC-CRP, and then sHDP-HMM. This demonstrates that BNP methods are more suitable to the task. The constrained spectral clustering-based method is competitive on the purity measure, but fares very poorly in terms of tracklet coverage. This is because, it forms many small pure clusters, and a few very large impure clusters which cover a huge fraction of the tracklets. Thus, a large number of tracklets are lost.
It may be noted that the number of significant clusters formed is a matter of concern, especially from the user’ perspective. A small number of clusters allow him/her to get a quick summary of the video. Ideally there should be one cluster per entity, but that is not possible due to the significant appearance variations (See Figure 5). The number of clusters formed per video by the different methods is indicated in Table 2. It appears that none of the methods have any clear advantage over the others in this regard. In the above experiments, we used tracklets with size . We varied this number and found that, for and even (dealing with detections individually), the performance of TC-CRF, TC-CRP and sHDP-HMM did not change significantly. On the other hand, the matrix returned by SBMR had higher rank (120-130 for ) as the number of tracklets increased.
|BBTs1e1||0.88 (48)||0.75 (36)||0.84 (44)||0.67 (48)||0.73 (45)|
|BBTs1e3||0.88 (50)||0.83 (40)||0.76 (37)||0.80 (15)||0.67 (43)|
|BBTs1e4||0.93 (40)||0.89 (36)||0.83 (29)||0.77 (31)||0.71 (41)|
|Maha22||0.91 (67)||0.87 (69)||0.86 (74)||0.94 (44)||0.83 (79)|
|Maha64||0.95 (113)||0.92 (105)||0.91 (97)||0.85 (88)||0.75 (81)|
|Maha65||0.97 (95)||0.89 (85)||0.90 (89)||0.86 (76)||0.82 (84)|
|Maha66||0.91 (76)||0.96 (73)||0.95 (80)||0.87 (84)||0.81 (81)|
|Maha81||0.89 (91)||0.89 (88)||0.84 (95)||0.87 (84)||0.74 (78)|
|Maha82||0.92 (52)||0.88 (50)||0.86 (58)||0.78 (63)||0.83 (64)|
Vii-D Online Inference
We wanted to explore the case of streaming videos, where the frames appear sequentially and old frames are not stored. This is the online version of the problem, the normal Gibbs Sampling will not be possible. For each tracklet , we will have to infer and based on , and the -vectors learnt from . Once again, is sampled as a block as above, and the term follows from the TC-CRP PPF (Eq IV-C). The same thing can be done for TC-CRF also. Instead of drawing one sample per data-point, an option is to draw several samples and consider the mode. In the absence of actual streaming datasets we performed the single-pass inference (Sec VII-D) on two of the videos from each set- Mahabharata and Big Bang Theory. We used the same performance measures as above. The existing tracklet clustering methods discussed in Sec VII-A are incapable in the online setting, and sticky HDP-HMM is the only alternative. The results are presented in Table 5, which show TC-CRP to be doing the best on the Mahabharata videos and TC-CRF on the Big Bang Theory ones. Notably, the figures for TC-CRP and TC-CRF in the online experiment are not significantly lower than those in the offline experiment (except one or two exceptions), unlike sHDP-HMM. This indicates that the proposed methods converge quickly, and so are more efficient offline.
|Purity||0.86 (56)||0.89(79)||0.84 (82)|
|Purity||0.87 (39)||0.73 (33)||0.50 (14)|
|Purity||0.92 (45)||0.88 (32)||0.75(28)|
Vii-E Outlier Detection / Discovery of False Tracklets
Face Detectors such as  are trained on static images, and applied on the videos on per-frame basis. This approach itself has its challenges , and the complex videos we consider in our experiments do not help matters. As a result, there is a significant number of false (non-face) detections, many of which occur in successive frames and hence get linked as tracklets. Identifying such junk tracklets not only helps us to improve the quality of output provided to the users, but may also help to adapt the detector to the new domain, by retraining with these new negative examples, as proposed in .
We make use of the fact that false tracklets are relatively less in number (compared to the true ones), and hence at least some of them can be expected to deviate widely from the mean of the tracklet vectors. This is taken care of in the TC-CRP tracklet model, through the component
that has very high variance, and hence is most likely to generate the unusual tracklets. We set this varianceas , where . The tracklets assigned are reported to be junk by our model. It is expected that high will result in lower recall but higher precision (as only the most unusual tracklets will go to this cluster), and low will have the opposite effect. We study this effect on two of our videos- Maha65 and Maha81 (randomly chosen) in Table 6 (See Fig. 7 for illustration). As baseline, we consider K-means or spectral clustering of the tracklet vectors. We may expect that one of the smaller clusters should contain mostly the junk tracklets, since faces are roughly similar (even if from different persons) and should be grouped together. However, for different values of (2 to 10) we find that the clusters are roughly of the same size, and the non-face tracklets are spread out quite evenly. Results are reported for the best ( for both). Note that because of the large number of tracklets (Table I) it is difficult to count the total number of non-face ones. So for measuring recall, we simply mention the number of non-face tracklets recovered (recall*), instead of the fraction
. It is clear that TC-CRP significantly outperforms clustering on both precision and recall*.
Vii-F Evaluation of TC enforcement
The aim of TC-CRP and TC-CRF is to encourage TC at the semantic level, that spatio-temporally close but non-overlapping tracklets should belong to the same entity. In the Bayesian models like sHDP-HMM, TC-CRP and TC-CRF, these cues are modelled with probability distributions, in WBSLRR with convex regularization and in constrained clustering they are encoded as hard constraints. We now evaluate how well the different methods have been able to enforce these cues. We create ground-truth tracks by linking the tracklets which are spatio-temporally close to each other (with respect to the chosen threshold in the generative process), and belong to the same entity. All the tracklets in each ground-truth track should be assigned to the same cluster. This is the task of tracklet linking. We measure what fraction of the these ground-truth tracks have been assigned entirely to single clusters by the different methods. We do not compare SBMR+ConsClus, since it uses hard constraints. The results are shown in Table 7. We find that TC-CRF is the best once again, followed by TC-CRP and sHDP-HMM. WBSLRR has significantly poorer performance, though it springs a surprise on BBTs1e1.
|Cars||0.94 (35)||0.92 (12)||1.00 (54)||0.24 (21)|
|Aeroplanes||0.95 (43)||0.87 (15)||0.84 (44)||0.21 (24)|
Viii Discovery of Non-person Entities
To emphasize the fact that our methods are not restricted to faces or persons, we used two short videos-one of cars and another of aeroplanes. The cars video consisted of 5 cars of different colors, while the aeroplanes video had 6 planes of different colors/shapes. These were created by concatenating shots of different cars/planes in the Youtube Objects datasets . The objects were detected using the Object-specific detectors . Since here the color is the chief distinguishing factor, we scaled the detections down to and reshaped them separately in the 3 color channels to get 2700-dimensional vectors. Here was used, as these videos are much shorter, and using long tracklets would have made the number of data-points too low. Both videos have 750 frames. The Cars video has 694 detections, and the Aeroplanes video has 939 detections. The results are shown in Table 8. Once again, TC-CRP does well.
Ix Semantic Video Summarization
In this section, we discuss how the above results on entity discovery can be used to obtain a sematic summary of the video. For this purpose we consider two approaches: entity-based and shot-based.
Ix-a Entity-based Summarization
The process of entity discovery via tracklet clustering results in formation of clusters.In case of the Bayesian methods like TC-CRF, TC-CRP and sHDP-HMM, each cluster can be represented by the mean vector of the corresponding mixture component. In case of non-Bayesian approaches like SBMR+Consclus and WBSLRR, it is possible to compute the cluster centers as the mean of the tracklet vectors assigned to each cluster. Each cluster vector can be reshaped to form a visual representation of the cluster. This representation of clusters provides us a visual list of the entities present in the video, which is what we call entity-based summarization of the video.
Any summary should have two properties: 1) It should be concise 2) It should be representative. Along these lines, an entity-based summary should have the property that it should cover as many entities as possible, with least number of clusters. On the other hand, the selected clusters should cover a sufficiently large fraction of all the tracklets. In our evaluation of entity discovery (Section VII) we have measured Entity Coverage, Tracklet Coverage and Number of significant clusters. These same measures are useful in evaluating the summarization. Entity Coverage and Tracklet Coverage should be high, and number of significant clusters should be low (See Figures 8,9). To make the evaluation more comprehensive, we define two more measures: 1) Conciseness: defined as the ratio of Entity Coverage to the number of significant clusters, and 2) Representativeness: defined as the ratio of the Tracklet Coverage to the number of significant clusters.
The results are shown in the Tables 9,10. We find that in terms of Conciseness, TC-CRP turns out to be the best, while the other methods are all comparable when averaged across the videos. In terms of Representativeness, TC-CRP is once again the best by a long way, while TC-CRF and sHDP-HMM are at par. The non-Bayesian methods are way behind.
Ix-B Shot-based Summarization
Another way of summarization is by a collection of shots.  follows this approach, and selects a subset of the shots based on the total number of characters (entities), number of prominent characters (entities) etc. A shot  is a contiguous sequence of frames that consist of the same set of entities. It is possible to organize the video into temporal segments based on the cluster indices assigned to the tracklets. In a frame , let denote the set of cluster labels assigned to the tracklets that cover frame . For two successive frames and , if we say that they belong to the same temporal segment, i.e. . But if , then we start a new temporal segment, i.e. . By this process, the frames of the video are partitioned into temporal segments. The cluster labels are supposed to correspond to entities, so each temporal segment should correspond to a shot. Each such segment can be easily represented with any one frame, since all the frames in a segment contain the same entities. This provides us a shot-based summarization of the video.
As in the case with entities, once again a large number of temporal segments are created by this process, with several adjacent segments corresponding to the same set of entities. This happens because often several clusters are formed for the same entity. Analogous to Entity Coverage, we define Shot Coverage as the total number of true shots that have at least one temporal segment lying within it. We then define significant segments as those which cover a sufficient number (say 100) of frames. Finally, we define Frame Coverage as the fraction of the frames which come under the significant segments.
To evaluate such shot-based summarization, once again we need to consider the two basic properties: conciseness and representativeness. These are measured in exact analogy to the entity-based summarization discussed above (See Figures 10,11). The Conciseness of the summary is defined as the ratio of the Shot Coverage to the number of significant segments, while the Representativeness of the summary is defined as the ratio of the Frame Coverage to the number of significant segments. The results are shown in Tables 11 and 12. This time we find that in terms of representativeness TC-CRF leads the way, followed by TC-CRP. In terms of conciseness the best performance is given by WBSLRR, which however does poorly in terms of representativeness.
In this paper, we considered an entity-driven approach to video modelling. We represented videos as sequences of tracklets, with each tracklet associated with an entity. We defined entity discovery as the task of discovering the repeatedly appearing entities in the videos, along with all their appearances, and cast this as tracklet clustering. We considered a Bayesian nonparametric approach to tracklet clustering which can automatically discover the number of clusters to be formed. We leveraged the Temporal Coherence property of videos to improve the clustering by our first model: TC-CRP. The second model TC-CRF was a natural extension to TC-CRP, to jointly model short temporal segments within a video, and further improve entity discovery. These methods were empirically shown to have several additional abilities like performing online entity discovery efficiently, and detecting false tracklets. Finally, we used these results for semantic video summarization in terms of the discovered entities.
-  Candes,E. & Recht,B. (2009) Exact Matrix Completion via Convex Optimization, Foundations of Computational Mathematics
Cai,J.-F., Candès, E. J. and Shen, Z. (2010)
A singular value thresholding algorithm for matrix completion, SIAM Journal of Optimization, Vol 20(4)
-  Keshavan,R., Montanari,A. & Oh,S. (2010) Matrix Completion from a few entries, IEEE Transactions on Information Theory
Wright,J., Sastry,S., Peng,Y., Ganesh,A. & Ma, Y. (2009)
Robust Principal Component Analysis, Advances in Neural Information Processing Systems (NIPS)
-  Ding, X., He, L. & Carin, L. Bayesian Robust Principal Component Analysis, IEEE Transactions on Image Processing
-  Babacan,S.D., Luessi,M., Molina,R. & Katsaggelos,A.K. (2012) Sparse Bayesian Methods for Low-Rank Matrix Estimation, IEEE Transactions on Signal Processing
-  http://www.cs.albany.edu/~davidson/Publications/KDDSlides.pdf
-  KWagstaff,K., Cardie,C., Rogers,S. & Schrdl,S. (2001) Constrained k-means clustering with background knowledge, Intl Conf. on Machine Learning (ICML)
-  Lu,Z. & Leen,T. (2007) Penalized Probabilistic Clustering, Neural Computation Vol. 19(6)
-  Shi,J. & Malik,J. (2000) Normalized cuts and image segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Vol 22(8)
Tierney,S., Gao,J. & Yi Guo,Y. (2014) Subspace Clustering for Sequential Data
, IEEE Intl. Conf on Computer Vision and Pattern Recognition (CVPR)
-  Wang,X., Qian,B. & Davidson,I., (2014) On Constrained Spectral Clustering and its Applications, Data Mining and Knowledge Discovery, Vol 28(1)
-  Kawale,J. & Boley,D. (2013) Constrained Spectral Clustering using L1 Regularization, SIAM Conference on Data Mining (SDM)
-  Xiao1,S., Tan,M., & Xu,D. (2014), Weighted Block-Sparse Low Rank Representation for Face Clustering in Videos, European Conference on Computer Vision (ECCV)
-  T. S. Ferguson (1973), A Bayesian Analysis of some Nonparametric Problems, The Annals of Statistics.
-  Sethuraman,J. (1991) A Constructive Definition of Dirichlet Priors
Gorur,D. & Rasmussen,C.E. (2010)
Dirichlet process Gaussian mixture models: Choice of the base distribution, Journal of Computer Science and Technology
-  Teh,Y.W, Jordan,M., Beal,M. & Blei,D. (2006) Hierarchical Dirichlet Processes, Journal of American Statistical Association (JASA)
-  Fox,E., Sudderth,E., Jordan,M. & Willsky,A. (2008) An HDP-HMM for Systems with State Persistence, Intl. Conf. on Machine Learning (ICML)
-  Mitra,A., Ranganath,B.N., & Bhattacharya,I., (2013) A Layered Dirichlet Process for Hierarchical Segmentation of Sequential Grouped Data, European Conference on Machine Learning (ECML-PKDD)
-  Blei,D.M. & Frazier,P.I. (2011) Distance Dependent Chinese Restaurant Processes, Journal of Machine Learning Research, Vol 12
-  Griffiths,T. & Grahramani,Z. (2005) Infinite Latent Feature Models and the Indian Buffet Process, Advances in Neural Information Processing Systems (NIPS)
-  S.Williamson, C.Wang, K.Heller, D.Blei; The IBP Compound Dirichlet Process and its Application to Focused Topic Modeling; ICML 2010
-  Du, L., Buntine, W. L. & Johnson, M. (2013) Topic Segmentation with a Structured Topic Model, North American Association for Computation Linguistics (NA-ACL-HLT)
-  Wu,B., Zhang,Y., Hu,B-G & Ji,Q. (2013) Constrained Clustering and Its Application to Face Clustering in Videos. IEEE Intl. Conf on Computer Vision and Pattern Recognition (CVPR)
-  Wu,B., Lyu,S., Hu,B-G & Ji,Q. (2013) Simultaneous Clustering and Tracklet Linking for Multi-Face Tracking in Videos, IEEE Intl. Conf on Computer Vision (ICCV)
-  Sharma,P. & Nevatia,R. (2013) Efficient Detector Adaptation for Object Detection in a Video, IEEE Intl. Conf on Computer Vision and Pattern Recognition (CVPR)
-  Tang,K., Ramanathan,V., Li,Fei-Fei & Koller,D. (2012) Shifting Weights: Adapting Object Detectors from Image to Video, Advances in Neural Information Processing Systems (NIPS)
-  Huang,C., Wu,B., & Nevatia,R. (2008) Robust object tracking by hierarchical association of detection responses, European Conf on Computer Vision (ECCV)
-  Viola,P. & Jones,M. (2001) Rapid object detection using a boosted cascade of simple features, IEEE Intl. Conf on Computer Vision and Pattern Recognition (CVPR)
-  Ji,H., Liu,C., Shen,Z. & Xu,Y. (2010) Robust Video Denoising using Low Rank Matrix Completion, IEEE Intl. Conf on Computer Vision and Pattern Recognition (CVPR)
Wright,J., Yang,Y., Ganesh,A., Sastry,S.S., & Ma,Y. (2009)
Robust Face Recognition via Sparse Representation, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol 31(2)
-  Peng,Y., Ganesh,A., Wright,J., Xu,W. & Ma,Y. (2010) RASL: Robust Batch Alignment of Images by Sparse and Low-Rank Decomposition, IEEE Intl Conf Computer Vision and Pattern Recognition (CVPR)
-  Yang,B. & Nevatia,R. (2012) An Online Learned CRF Model for Multi-Target Tracking, IEEE Intl Conf Computer Vision and Pattern Recognition (CVPR)
-  Andriluka,M., Roth,S. & Schiele,B. (2008)People-tracking-by-detection and people-detection-by-tracking, IEEE Intl Conf. Computer Vision and Pattern Recognition (CVPR)
-  Felzenszwalb,P.F., Girshick,R.B., McAllester,D. & Ramanan,D. (2010) Object Detection with Discriminatively Trained Part-based Models, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol 32(9)
-  Arandjelovic,O. & Cipolla,R. Automatic Cast Listing in feature-length films with Anisotropic Manifold Space, IEEE Intl Conf. Computer Vision and Pattern Recognition (CVPR)
-  Zhang,Y., Xu,C., Lu,H. & Huang,Y. (2009) Character Identification in Feature-length Films using Global Face-name Matching IEEE Transactions on Multimedia Vol 11(7)
-  Tapaswi,M., Bauml,M. & Stiefelhagen,R.: ”Knock! Knock! Who is it?” Probabilistic Person Identification in TV-Series, IEEE Conf. Computer Vision and Pattern Recognition (CVPR)
-  http://johmathe.name/shotdetect.html
-  D.Potapov, M.Douze, Z.Harchaoui, C.Schmid; Category-Specific Video Summarization; ECCV 2014
-  W-C Chiu, M.Fritz; Multi-Class Video Co-Segmentation with a Generative Multi-Video Model; CVPR 2013
-  Cong,Y., Yuan,J. & Luo,J. (2012) Towards Scalable Summarization of Consumer Videos Via Sparse Dictionary Selection, IEEE Transactions on Multimedia (2012)
-  Sang,J. & Xu,C. (2010) Character-based Movie Summarization, ACM Multimedia (2010)
-  Liang,C., Xu,C., Cheng,J. & Lu,H. (2011) TVParser: An automatic TV video parsing method, IEEE Intl. Conf. Computer Vision and Pattern Recognition (CVPR)
-  http://people.ee.ethz.ch/presta/youtube-objects/website/youtube-objects.html