The ability to accurately monitor cellular and sub-cellular structures in their native biological environment has enormous potential in addressing open questions in cell biology. In various applications, one of the key steps for understanding biological phenomena is to assess the motion of these structures. Recent developments in time-lapse cell microscopy imaging systems have had a great impact on the analysis of these dynamics. However, visual inspection of sequences acquired by these imaging techniques requires manual tracking of many tiny structures in numerous noisy images. Thus, automated multi-target tracking methods have been extensively used in different biological applications in the last decade [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25].
Despite significant technical advances made in automatically tracking moving objects, tracking microscopic structures, known as particles , remains a challenging task due to the complex nature of biological sequences. The microscopic sequences are usually populated with similar tiny structures having intricate motion patterns and sophisticated interactions with other structures. Moreover, the structures may enter or disappear from the field of view or be occluded by other objects. In addition in some imaging techniques, i.e. fluorescence microscopy imaging, the sequences are contaminated with a high degree of noise. Under these conditions, detection methods usually fail to detect all particles and generate many spurious measurements (clutter) [27, 14]. To be successfully applied in many biological applications, multi-target tracking methods should be able to track an unknown and time-varying number of similar particles in the presence of clutter and detection uncertainty.
To this end, many particle tracking approaches have been proposed in literature [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24]. Some of the most popular particle tracking approaches are based on detection followed by a deterministic linking procedure. Here, each particle is separately detected in each frame. Then, a deterministic solution, e.g. an optimization technique, links the corresponding targets between frames [3, 4, 5, 6, 7]. The performance of these algorithms is often sensitive to the detection algorithm and may degrade in the presence of complex target dynamics and highly cluttered detections resulting from very noisy sequences.
Bayesian filtering approaches are another class of tracking algorithms that have become popular for particle tracking applications in recent years [10, 11, 12, 13, 14, 15, 24, 16, 17, 18, 19, 20, 21, 22, 23]. These approaches better deal with tracking multiple particles in cluttered measurements by incorporating prior knowledge of target dynamics and measurement models [14, 15, 19, 17, 16, 18]. However, knowledge of the clutter rate and detection profile of the chosen detection method, are of critical importance in Bayesian multi-target tracking.
Most existing multi-target tracking solutions assume known and fixed detection and false alarm parameters. However, these parameters cannot be computed in many practical applications and worse, it is not even known whether they are time-invariant. In biological imaging techniques, the noise characteristic and the background intensity of sequences may change during the acquisition process which make detection profile and clutter rate time-variant. For instance, injection of a stimulus such as insulin into pancreatic beta cells increases the noise level and overall intensity of the sequences acquired by total internal reflection fluorescence microscopy (TIRFM)  (Fig. 1). Thus, the ability of multi-target trackers to accommodate unknown clutter and detection parameters is crucial in time-lapse cell microscopy since mismatches in these model parameters inevitably result in erroneous tracking outputs.
The key contribution of this paper is to propose an effective solution to the problem of tracking multiple maneuvering particles in unknown and time-varying false alarm and detection rates. To the best of our knowledge, this practical problem has not been discussed in the biological signal processing literature so far. To address this challenging problem, we use the recent generation of Bayesian filters based on random finite set theory. This framework provides an elegant mathematical formulation for multi-target systems and allows us to deal with the aforementioned complexities. Our proposed approach is based on the Cardinalized Probability Hypothesis Density (CPHD) filter for unknown clutter rate and detection profile known as the --CPHD filter . Clutter rate and detection probabilities are estimated by the --CPHD filter bootstrapped onto a CPHD filter that outputs target estimates. This bootstrap idea was inspired by  which requires known and uniform probability of detection. Our proposed solution can accommodate unknown and non-uniform probability of detection. To cope with maneuvering motion of sub-cellular structures [19, 18, 21, 17, 16, 20], we also propose the multiple model implementation of the aforementioned filters in this paper. We will show that the proposed method is able to deal with tracking maneuvering structures in the presence of unknown and time-varying clutter rate and detection profile.
The Bayesian estimation paradigm deals with the problem of inferring the state of a target from a sequence of noisy measurements. The state vector contains all relevant dynamic information about the target while the measurements contain what can be directly observed from the sequences. The notion of Bayes optimality is fundamental to the Bayesian estimation paradigm and is well-defined for single-target tracking. While Bayes optimal tracking techniques such as Kalman and particle filters are formulated for a single-target, they can be algorithmically extended to track a time-varying number of targets by combining them with data association [32, 33, 34]. However, the notion of Bayes optimality does not carry over. Nonetheless this approach to multi-target tracking has been used in a wide range of applications, including particle tracking [13, 21, 22, 14, 19, 18, 11, 12, 23, 20].
The random finite set (RFS) approach, introduced by Mahler generalizes the notion of Bayes optimality to multi-target system using RFS theory . The key difference with other approaches is that the collection of target states at any given time is treated as a set-valued multi-target state. This representation provides an elegant formulation for multi-target system which avoids the need for data association. The RFS approach has generated substantial interest in recent years with the development of the Probability Hypothesis Density (PHD) filter , Cardinalized PHD (CPHD) filter , multi-Bernoulli filters [37, 38], labeled RFS filter  and a host of many applications  including cell and particle tracking [16, 17, 15, 24].
Ii-a RFS for multi-target tracking
In the Bayesian estimation paradigm, the state and measurement are treated as realizations of random variables which are suitable for single target and single measurement system. However, in a multi-target system, the collection of states and measurements can be naturally represented as finite sets.
Let and be the states of all targets and all measurements at time , respectively. Over time, some of these targets may disappear, new targets may appear, and the surviving targets evolve to new states. Moreover, due to poor detector performance, only some targets are detected at each time step and many measurements are spurious detections (clutter). Thus, we can conveniently represent the targets and measurements at each time slice with two finite sets as
Consequently, the concept of a random finite set (RFS) is required to cast the multi-target estimation problem in the Bayesian framework. Intuitively, an RFS is a finite-set-valued random variable. What distinguishes an RFS from a random vector is that the number of constituent variables is random and the variables themselves are random, distinct and unordered. Mahler’s Finite Set Statistics (FISST) provides powerful yet practical mathematical tools for dealing with RFSs [35, 31], based on the notion of integration and density that is consistent with point process theory . The centerpiece of the RFS approach is the so called Bayes multi-target filter, a generalization of the single-target (optimal) Bayes filter to accommodate multiple targets based on multi-target dynamic and measurement models.
Ii-A1 Multi-Target Dynamical Model
Given a multi-target state at time , each target either continues to exist at time with probability and moves to a new state with probability density , or dies with probability and takes on the value . Thus, given a state at time , its behavior at time is modeled by a Bernoulli RFS, , which has FISST density111For more detail on RFS density and probability density please see  given by
Assuming that individual targets move independently, each of the above Bernoulli RFSs are mutually independent conditioned on . Thus the survival or death of all existing targets from time to time is modeled by the multi-Bernoulli RFS
The appearance of new targets at time is modeled by an RFS of
spontaneous births which is usually specified as an i.i.d. cluster RFS with
intensity function and cardinality distribution 222An i.i.d. cluster RFS can be statistically represented by its intensity and cardinality distribution. The cardinality distribution, , provides a discrete probability distribution over the
number of the constituent random variables,
, provides a discrete probability distribution over the number of the constituent random variables,, while the intensity distribution at the point in the state space is interpreted as the instantaneous expected number of the variables that exist at that point .. Consequently, the RFS multi-target state at time is given by the union
The likelihood of the multi-target state , at time , is described by the multi-target transition density which incorporates target dynamics, births and deaths .
Ii-A2 Multi-Target Measurement Model
Given a multi-target state at time , each target , at time , is either detected with probability and generates an observation with likelihood , or missed with probability and generates the value , i.e. each target generates a Bernoulli RFS, , which has FISST density given by
The set of measurements generated by , at time , is modeled by the multi-Bernoulli RFS
In addition, the sensor receives a set of false/spurious measurements or clutter, modeled by an RFS , which is usually specified as an i.i.d. cluster RFS with intensity function and cardinality distribution . Consequently, at time , the multi-target measurement generated by the multi-target state is formed by the union
The likelihood of the multi-target measurement , given the multi-target state, at time , is described by the multi-target measurement likelihood which incorporates detection uncertainty and clutter.
Ii-A3 Bayes Multi-Target Filter
The multi-target posterior density contains all statistical information about the multi-target state at time and can be recursively computed by the aforementioned multi-target transition density and measurement likelihood using the following prediction and update equations.
where is the set integral .
This filter generalizes the notion of Bayes optimality to multi-target system using RFS theory. Further details can be found in .
Ii-B The PHD filters
Although the above framework provides an elegant Bayesian formulation of the multi-target filtering problem, it can be computationally expensive for applications with large number of targets . In , Mahler proposed to propagate the probability hypothesis density (PHD), or posterior intensity distribution, of the targets. The primary weakness of the PHD filter is a loss of higher order cardinality information which causes erratic estimates in the number of targets especially when the density of targets is high [36, 42, 16].
Mahler  subsequently proposed the Cardinalized PHD (CPHD) filter which jointly propagates the posterior intensity function, , and cardinality distribution, . The propagation of the cardinality distribution makes the filter more robust in the estimation of the number of targets .
As with all Bayesian filtering frameworks, this filter also requires some predefined models such as the single target Markov transition density for the dynamic model, measurement likelihood and probabilities of survival and detection . Moreover, the models for new born targets and false alarms (clutter) are described by the birth cardinality distribution and the birth intensity function , and the clutter cardinality distribution and the clutter intensity function , respectively . Obviously, this filter requires knowledge of the detection probability and clutter distributions similar to other Bayesian filters.
In , it was shown that the CPHD filter can also be reformulated such that it jointly estimates clutter distributions and detection probability while tracking. We refer to this version of the filter as the --CPHD filter. However, the --CPHD cannot naturally perform as well as the CPHD filter when exact knowledge of detection probability and clutter density is available.
Iii The Bootstrap CPHD Filter
To propose an effective multi-target particle tracker for practical applications with unknown and time-varying clutter rate and detection profile, we borrow the bootstrap idea proposed in . However, in addition to the clutter rate, we calculate the detection probability, using the --CPHD filter, and feed these to a robust multi-target tracking filter such as the CPHD filter (Fig. 2). We use the CPHD filter for target estimation because it is a good trade-off between accuracy and speed. Nonetheless, it can be replaced by any multi-target filter that requires knowledge of false alarm and detection rate. In this paper, we assume that and the clutter distributions, and , are the meta parameters to be estimated for the CPHD filter in each time frame. However, we will show that the estimation of the mean of the clutter rate, , is sufficient to uniquely define the and .
To deal with maneuvering dynamics, we assume that target motion can be characterized by multiple dynamic models. Thus, here, we propose the multiple model (or jump-Markov) implementation of both the CPHD and --CPHD filters and use these filters as the tracker and the parameter estimator, respectively.
Iii-a Multiple Model CPHD filter
Complex maneuvering motions can be often characterized by multiple simpler dynamic models. The idea of tracking a target with multiple motion models is to augment the state of the targets by the index of the model .
For notational convenience, we remove the time index in our formulation throughout the paper. However, the random variables and distributions are generally time-indexed. All random variables at time and at time are simply denoted by and , respectively.
Let and denote the kinematic state space and the multiple models state space, respectively. Define the augmented state space as , where denotes the Cartesian product. The underscore notation is used for the augmented state space so that denotes for the kinematic state and for the model state. The integral of a function is given by
The measurement likelihood and probabilities of survival and detection for the augmented state vector can be respectively written as
Also, for the augmented transition density and birth intensity, we have the following factored forms,
where is the probability transition from model to and is the probability of birth for model . As described in Section II, the filter also requires the knowledge of the birth cardinality distribution .
is a known probability distribution, e.g. uniform distribution, representing how the false alarms are distributed over the measurement space and. Therefore, given , knowledge of the parameter is sufficient to define the clutter distribution.
Substituting the aforementioned augmented model terms (Eqs. 11-17) into the conventional CPHD equations  and using Eq. 10 yields the recursive equations for the multiple model or jump Markov CPHD filter (see Appendix A).
Fig. 3 depicts a simple graphical representation of the proposed multiple model CPHD filter. At each time step , the intensity and cardinality distributions are predicted and updated from the intensity and cardinality distributions in the previous time step and using the augmented model terms (Eqs. 11-17). The intensity distribution provides information on the state of the targets while the number of targets can be estimated using the cardinality distribution. Specifically, the number of targets is estimated using the posterior mode . Note that this filter requires the clutter rate and the detection probability as prior knowledge.
Iii-B Multiple Model --CPHD filter
We borrow a key idea from Mahler et al.  to develop a multiple model CPHD filter with unknown clutter rate and detection profile. To estimate clutter, the idea is to model it as a random finite set of false targets which is statistically independent from the set of actual targets. These false targets are defined by their own models such as birth, death, survival and detection probabilities and transition model. Therefore, the multi-target state is composed of a disjoint combination of these two finite sets including actual targets and clutter. This hybrid multi-target state allows us to track both actual and false targets simultaneously. To deal with multiple model dynamics and unknown detection probability, the state of targets are augmented by the index of the models and the unknown detection probability. Finally this hybrid and augmented state is estimated from the sequence of finite sets of measurements generated by both real targets and clutter while accommodating multiple model dynamics and estimating the detection probability and clutter rate.
Let denote the state space for actual targets, denote the state space for clutter generators and , and denote the state space for the unknown detection probability, and the multiple models for actual targets and clutter, respectively. Define the hybrid and augmented state space as
where denotes a disjoint union. The double dot notation is used throughout to denote a function or variable defined on the hybrid state space and the underscore notation is used for the augmented state space. Therefore, represents a hybrid and augmented state such that and denotes the actual and clutter states comprising the kinematic state and augmented components, respectively. The integral of a function is given by
. As a result, multiple model clutter generators with uniform-Poisson distribution can be substituted by a single model clutter generator with uniform distribution and higher Poisson mean. Therefore, we continue our derivations with the single model clutter generators by eliminating the variablefrom the equations. Deriving the equations for applications with multiple clutter generators and non-uniform distribution is straightforward.
In addition, since clutter generators are identical and the false targets do not follow any motion pattern, it is reasonable to ignore any functional dependence on the state of a clutter generator , . As a result, the probabilities of survival and detection and the measurement likelihood for the hybrid and augmented state vector can be respectively written as
where is clutter density which is often assumed to be uniform distribution in the measurement space. As the survival probabilities and the measurement likelihoods are independent from the detection probabilities, the terms and are removed from the equations. Obviously, the detection probabilities are only dependent on and .
For the hybrid and augmented birth intensity and transition density, we have the following factored forms,
where is the transition density for the detection probability and due to the independence of false alarms from the clutter state . Furthermore, the cardinality distribution of birth for the hybrid space is .
By substituting the hybrid and augmented state space model terms (Eqs. 20–24) into the conventional CPHD equations  and using Eq. 19, the recursive equations for this filter can be calculated (see Appendix B).
Fig. 4 shows a simple graphical representation of the proposed multiple model --CPHD filter. At each time step , the target’s intensity , the clutter intensity and the hybrid cardinality distribution are predicted and updated from the target’s intensity , the clutter intensity and the hybrid cardinality distribution in the previous time step and using the hybrid model terms (Eqs. 20–24).
The posterior cardinality provides information on the total number of real and clutter targets. Therefore, the number of actual targets cannot be estimated using the posterior mode , since includes both real targets and clutter. In this filter, the posterior mean is used for estimation of the number of real targets . Therefore, in contrast with the CPHD filter, this filter cannot benefit from propagation of the cardinality distribution for the estimated number of the targets. Similar to the PHD filter, this leads the erratic estimates of the number of targets which worsens the tracking result. To this end, we use this filter only as the estimator in our bootstrap filter. The estimated detection and mean clutter are respectively calculated as follows .
where is the inner product operator, and .
Iv Experimental results
To evaluate the performance of our proposed filters, we apply them to multi-target tracking in 2-D Total Internal Reflection Fluorescence Microscopy (TIRFM) sequences. TIRFM is an imaging technique that enables visualization of sub-cellular structures such as vesicles that are on or close to the plasma membrane of cells . The vesicles are very tiny sub-cellular structures and are seen in TIRFM sequences as small particles (bright spots) moving with varying dynamics. These small particles appear and disappear from the field of view and can be occluded by other structures. Due to limitations in the TIRFM acquisition process, the sequences are contaminated with a high level of noise. We consider the case where the main characteristics of the sequences such as noise level and the background intensity gradually increase over time (Fig. 1).
In this specific application, we are interested in the overall motion of the vesicles before and after injection of insulin, not motion of a single object. In this case, the primary concern is not how well a tracking approach maintains the identity of a tracked object over time. Instead, we are mainly concerned with how well a tracker avoids false and missed tracks.
We compare the results of our bootstrap filter against those of several popular state-of-the-art particle tracking methods including two reliable deterministic linking techniques, P-Tracker  and U-Tracker , and two robust detection based traditional Bayesian filtering approaches, MHT  and IMM-JPDA .333The performance of the first three trackers, P-Tracker, U-Tracker and MHT, has been evaluated in a recent particle tracking challenge and their results have been reported in . According to this article, they can be assumed as three top performing methods for similar applications. Our results are also compared against the result of a multiple model PHD filter . Finally, the results of our other derived filters such as the multiple-model CPHD (MM-CPHD) and --CPHD (MM---CPHD) filters are also reported here in order to show the efficiency of our bootstrap idea.
Iv-a Setup and Implementation Details
To fairly evaluate the performance of all tracking algorithms, the same detection lists were provided for all competing tracking methods. We chose the detector proposed in  as it performs reliably in our synthetic and real sequences.
Having ground truth, we can accurately calculate the clutter (false positive) and the detection probability (true positive rate) of the chosen detector for each time frame. True positive and false positive were calculated based on the optimal sub-pattern assignment between the ground truth and detected point sets. In this case, if the distance between two optimally assigned points is less than a predefined distance ( pixel), the detected point is counted as true positive, otherwise it is a false positive.
For the MHT, IMM-JPDA, MM-PHD and MM-CPHD filters, the average number of false detections per frame and the mean value of the detection probability (the optimal value for these parameters) were used as the predefined clutter rate and detection probability. However, accurate knowledge of these values is not possible in many practical applications and thus, the reported results for these filters are optimistic. For the MM---CPHD filter and similarly for our bootstrap (B-MM-CPHD) filter, the clutter rate and detection probability are adaptively estimated using our proposed framework.
To ensure the validity of our experiments, for the independently implemented tracking methods such as MHT, P-Tracker and U-Tracker, we attempted to either estimate their parameters from the ground truth or to find the values that resulted in their best performance. Moreover, the multiple motion model implementation of all competing tracking methods was used. For the IMM-JPDA and different PHD and CPHD filters, we chose identical state and measurement vectors and dynamic models. We modeled the state of each particle by its position, , and velocity, . The measurements vector also contained the estimated position of the particles as . To model maneuvering motion of particles, two linear dynamics including random walk and small acceleration motion models were used [19, 18].
Since the target dynamics and measurement models in this application can be properly expressed by multiple linear and Gaussian terms, we used the Gaussian mixture implementation  and the Beta-Gaussian mixture approach  for analytical implementations of the MM-CPHD and MM---CPHD filters, respectively. The birth intensity distribution16].
As previously discussed, the PHD and CPHD filters propagate the intensity distribution of all targets in each time frame . By calculating the estimated number of targets , the state of all targets in each time frame can be easily extracted using [45, 41, 42]444For example, in the Gaussian mixture implementations, the mean of Gaussian terms with the highest weights in can be used as the estimation of the state vector at each time frame .. However, in this filtering framework, the temporal correspondences between the estimated states is not considered. In other words, the output of the PHD and CPHD filters is a set of the estimated states for each time frame without considering the identity of trajectories. Thus, the dynamics of an individual target cannot be evaluated. To deal with this, some authors combine the PHD filter with a track management technique to maintain the identity of tracks [46, 47]. To avoid any computational burden due to a track management step, we simply use our tag propagation scheme , which only propagates the identity of the intensity distributions for all the PHD and CPHD filters. As it only considers the previous time step to propagate the identities , this is not the most reliable approach for identity-to-track assignment. However, since we are interested in the overall, not individual, motion of the vesicles, we used this approach in this application.
Iv-B Evaluation Metric
To qualitatively assess the performance of these tracking methods, we used a recent and popular metric based on optimal sub-pattern assignment (OSPA)  followed by an extension of this metric for multi-target tracking applications, known as OSPA-T .
Iv-B1 OSPA Metric
This metric measures the distance between two sets of points, which can be the set of estimated tracks and the ground truth tracks at each time frame, and represents different aspects of multi-target tracking performance such as track accuracy, track truncation and missed or false tracks by a single value.
For two arbitrary sets and , where , the metric of order is defined as
where is all feasible sets of permutations on and
where is the cut-off parameter. For , is calculated, instead.
In Eq. 26, the first term, known as the location error, measures track accuracy while the second term, the so called cardinality error, represents the error for missed or false tracks. In this metric, the parameter
controls the sensitivity to outlier estimates that are distant from the true targets. Moreover, the metric penalizes the cardinality error by the cut-off parameteras a higher value for this parameter penalizes the false and missing targets more. At a first glance, the selection of the parameters and seems to be critical for the performance of different methods. However, it was proved that the relation between the performance of different methods is independent of the parameters and and different values for these parameters only change the scale of the error, and do not affect the method ranking .
Although the OSPA metric measures the multi-target tracking errors such as track accuracy and missed or false tracks, it does not evaluate some other errors such as inconsistent labeling (label switching between the targets in crossing cases) and incorrect label initiation in the case of track truncation. An extension of the OSPA metric, known as OSPA-T555Contrary to the OSPA, the OSPA-T is not a metric in the formal mathematical sense. However, it can be used as a performance measure., considers the aforementioned errors by measuring the distance between two sets of labeled points.
Let us assume that we have two labeled sets such that and . In a multi-target tracking problem, these would be the ground truth and estimated state of the targets with their labels in each time frame. In this performance measure, the label correspondence between the estimated and ground truth sets is first performed based on an optimal global assignment using the trajectories’ temporal information . Then, in Eq. 26 is substituted by the following distance.
where if and are the corresponding labels, and otherwise. The parameter should be between and and controls the penalty assigned to the labeling error. The case assigns no penalty, and assigns the maximum penalty. In this paper, we reported the results with .
Iv-C Evaluation on synthetic data
To quantitatively evaluate the tracking algorithms, they were first evaluated using realistic synthetic movies generated by the framework proposed in . The sequences simulated using this framework reflect the difficulties existing in real TIRFM sequences while providing accurate ground truth. In these sequences, the aim is to mimic the effect of the injection of a stimulus such as insulin into a pancreatic cell. Fig. 5 shows two frames of the synthetic sequences using this framework which are comparable with the real TIRFM sequences shown in Fig. 1.
Each synthetic sequence was simulated with spatial resolution of nm/pixel and temporal resolution fps and consists of time-varying number of targets (on average particles per frame) moving through time frames inside a cell membrane (an estimated background) that is extracted from real TIRFM sequences with effective region pixels. The spots were generated in different sizes, pixels ( nm). The dynamics of the targets were modeled using random walk and linear movement. Furthermore, targets can switch between these two dynamics. The number of intersecting and touching spots in each frame was counted according to the Rayleigh resolution . The averaged percentage of intersecting and touching spots per frame in these sequences is equal to .
Due to the 3-D motion of the structures, their intensity changes according to the TIRFM exponential equation . Therefore, they may either temporarily or permanently disappear from the frames. New born targets may also gradually appear from the background. The sequences are contaminated with Poisson noise and the main characteristics of the sequences such as background intensity and noise level gradually change based on a model extracted from real sequences. Due to spatio-temporally varying noise level and backgrounds and dynamic intensity of spots, the signal-to-noise ratio (SNR)666SNR value is used for representing the level of noise and is calculated using definition of Smal et al.  which is the difference in intensity between the object and the background, divided by the standard deviation of the object noise. of an object cannot be constant. Instead, it varies between and .
Similar to the real TIRFM data, the spots in the synthetic sequences fade over time as noise level and background intensity gradually increases. From a biological perspective, it is important to assess the dynamics of vesicles after injection of the stimulus which leads the escalation of noise level and background intensity. Therefore, the threshold in the detection method needs to be set low to ensure consistent detections of the objects in order to avoid early track termination. To this end, we chose a value for the threshold such that the averaged detection probability for all sequences is about . However, this scenario noticeably increases the clutter rate and its variation over time. Fig. 6 shows that the mean clutter rate estimated using the proposed MM---CPHD filter can appropriately follow the quick changes in the ground truth clutter rate.
In this experiment, the performance of the trackers was evaluated using these time-varying and highly cluttered detections and their results are reported in Table I. In this Table, the errors are averaged over the number of frames in all synthetic sequences. The results show that our bootstrap filter benefits from a reliable estimator, the MM---CPHD filter, which accurately estimates the detection probability and clutter rate. Consequently compared to the other tracker, its tracking results contain fewer false and missing tracks, which decreases the cardinality error significantly. Similarly, this error is also low in the MM---CPHD filter. The other Bayesian trackers cannot benefit from these estimations as their parameters are fixed. Therefore, they have higher cardinality error. In comparison with traditional Bayesian trackers such as the IMM-JPDA and MHT trackers, the RFS filters have relatively better cardinality error as they properly incorporate birth, death and clutter models in their formulations. However, the higher difference between the OSPA and OSPA-T errors of the all RFS filters represent that their results include more identity switch errors compared to the other trackers due to the simple tag propagation scheme used.
The performance of the P-tracker as one of the deterministic trackers seems to be sensitive to cluttered measurements. Therefore, it has the highest cardinality error of the trackers. In contrast, the results show that another deterministic tracking scheme, U-tracker, can robustly track the particles while dealing with highly cluttered detections.
The P-Tracker has the lowest location error of all trackers. Although this error reflects the accuracy of the trackers in tracking particles, its lower value can be also an artifact of the tracker’s poor performance. The trackers with higher cardinality error have relatively lower location error and vice versa.
The method rankings may change in other scenarios, e.g. where the clutter rate is low and its variation over time can be ignored. In our application, the clutter rate and its variation over time can be decreased by increasing the detection threshold value. For example, we chose a value such that the averaged clutter rate and detection probability are respectively about and . In this case, the detected points from a target are inconsistent and many faint particles are not detected after initial time frames. This case is undesirable for our application due to many truncated tracks with short life time. Nevertheless, we report the tracker’s OSPA and OSPA-T errors in this case in Table II in order to compare their different performance in the low and high clutter rate scenarios. Note that both false tracks in high clutter scenario and missed or truncated tracks in low clutter scenario increase the OSPA and OSPA-T errors.
The results show that the trackers such as the IMM-JPDA filter and P-Tracker which have the worst performance in highly cluttered measurements perform better for a low clutter rate with long missed detections compared to other trackers. In contrast, the U-Tracker cannot perform reliably in this case as it has the highest cardinality error. Estimating declining trend in the detection probability (Fig. 7) helps our bootstrap tracker to still have lower OSPA error compared to the other trackers, but relatively high OSPA-T error due to the simple tag propagation scheme.
In a nutshell, the performance of the trackers varies according to the detector reliability. Our bootstrap approach has superior performance for highly cluttered detections with the time-varying rates.
Iv-D Evaluation on real data
The tracking methods were also tested on a real TIRFM sequence with spatial and temporal resolution of nm/pixel and fps respectively. The image sequences were acquired from a pancreatic beta cell injected by insulin during the acquisition.
In order to prepare a ground truth, an independent expert manually tracked all visible structures ( trajectories from spots) in a part of the cell within time frames using the freely available software MTrackJ . The annotated trajectories were double-checked by another biologist to ensure reliability of the ground truth.
In order to detect and track all structures, especially after injection of insulin, the threshold in the detection method was set low, which noticeably increases clutter rate and its variation over time (Fig. 8). The results of the trackers for the real data are reported in Table III. In this Table, the errors are averaged over the number of time steps in the real image sequence.
The higher clutter rate along with significantly higher number of time frames and numerous faint structures existing in real sequences cause higher values of the OSPA and OSPA-T errors of the all trackers for real sequences compared to the synthetic data. In addition, the results confirm our arguments about the performance of the trackers in the synthetic sequences for the same high clutter scenario. The ability of our B-MM-CPHD to properly track true targets while accurately estimating the clutter rate and detection probability (Fig. 8) leads to the lowest cardinality and also OSPA errors.
Since there were always faint structures in real sequences that even experienced annotators were unable to detect or determine whether they are real or false targets, the reliability of the manual ground truth cannot be completely guaranteed. In this case, the quantitative results may be biased to a specific method. To maximize the validity of our comparison, the results of the tracking were also visually assessed by the experts. This assessment also showed that our bootstrap filter can better detect and track the real vesicles, especially faint ones, while avoiding tracking false targets.
In Fig. 9 (a), the tracking results of several crossing particles with maneuvering motions using the proposed bootstrap filter are shown. The results show that it can properly deal with the maneuvering motion of the targets. However, the results are not error free and include some missing and false tracks and switching labeling errors. Fig. 9 (b) also demonstrates the performance of our proposed bootstrap filter in tracking two faint particles moving through the synthetic and real sequences with time-varying background and noise level.
The estimation of clutter rate and detection probability helps improve tracking of the targets in particle tracking applications where measurements include many spurious and missed detections with unknown and time-varying parameters. The RFS framework provides a principled solution to deal with the estimation of these parameters which was not possible in previous approaches. The --CPHD filter is one of the filters derived based on RFS theory which is able to estimate these parameters. However, the filter cannot naturally perform as well as the CPHD filter with known parameters. To this end, in this paper we proposed a bootstrap filter by combination of the CPHD and --CPHD filters. To accommodate the maneuvering motion, we also proposed a multiple model implementation of the filters. Therefore, the clutter rate and detection probability are estimated by the multiple model --CPHD filter bootstrapped onto a multiple model CPHD filter that outputs target estimates.
The proposed approach was evaluated on a challenging particle tracking application where vesicles move in noisy sequences of total internal reflection fluorescence microscopy while the noise characteristic and background intensity of the sequences change during the acquisition process. In this application which the clutter rate and detection probability are time-varying, we demonstrated that our bootstrap filter is able to better track the real targets and more reliably avoid false tracks compared to the other RFS filters such as the MM-PHD, MM-CPHD and MM---CPHD as well as other state-of-the-art particle tracking methods such as the IMM-JPDA, MHT, P-Tracker and U-Tracker in both synthetic and real sequences. However, in the applications where the motion of each individual targets is required, the tag propagation scheme may need to be changed to a track management algorithm, e.g., that proposed in [47, 46] or more generally, the tracker in the bootstrap filter can be replaced by any multi-target tracker that requires knowledge of false alarm and detection rate and performs reliably in the label assignments. In the RFS concept, a principled solution to the track labeling problem using labeled RFS  has been recently proposed which can be also used for this purpose.
Obviously, our approach may not be a good choice for some applications. For example, in the cases where the particles can be easily detected or the rates for false alarms and missed detections are negligible, known or time-invariant, there is no point to adaptively estimate the clutter rate and the detection profile. In addition, to estimate these time-varying rates, the MM---CPHD requires some probabilistic priors describing how the clutter rate and the detection probability change over time . Therefore, this approach requires more parameters compared to other trackers. Moreover, the bootstrap method filters twice in each frame in order to estimate the state of the targets. Consequently, its processing time is more than each of its component filters, MM---CPHD and MM-CPHD. Our non-optimized MATLAB code was run on an ordinary PC (Intel Core Quad, GHz CPU, GB RAM). The average CPU processing time per frame per target in the highest considered clutter rate (around detected positions per frame) for the multiple model CPHD, --CPHD and our bootstrap filter are about ms, ms and ms respectively.
Due to linearity of the dynamic and measurement models in our application, we only used the multiple model Gaussian linear implementation of all filters. The recursive equations derived in the appendices can be used for both Gaussian linear and non-Gaussian non-linear system models. In order to apply the proposed framework to applications with non-linear non-Gaussian system models, the sequential Monte Carlo (SMC) implementation of the filters is required. This implementation will also allows us to further evaluate the performance of our bootstrap filter and compare it against the SMC-PHD , SMC-CPHD  and the traditional particle  filters.
For completeness, we now derive the recursive equations for both the multiple model CPHD and --CPHD filters in this section. The following notations are used in throughout this Appendix. The binomial and permutation coefficients are denoted by and , respectively. is the inner product operation between two continuous or two discrete functions. The elementary symmetric function of order defined for a finite set of real numbers is denoted by
where is the cardinality of a set and .
A Multiple Model CPHD Recursions
Prediction step: Suppose at time , the posterior cardinality distribution and posterior intensity are known. The predicted cardinality distribution and predicted intensity are calculated by
Note that in the multiple model approach, the inner product function operates on both the kinematic state and the model, i.e. .
Update step: If at time , the predicted cardinality distribution , predicted intensity and set of measurement are given, the updated cardinality distribution and updated intensity are calculated by
B Multiple Model --CPHD Recursions
Prediction step: Suppose at time , the hybrid cardinality distributions and the posterior intensity distribution for actual targets and clutter generators are given. The hybrid predicted cardinality distribution and predicted intensity for actual targets and clutter generators are calculated by
Update step: If at time , the predicted intensity for actual targets , the predicted intensity for clutter generators , the predicted hybrid cardinality distribution and set of measurement are all given and the function defined as follows:
where and .
The updated cardinality distribution and the updated intensity distribution for actual targets and clutter generators are given as follows
The authors would like to thank Dr. William E. Hughes and the research team from the Garvan Institute of Medical Research, NSW, Australia, for providing the real TIRFM sequences and their manual ground truth. The authors also thank Dr. Anthony Dick from the University of Adelaide whose comments and suggestions improved presentation of the paper.
-  A. Matov, M. M. Edvall, G. Yang, and G. Danuser, “Optimal-flow minimum-cost correspondence assignment in particle flow tracking,” Comput. Vis. Image Und., vol. 115, no. 4, pp. 531–540, 2011.
-  S. Bonneau, M. Dahan, and L. D. Cohen, “Single quantum dot tracking based on perceptual grouping using minimal paths in a spatiotemporal volume,” IEEE Trans. Image Process., vol. 14, no. 9, pp. 1384–1395, 2005.
-  G. Mashanov and J. Molloy, “Automatic detection of single fluorophores in live cells,” Biophys. J., vol. 92, no. 6, pp. 2199–2211, 2007.
-  K. Jaqaman, D. Loerke, M. Mettlen, H. Kuwata, S. Grinstein, S. L. Schmid, and G. Danuser, “Robust single-particle tracking in live-cell time-lapse sequences,” Nat. methods, vol. 5, no. 8, pp. 695–702, 2008.
-  M. Dewan, M. Ahmad, and M. Swamy, “Tracking biological cells in time-lapse microscopy: an adaptive technique combining motion and topological features,” IEEE Trans. Biomed. Eng., vol. 58, no. 6, pp. 1637–1647, 2011.
-  I. Sbalzarini and P. Koumoutsakos, “Feature point tracking and trajectory analysis for video imaging in cell biology,” J. Struct. Biol., vol. 151, no. 2, pp. 182–195, 2005.
-  D. Padfield, J. Rittscher, and B. Roysam, “Coupled minimum-cost flow cell tracking for high-throughput quantitative analysis,” Med. Image Anal., vol. 15, no. 4, pp. 650–668, 2011.
-  E. Meijering, I. Smal, and G. Danuser, “Tracking in molecular bioimaging,” IEEE Signal Proc. Mag., vol. 23, no. 3, pp. 46–53, 2006.
-  D. House, M. Walker, Z. Wu, J. Wong, and M. Betke, “Tracking of cell populations to understand their spatio-temporal behavior in response to physical stimuli,” in Proc. IEEE Conf. Comput. Vis. Mach. Intell. (CVPR 2009), 2009, pp. 186–193.
L. Liang, H. Shen, P. De Camilli, D. Toomre, and J. Duncan, “An expectation maximization based method for subcellular particle tracking using multi-angle TIRF microscopy,” inMedical Imag. Comput. Computer-Assist. Interv. (MICCAI 2011), 2011, pp. 629–636.
-  N. H. Nguyen, S. Keller, E. Norris, T. T. Huynh, M. G. Clemens, and M. C. Shin, “Tracking colliding cells in vivo microscopy,” IEEE Trans. Biomed. Eng., vol. 58, no. 8, pp. 2391–2400, 2011.
-  A. Genovesio, T. Liedl, V. Emiliani, W. Parak, M. Coppey-Moisan, and J. Olivo-Marin, “Multiple particle tracking in 3-D+t microscopy: Method and application to the tracking of endocytosed quantum dots,” IEEE Trans. Image Process., vol. 15, no. 5, pp. 1062–1070, 2006.
-  L. Yang, Z. Qiu, A. Greenaway, and W. Lu, “A new framework for particle detection in low-SNR fluorescence live-cell images and its application for improved particle tracking,” IEEE Trans. Biomed. Eng., vol. 59, no. 7, pp. 2040–2050, 2012.
-  N. Chenouard, I. Bloch, and J.-C. Olivo-Marin, “Multiple hypothesis tracking for cluttered biological image sequences,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 11, pp. 2736–2750, 2013.
-  R. Juang, A. Levchenko, and P. Burlina, “Tracking cell motion using GM-PHD,” in Proc. IEEE Int. Symp. Biomed. Imag. (ISBI 2009), 2009, pp. 1154–1157.
-  S. H. Rezatofighi, S. Gould, B.-N. Vo, K. Mele, W. E. Hughes, and R. Hartley, “A multiple model probability hypothesis density tracker for time-lapse cell microscopy sequences,” in Inform. Process. Medical Imag. (IPMI 2013), 2013, pp. 110–122.
-  T. Wood, C. Yates, D. Wilkinson, and G. Rosser, “Simplified multitarget tracking using the PHD filter for microscopic video data,” IEEE Trans. Circ. Syst. Vid., vol. 22, no. 5, pp. 702–713, 2012.
-  L. Feng, Y. Xu, Y. Yang, and X. Zheng, “Multiple dense particle tracking in fluorescence microscopy images based on multidimensional assignment,” J. Struct. Biol., vol. 173, no. 2, pp. 219–228, 2011.
-  S. H. Rezatofighi, S. Gould, R. Hartley, K. Mele, and W. E. Hughes, “Application of the IMM-JPDA filter to multiple target tracking in total internal reflection fluorescence microscopy images,” in Medical Imag. Comput. Computer-Assist. Interv. (MICCAI 2012), 2012, pp. 357–364.
-  K. Li, E. D. Miller, M. Chen, T. Kanade, L. E. Weiss, and P. G. Campbell, “Cell population tracking and lineage construction with spatiotemporal context,” Med. Image Anal., vol. 12, no. 5, pp. 546–566, 2008.
-  I. Smal, E. Meijering, K. Draegestein, N. Galjart, I. Grigoriev, A. Akhmanova, M. Van Royen, A. Houtsmuller, and W. Niessen, “Multiple object tracking in molecular bioimaging by Rao-Blackwellized marginal particle filtering,” Med. Image Anal., vol. 12, no. 6, pp. 764–777, 2008.
-  I. Smal, K. Draegestein, N. Galjart, W. Niessen, and E. Meijering, “Particle filtering for multiple object tracking in dynamic fluorescence microscopy images: Application to microtubule growth analysis,” IEEE Trans. Med. Imag., vol. 27, no. 6, pp. 789–804, 2008.
-  W. Godinez, M. Lampe, S. Wörz, B. Müller, R. Eils, and K. Rohr, “Deterministic and probabilistic approaches for tracking virus particles in time-lapse fluorescence microscopy image sequences,” Med. Image Anal., vol. 13, no. 2, pp. 325–342, 2009.
-  R. Hoseinnezhad, B.-N. Vo, B.-T. Vo, and D. Suter, “Visual tracking of numerous targets via multi-Bernoulli filtering of image data,” Pattern Recogn., vol. 45, no. 10, pp. 3625–3635, 2012.
-  L. Yuan, Y. F. Zheng, J. Zhu, L. Wang, and A. Brown, “Object tracking with particle filtering in fluorescence microscopy images: Application to the motion of neurofilaments in axons,” IEEE Trans. Med. Imag., vol. 31, no. 1, pp. 117–130, 2012.
-  N. Chenouard, I. Smal, F. De Chaumont, M. Maška, I. F. Sbalzarini, Y. Gong, J. Cardinale, C. Carthel, S. Coraluppi, M. Winter et al., “Objective comparison of particle tracking methods,” Nat. methods, vol. 11, pp. 281––289, 2014.
-  I. Smal, M. Loog, W. Niessen, and E. Meijering, “Quantitative comparison of spot detection methods in fluorescence microscopy,” IEEE Trans. Med. Imaging, vol. 29, no. 2, pp. 282–301, 2010.
-  J. Burchfield, J. Lopez, K. Mele, P. Vallotton, and W. Hughes, “Exocytotic vesicle behaviour assessed by TIRFM,” Traffic, vol. 11, pp. 429–439, 2010.
-  R. P. Mahler, B.-T. Vo, and B.-N. Vo, “CPHD filtering with unknown clutter rate and detection profile,” IEEE Trans. Signal Process., vol. 59, no. 8, pp. 3497–3513, 2011.
-  M. Beard, B.-T. Vo, and B.-N. Vo, “Multi-target filtering with unknown clutter density using a bootstrap GMCPHD filter,” IEEE Signal Proc. Let., vol. 20, no. 4, pp. 323–326, 2013.
-  R. P. Mahler, Statistical multisource-multitarget information fusion. Artech House Boston, 2007, vol. 685.
-  Y. Bar-Shalom, Tracking and data association. Academic Press Professional, Inc., 1987.
-  S. S. Blackman, Multiple-target tracking with radar applications. Artech House, Inc.,, 1986.
-  C. Hue, J.-P. Le Cadre, and P. Pérez, “Tracking multiple objects with particle filtering,” IEEE Trans. Aerosp. Electron. Syst., vol. 38, no. 3, pp. 791–812, 2002.
-  R. Mahler, “Multitarget Bayes filtering via first-order multitarget moments,” IEEE Trans. Aerosp. Electron. Syst., vol. 39, no. 4, pp. 1152–1178, 2003.
-  ——, “PHD filters of higher order in target number,” IEEE Trans. Aerosp. Electron. Syst., vol. 43, no. 4, pp. 1523–1543, 2007.
-  B.-T. Vo, B.-N. Vo, and A. Cantoni, “The cardinality balanced multi-target multi-Bernoulli filter and its implementations,” IEEE Trans. Signal Process., vol. 57, no. 2, pp. 409–423, 2009.
-  B.-N. Vo, B.-T. Vo, N.-T. Pham, and D. Suter, “Joint detection and estimation of multiple objects from image observations,” IEEE Trans. Signal Process., vol. 58, no. 10, pp. 5129–5141, 2010.
B.-T. Vo and B.-N. Vo, “Labeled random finite sets and multi-object conjugate priors,”IEEE Trans. Signal Process., vol. 61, no. 13, pp. 3460–3475, 2013.
-  R. P. Mahler, Advances in Statistical Multisource-Multitarget Information Fusion. Artech House Boston, 2014.
-  B.-N. Vo, S. Singh, and A. Doucet, “Sequential Monte Carlo methods for multitarget filtering with random finite sets,” IEEE Trans. Aerosp. Electron. Syst., vol. 41, no. 4, pp. 1224–1245, 2005.
-  B.-T. Vo, B.-N. Vo, and A. Cantoni, “Analytic implementations of the cardinalized probability hypothesis density filter,” IEEE Trans. Signal Process., vol. 55, no. 7, pp. 3553–3567, 2007.
-  S. Pasha, B.-N. Vo, H. Tuan, and W. Ma, “A Gaussian mixture PHD filter for jump Markov system models,” IEEE Trans. Aerosp. Electron. Syst., vol. 45, no. 3, pp. 919–936, 2009.
-  S. H. Rezatofighi, R. Hartley, and W. E. Hughes, “A new approach for spot detection in total internal reflection fluorescence microscopy,” in Proc. IEEE Int. Symp. Biomed. Imag. (ISBI 2012), 2012, pp. 860–863.
-  B.-N. Vo and W. Ma, “The Gaussian mixture probability hypothesis density filter,” IEEE Trans. Signal Process., vol. 54, no. 11, pp. 4091–4104, 2006.
-  L. Lin, Y. Bar-Shalom, and T. Kirubarajan, “Track labeling and PHD filter for multitarget tracking,” IEEE Trans. Aerosp. Electron. Syst., vol. 42, no. 3, pp. 778–795, 2006.
-  K. Panta, B.-N. Vo, and S. Singh, “Novel data association schemes for the probability hypothesis density filter,” IEEE Trans. Aerosp. Electron. Syst., vol. 43, no. 2, pp. 556–570, 2007.
-  D. Schuhmacher, B.-T. Vo, and B.-N. Vo, “A consistent metric for performance evaluation of multi-object filters,” IEEE Trans. Signal Process., vol. 56, no. 8, pp. 3447–3457, 2008.
-  B. Ristic, B.-N. Vo, D. Clark, and B.-T. Vo, “A metric for performance evaluation of multi-target tracking algorithms,” IEEE Trans. Signal Process., vol. 59, no. 7, pp. 3452–3457, 2011.
-  S. H. Rezatofighi, W. T. E. Pitkeathly, S. Gould, R. Hartley, K. Mele, W. E. Hughes, and J. G. Burchfield, “A framework for generating realistic synthetic sequences of total internal reflection fluorescence microscopy images,” in Proc. IEEE Int. Symp. Biomed. Imag. (ISBI 2013), 2013.
-  L. Rayleigh, “Investigations in optics, with special reference to the spectroscope,” Philos. Mag., vol. 8, no. 49, pp. 261–274, 1879.
-  E. Meijering, “MTrackJ: A Java program for manual object tracking,” [Online].Available:http://www.imagescience.org/meijering/software/mtrackj/.