Tracking-Assisted Segmentation of Biological Cells

10/19/2019 ∙ by Deepak K. Gupta, et al. ∙ 36

U-Net and its variants have been demonstrated to work sufficiently well in biological cell tracking and segmentation. However, these methods still suffer in the presence of complex processes such as collision of cells, mitosis and apoptosis. In this paper, we augment U-Net with Siamese matching-based tracking and propose to track individual nuclei over time. By modelling the behavioural pattern of the cells, we achieve improved segmentation and tracking performances through a re-segmentation procedure. Our preliminary investigations on the Fluo-N2DH-SIM+ and Fluo-N2DH-GOWT1 datasets demonstrate that absolute improvements of up to 3.8 segmentation and tracking accuracy, respectively.



There are no comments yet.


page 1

page 2

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Small sized biomedical data, such as images of biological cells or lymphocytes, are often difficult to acquire and even harder to visualise due to the restricting scale of particles, the low resolution scans and the viscosity of moving cells Saltz et al. (2018); Swiderska-Chadaj et al. (2019). As illustrated in Figure 1, the features of the nuclei in each frame are not always clearly visible and their position is very volatile. Being able to detect individual cells and track their trajectory through time will help automate treatment observation and disease spread detection Coudray et al. (2018).

Figure 1: Schematic representation of cell states in two different frames (modified from an image of HL60 cells from the Fluo-N2DH-SIM+ dataset). One cell fades out (cell death) and two collide, in a later frame.

Ronneberger et al. 2015 [7] introduced the U-Net architecture, which has demonstrated state-of-the-art performance on many biomedical image segmentation tasks Ronneberger et al. (2015); Falk et al. (2019). Since then, several cell tracking approaches have utilised the success of U-Net to boost their performances Li et al. (2018). However, due to the constant change of the position, shape and status of the cells in the data, most such approaches fail to accurately detect cells that combine, split or die (leave the image) Christ et al. (2016).

In this paper, we propose to improve the segmentation of biological cells by augmenting with Siamese matching. The initial U-Net segmentation results are combined with cell detection from a Siamese matching-based tracker Tao et al. (2016) to improve the cell recognition in subsequent frames. Next, a dedicated detection mechanism for cell collisions, mitosis (splitting of cells) and apoptosis (cell death) is introduced which attempts to model the movement behaviour of the nuclei. Based on the corrections, we re-segment the cells using random walker algorithm. Grady (2006).

2 Approach

Figure 2: An illustration of Mitosis detection through cell tracking

We propose to use Siamese matching approach to detect the cell collisions as well as mitosis. Our model combines the U-Net segmentation with an object tracking mechanism in order to improve upon the initial predictions. The various steps are described below.

Initial Segmentation. The initial segmentation is done using the U-Net implementation of Ronneberger et al. (2015)

, and the results are used as the baseline segmentation for the chosen sequence. During training, data augmentation consisting of random flips and shifts is employed, and as a final step, a nearest neighbour interpolation algorithm is used convert the U-Net results of

pixels to the desired resolution. After this process is finished, cells are detected and defined as being connected regions of positively labelled pixels in the segmentation map.

Tracking. To predict the new location of a cell in a consecutive frame, we use the Siamese tracker of Bertinetto et al. (2016) and use a pre-trained model, without any further training on cellular footage. SiamFC is adapted for grayscale images and a search space of pixels is used. Tracking is done in the forward as well as the backward directions. During tracking, given that our segmentation and location predictions are independent, we refine the tracking performance and detect the occurrence of mitosis and collision events.

Let denote the frame in a sequence of length , and be the set of detected cells in this frame. These are used to initialize the tracker at step . For cell , we refer to the predicted locations by the tracker in and as forward and backward predictions, respectively. Collision and mitosis are then detected as follows.

Collision detection. Collision refers to scenarios where two cells share a fraction of their boundary, and this can often be mistaken as a single cell during segmentation. When processing a new frame , where , we start off by performing collision detection in which a cell is considered to be a lump of multiple individual cells if the centroids of two or more cells in lie within the tracked region . If this is the case, is re-segmented according to a procedure based on Random Walker algorithm. More details on this are described later in Re-segmentation section. This collision detection procedure continues until each cell in matches at most one cell in or until the re-segmentation procedure fails and does not yield improvement anymore.

Mitosis detection. We then continue by matching cells in to the detected cells in . This matching happens in a manner similar to the collision detection, namely a cell is matched to a cell if the centroid of is inside the region . Different from collision detection, however, is also matched to if the centroid of the region lies within the boundaries of the cell . This matching procedure yields a set of matches for each cell , which we denote as and its size as . The state of =cell is then determined according to:


In case of mitosis, the cell splits, thus the tracking of ends and the cells in are initialised with two new trackers which have as their parent. After continuations have been determined for all cells in , cells in that are not linked to any cell in are interpreted as newly detected cells which start their life in without link to a parent cell. An illustration is shown in Figure 2.

Figure 3: Cell re-segmentation over initial U-Net segmentation

Re-segmentation. In case of a detected collision of two or more cells into a cell , we re-segment in such a manner that the new number of segments matches the number of colliding cells. This is achieved using the random walker segmentation algorithm as described in Grady (2006). To prevent over-segmentation of the cell , which adversely affects segmentation accuracy, we use the relative position of the centroids of the cells in as the seeds for the segmentation algorithm. An illustration of re-segmented cells is shown in Figure 3.

3 Results

The results of this work are compared against the vanilla U-Net, Ronneberger et al. (2015), as the baseline approach. The accuracy scores, as described in Ulman et al. (2017), are used as the decisive metric for model performance. The segmentation and tracking performance of the two methods is listed in Table 1. As can be seen, the method proposed outperforms the baseline approach on both datasets in both segmentation and tracking with a maximum increase in performance of 3.8% and 3.4% on the segmentation and tracking of the Fluo-N2DH-SIM+ 02 collection, respectively.

The increased performance of the method proposed by this work indicates that the mitosis, apoptosis and cell fusion interpretation of the collision detection mechanism indeed help the tracker network improve the segmentation performance. Furthermore, this finding highlights the importance of modelling the movement behaviour of the cells to better capture the pattern nature of cells in consecutive frames.

Segmentation accuracy Tracking accuracy
Dataset U-Net Our method U-Net + SiamTracker Our method
Fluo-N2DH-SIM+ 01 0.919 0.924 0.986 0.992
Fluo-N2DH-SIM+ 02 0.800 0.838 0.922 0.956
Fluo-N2DH-GOWT1 01 0.739 0.746 0.973 0.978
Fluo-N2DH-GOWT1 02 0.827 0.837 0.869 0.875
Table 1: Accuracy results for segmentation and tracking of cells. Our method includes the modules for detecting collisions, mitosis and apoptosis on top of U-Net and Siamese matching.

4 Conclusions

Deep learning trackers such as U-Net work well on biological cell tracking problems. However, they still suffer in the presence of events such as cell collisions, mitosis and apoptosis. In this work, we proposed to combine Siamese matching Tao et al. (2016); Bertinetto et al. (2016) with U-Net Ronneberger et al. (2015) to accurately identify these difficult scenarios. Siamese matching-based detection efficiently provides a more precise understanding and modelling of individual cells, thereby providing better grasp of latent information about them. Based on the proposed approach, we obtained absolute improvements of up to 3.8 % and 3.4% for segmentation and tracking accuracy, respectively. Our future work includes rigorously testing the proposed methodology on a more diverse set of cell tracking data.


  • L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. S. Torr (2016) Fully-convolutional siamese networks for object tracking. Cited by: §2, §4.
  • P. Christ, M. E. Elshaer, F. Ettlinger, S. Tatavarty, M. Bickel, P. Bilic, M. Rempfler, M. Armbruster, F. Hofmann, M. D’Anastasi, et al. (2016)

    Automatic liver and lesion segmentation in ct using cascaded fully convolutional neural networks and 3d conditional random fields

    In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 415–423. Cited by: §1.
  • N. Coudray, P. S. Ocampo, T. Sakellaropoulos, N. Narula, M. Snuderl, D. Fenyö, A. L. Moreira, N. Razavian, and A. Tsirigos (2018) Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning. Nature medicine 24 (10), pp. 1559–1566. Cited by: §1.
  • T. Falk, D. Mai, R. Bensch, Ö. Çiçek, A. Abdulkadir, Y. Marrakchi, A. Böhm, J. Deubner, Z. Jäckel, K. Seiwald, et al. (2019) U-Net: deep learning for cell counting, detection, and morphometry. Nature methods 16 (1), pp. 67–70. Cited by: §1.
  • L. Grady (2006) Random walks for image segmentation. IEEE Transactions on Pattern Analysis & Machine Intelligence (11), pp. 1768–1783. Cited by: §1, §2.
  • X. Li, H. Chen, X. Qi, Q. Dou, C. Fu, and P. Heng (2018) H-denseunet: hybrid densely connected unet for liver and tumor segmentation from ct volumes. IEEE transactions on medical imaging 37 (12), pp. 2663–2674. Cited by: §1.
  • O. Ronneberger, P. Fischer, and T. Brox (2015) U-Net: convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241. Cited by: §1, §2, §3, §4.
  • J. Saltz, R. Gupta, L. Hou, T. Kurc, P. Singh, V. Nguyen, D. Samaras, K. R. Shroyer, T. Zhao, R. Batiste, et al. (2018) Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images. Cell reports 23 (1), pp. 181–193. Cited by: §1.
  • Z. Swiderska-Chadaj, H. Pinckaers, M. van Rijthoven, M. Balkenhol, M. Melnikova, O. Geessink, Q. Manson, M. Sherman, A. Polonia, J. Parry, et al. (2019) Learning to detect lymphocytes in immunohistochemistry with deep learning. Medical Image Analysis, pp. 101547. Cited by: §1.
  • R. Tao, E. Gavves, and A. Smeulders (2016) Siamese instance search for tracking.

    Proceedings of the IEEE conference on computer vision and pattern recognition

    , pp. 1420–1429.
    Cited by: §1, §4.
  • V. Ulman, M. Maška, K. Magnusson, O. Ronneberger, C. Haubold, N. Harder, P. Matula, P. Matula, D. Svoboda, M. Radojevic, I. Smal, K. Rohr, J. Jaldén, H. Blau, O. Dzyubachyk, B. Lelieveldt, P. Xiao, Y. Li, S. Cho, A. Dufour, J. Olivo-Marin, C. Reyes-Aldasoro, J. Solis-Lemus, R. Bensch, T. Brox, J. Stegmaier, R. Mikut, S. Wolf, F. Hamprecht, T. Esteves, P. Quelhas, Ö. Demirel, L. Malmström, F. Jug, P. Tomancak, E. Meijering, A. Muñoz-Barrutia, M. Kozubek, and C. Ortiz-de-Solorzano (2017) An objective comparison of cell-tracking algorithms. Nature Methods 14 (12), pp. 1141–1152. Cited by: §3.