Log In Sign Up

Liver Steatosis Segmentation with Deep Learning Methods

by   Xiaoyuan Guo, et al.

Liver steatosis is known as the abnormal accumulation of lipids within cells. An accurate quantification of steatosis area within the liver histopathological microscopy images plays an important role in liver disease diagnosis and trans-plantation assessment. Such a quantification analysis often requires a precise steatosis segmentation that is challenging due to abundant presence of highly overlapped steatosis droplets. In this paper, a deep learning model Mask-RCNN is used to segment the steatosis droplets in clumps. Extended from Faster R-CNN, Mask-RCNN can predict object masks in addition to bounding box detection. With transfer learning, the resulting model is able to segment overlapped steatosis regions at 75.87 Recall,65.88 liver disease diagnosis and allograft rejection prediction in future clinical practice.


page 2

page 3

page 4


Instance Segmentation and Object Detection with Bounding Shape Masks

Many recent object detection algorithms use the bounding box regressor t...

Adapting Mask-RCNN for Automatic Nucleus Segmentation

Automatic segmentation of microscopy images is an important task in medi...

Opinions Vary? Diagnosis First!

In medical image segmentation, images are usually annotated by several d...

Segmentation of Overlapped Steatosis in Whole-Slide Liver Histopathology Microscopy Images

An accurate steatosis quantification with pathology tissue samples is of...

End-to-End Parkinson Disease Diagnosis using Brain MR-Images by 3D-CNN

In this work, we use a deep learning framework for simultaneous classifi...

Malaria Detection and Classificaiton

Malaria is a disease of global concern according to the World Health Org...

1 Introduction

Due to abnormal retention of lipids in hepatocytes, liver steatosis can result from alcohol, obesity, and type II diabetes mellitus [1]. In addition, it serves as the hallmark of a large number of diseases, including non-alcoholic fatty liver disease (NAFLD), alcoholic fatty liver disease, and hepatotoxicity in diverse medical conditions [2]. Therefore, it is essential to achieve accurate quantification of steatosis droplet regions for an accurate disease diagnosis and liver transplantation evaluation [3].

The prevalent gold standard for steatosis assessment is via human visual inspections of liver tissue sections, a process known to be time-consuming and subject to observer variability [4]. Hailed as a new alternative solution, digital pathology is an emerging field that uses digital high-resolution images of tissue sections for machine-based image processing. Although multiple automated methods for liver steatosis measurement have demonstrated computational advantages over human reviewing process [4, 7, 8, 9, 10, 11, 12], they are not sufficiently accurate to support precise steatosis quantification, especially when overlapped steatosis regions with weak separating borders are in presence. As a result, it still remains challenging to develop a robust image analysis program to support precise liver steatosis analysis. As each tissue slide is projected to a two-dimensional microscopy image space, it is not unusual to identify a large number of tissue regions with overlapped steatosis droplets in clumps. Such spatial alignment nature, combined with substantial variations of size, staining color, and structure appearance, presents a technical barrier for individual steatosis droplet segmentation, leading to erroneous steatosis feature computation and size quantification.

Recently, deep learning has become a successful alternative solution to biomedical image analysis. In this work, we adopt the Mask-RCNN based deep learning method [5] and successfully customize it to segment overlapped steatosis droplets in whole-slide histopathology images of live tissue sections. To establish a large training data efficiently, we propose to transfer our prior work on nuclei segmentation and have a domain expert to screen results for an accurate training data set generation [6]. The proposed method can separate highly clumped steatosis droplets and recover their precise contours with promising accuracy.

2 Method

Our work for steatosis analysis is enlightened by the Mask-RCNN segmentation method proposed for object instance segmentation [5]. Extended from Faster R-CNN [13], Mask-RCNN replaces the Region of Interest (ROI)-pooling operation with ‘ROI-Align’ for solving the misalignment problem. This change in architecture enables segmentation of individual objects from different categories and results in substantial improvement in the segmentation accuracy. Due to the promising performance of Mask-RCNN for instance segmentation, we propose to customize this architecture for steatosis segmentation.

Figure 1: Schema of steatosis segmentation method.

The method schema is presented in Fig. 1 where three primary components are presented: training data preparation with our prior work on nuclei segmentation [6], model training with transfer learning, and overlapped steatosis segmentation in testing images.

2.1 Training Data Preparation

Due to the two-dimensional image space projection, a large amount of densely aligned steatosis droplets can touch together with blurred dividing boundaries. Therefore, it is not feasible for pathologists to annotate all steatosis masks for efficient training data production. This is a common problem for deep learning model training in a wide scope of research investigations.

To facilitate training data preparation, we modify our previous nuclei segmentation method [6]

and generate initial segmentation masks for steatosis instances. As our previous method aims at separating clumped nuclei in fluorescence in-situ hybridization images, it can not be directly applied to bright field histopathology images for steatosis droplet segmentation. As a result, we modify such method as follows. First, we convert the input color image to its gray-scale representation that is further binarized by a normalized threshold. All non-tissue areas in the image background are excluded for further analysis. Next, overlapped steatosis candidates are identified by rejecting connected foreground regions where solidity is over 0.95. A high curvature point voting method is used to detect dividing candidate points. They are connected based on the fitting quality evaluation by an ellipse fitting model, spatial proximity, shape convexity, and curvature information. Finally, we recover dividing curves by local shape based intensity analysis in a sector-shaped searching space, and produce the corresponding isolated steatosis masks 


Although this process produces satisfied results for a large number of touching steatosis droplets, there are partitioned steatosis instances that are not matched with their histology structures as reviewed by the domain expert. Such results are removed from the training data set. As the number of such instances is limited, these unlabeled foreground regions would have little impact on the generalization ability of the trained model. In this way, a good training data set is established in an efficient manner. The resulting data set includes 451 images of with corresponding mask sets. Each image patch has multiple masks , with each mask image containing one steatosis droplet, essential for solving the overlapped steatosis problem. With this training data set, Mask-RCNN is able to learn how to segregate overlapped steatosis droplets through image-mask pairs.

2.2 Deep Learning Model

There are three primary components in Mask-RCNN: the backbone, Region Proposal Network(RPN), and ‘ROI-Align’. The backbone is composed with Convolutional Neural Networks (CNNs) that can extract multi-level image features. We use modified resnet41, resnet50, and resnet65 


as our backbone CNNs. The second component RPN scans the input image with a sliding-window and detects steatosis droplet regions in our study. ‘ROI-Align’ further analyzes ROIs from the RPN and interpolates the feature maps from the neural network backbone at multiple locations. In this way, it can handle the incorrect alignment from Faster R-CNN 


. With these deep learning components, the resulting model can classify objects into different classes, provide object positions with bounding boxes, and produce a mask for each detected object. In the training process, these three components are orchestrated to minimize the following multi-mask loss function for each steatosis instance 



where , and are classification loss, bounding box loss, and mask prediction loss, respectively. More specifically, where

is the class probability of the instance

; , where and are the centroid coordinates, width, and height of predicted and ground-truth bounding box for the instance . Additionally, we have the function defined as: .

, where is the predicted mask probability and is the ground-truth mask label at pixel in a region.

Typical steatosis segmentation results are demonstrated at the bottom part of Fig. 1. The output image on the left presents the steatosis mask prediction with individual steatosis objects color-coded, whereas output images in the middle and right illustrate the predicted steatosis bounding boxes and the classification probabilities, respectively.

Figure 2: Segmentation process for a whole-slide microscopy image.

3 Experiments

With our prior segmentation method for nuclei [6], we generate segmentation masks at the highest image resolution. Our training data set is efficiently generated after a domain pathologist removes erroneous masks. The final data set contains 451 liver images of with ground-truth masks of which 387, 45, and 19 images are used for training, validation, and testing, respectively.

Random neural network initializations can result in an overwhelmingly expensive time cost for model training. Demonstrating its strength for problem solving at a reduced computational cost, transfer learning [15]

enables the pre-trained models to serve as the initial point for customized training, and has become popular in a large number of deep learning studies. In our experiment, network weights from the pre-trained COCO model are adopted to initialize our training process. Three backbones network structures, i.e. modified Resnet41, Resnet50 and Resnet65, are used, respectively. For these network backbones, we train the head layer for 30, 20, and 30 epochs, respectively. After head training, all layers are trained to achieve the best segmentation accuracy with 80, 50, 50 epochs, respectively. These epoch numbers are determined heuristically. To minimize the total loss in the training process, back-propagation and Stochastic Gradient Descent(SGD) are utilized. We run experiments on two GPUs (12GB RAM Tesla K80, NVIDIA Inc.) for 300 iterations, with six images per GPU for each mini-batch. The initial learning rate is 0.02, decreased by 10-fold for each 300 iterations. Additionally, online data augmentation techniques are used to further scale up data set size, improve the training performance, and increase generalizability and robustness of the trained model. These techniques include random affine transform, random flipping, and Gaussian blurring.

The process of applying the trained network to whole-slide microscopy images is presented in Fig. 2 where the left image presents the overall view of a representative whole-slide image, with a green box illustrating the close-up view of a small tissue part. The image in the middle demonstrates clustered steatosis droplets in a small liver tissue region at the full image resolution, while segmentation results of steatosis droplets by our trained network are illustrated with distinct colors in the right image.

To make this method flexible for diverse liver disease diagnosis and transplantation evaluation settings, we characterize each recognized steatosis candidate by the eccentricity, size, and perimeter, and record the segmentation score resulting from the neural network prediction. We provide user-defined thresholds for these features, enabling customized steatosis droplets retention. In this way, domain pathologists can have a convenient way to select cutoff values for these features and get readily informed of the number and the morphological profiles of the retained steatosis objects. We demonstrate the neural network segmentation results before and after such post-processing in Fig. 3 where we keep steatosis candidates with size, perimeter and eccentricity within , , and times of the average steatosis, respectively.

Figure 3: Segmentation result (Left) before and (Right) after post-processing, with black boxes in the left image indicating the discarded steatosis regions.
Figure 4: In columns from left to right, we demonstrate original images, ground-truth, segmentation results from method [6], Mask-RCNN with modified Resnet41, Resnet50, and modified Resnet65, respectively.

Representative segmentation results are presented in Fig. 4 where the original input images, ground-truth segmentation, results from our earlier method [6], outputs of presented deep learning approach with modified Resnet41, Resnet50 and modified Resnet65 backbones are presented in columns from left to right. It is noted that deep learning methods, especially the network with backbone Resnet50, present better results than our earlier work as they present less under-segmentation results. Additionally, deep learning methods present good performances on handling clumped steatosis clusters with complex topology.

To quantitatively evaluate the presented method, we compare results from deep learning algorithm with the ground-truth data. Table 1 presents evaluation results of averaged steatosis measures by four metrics, including Average Precision, Recall Ratio, F1-score, and Jaccard index. It is noticed that the trained network with Resnet50 achieves the best precision, recall ratio, F1-score, and Jaccard index.

4 Conclusion

We propose to use the deep learning method to solve the overlapped liver steatosis segmentation problem. Due to lack of labelled steatosis droplets from liver microscopy images for training, we modify our earlier nuclei segmentation method to generate liver steatosis training data after an efficient screening process by a domain expert. The trained neural network model is demonstrated to segment liver steatosis droplets, especially those in clumps with promising accuracy. Quantitative evaluations suggest that deep learning technology enables accurate and high-performance steatosis segmentation, a promising tool for enhancing liver disease diagnosis and transplantation assessment.

Method AP Recall F1-score Jaccard
Clump_seg [6] 52.18% 45.50% 45.03% 67.42%
ResNet41 67.61% 58.37% 62.06% 73.18%
ResNet50 75.87% 60.66% 65.88% 76.97%
ResNet65 69.55% 55.60% 61.69% 74.38%
Table 1: Method performance evaluation and comparison.

5 Acknowledgement

This research is supported in part by grants from National Institute of Health K25CA181503 and National Science Foundation ACI 1443054 and IIS 1350885, and CNPq.


  • [1] Lee, M.J. and Bagci, P. and Kong, J. and Vos, M.B. and Sharma, P. and Kalb, B. and Martin, D.R. and Adsay, N.V. and Farris, A.B., “Liver steatosis assessment: correlations among pathology, radiology, clinical data and automated image analysis software,” Patho. Res. & Prac., vol:209(6), pp.371–379, 2013.
  • [2] Chalasani, N. and Younossi, Z. and Lavine, J.E. et al., “The diagnosis and management of non-alcoholic fatty liver disease: practice guideline by the American Gastroenterological Association, American Association for the Study of Liver Diseases, and American College of Gastroenterology,” Gastroenterology, vol:142(7), pp.1592–1609, 2012.
  • [3] Choi, W.T. and Jen, K.Y. and Wang, D.L. and Tavakol, M. and Roberts, J.P. and Gill, R.M., “Donor Liver Small Droplet Macrovesicular Steatosis Is Associated with Increased Risk for Recipient Allograft Rejection,” Am. J. Surg. Pathol., vol:41(3), pp.365–373, 2017.
  • [4] Marsman, H. and Matsushita, T. and Dierkhising, R., “Assessment of donor liver steatosis: pathologist or automated software?,” Human pathology, vol:35(4), pp.430–435, 2004.
  • [5] He, K.M. and Gkioxari, G. and Dollár, P. and Girshick, R., “Mask R-CNN,”

    IEEE International Conference on Computer Vision

    , pp.2980–2988, 2017.
  • [6] Guo, X.Y. and Yu, H.Y. and Rossetti, B. and Teodoro, G. and Brat, D.J. and Kong, J., “Clumped Nuclei Segmentation with Adjacent Point Match and Local Shape based Intensity Analysis for Overlapped Nuclei in Fluorescence In-Situ Hybridization Images,” IEEE International Conference on Engineering in Medicine and Biology, pp.3410–3413, 2018.
  • [7] Zaitoun, A.M. and Al Mardini, H. and Awad, S. et al., “Quantitative assessment of fibrosis and steatosis in liver biopsies from patients with chronic hepatitis,” J. Clin. Pathol., vol:54(6), pp.461–465, 2001.
  • [8] Liquori, G.E. and Calamita, G. and Cascella, D. and Mastrodonato, M. and Portincasa, P. and Ferri, D.,

    “An innovative methodology for the automated morphometric and quantitative estimation of liver steatosis,”

    Histol. Histopathol., vol:24(1), pp.49–60, 2009.
  • [9] Fiorini, R.N. and Kirtz, J. and Periyasamy, B. et al., “Development of an unbiased method for the estimation of liver steatosis,” Clin. Transplant, vol:18(6), pp.700–706, 2004.
  • [10] Roy, M. and Wang, F.S. and Teodoro, G. and Vos, M.B. and Farris, A.B. and Kong, J., “Segmentation of Overlapped Steatosis in Whole-Slide Liver Histopathology Microscopy Images,” IEEE International Conference on Engineering in Medicine and Biology, pp.810–813, 2018.
  • [11] Homeyer, A. and Nasr, P. and Engel, C. et al., “Automated quantification of steatosis: agreement with stereological point counting,” Diagnostic pathology, vol:12(1), pp. 80, 2017 pp.1592–1609, 2012.
  • [12] Kong, J. and Lee, M.J. and Bagci, P. and Sharma, P. et al., “Computer-based image analysis of liver steatosis with large-scale microscopy imagery and correlation with magnetic resonance imaging lipid analysis,” IEEE International Conference on Bioinformatics and Biomedicine, pp.333–338, 2011.
  • [13] Ren, S.Q. and He, K.M. and Girshick, R. and Sun, J., “Faster r-cnn: Towards real-time object detection with region proposal networks,” Advances in neural information processing systems, pp.91–99, 2015.
  • [14] He, K.M. and Zhang, X.Y. and Ren, S.Q. and Sun, J., “Deep residual learning for image recognition,” IEEE Intl. Conf. on CVPR, pp.770–778, 2016.
  • [15] Pan, S.J. and Yang, Q., “A survey on transfer learning,” IEEE Transactions on knowledge and data engineering, pp.1345-1359, 2010.