Ultrasonic imaging relies on the principle of acoustic energy propagating through tissues that is partially absorbed, attenuated or backscattered to a varying extent based on the nature of different tissues. The backscattered signal is received by the transducer and processed to render a B-mode image. Radiologist’s infer the anatomy of an organ and pathology of a lesion by visually observing the stochastic characteristics of speckles. However, co-located heterogeneous tissue composition often results in non-unique stochastic patterns in speckle intensity. This is derived as a result of mixing of the backscattered signals, thereby challenging the visual reader’s ability. The challenges associated with reporting ultrasound images are primarily on account of (i) the stochastic nature of speckle intensity and the low signal-to-noise ratio leading to lowered contrast between structures , (ii) imaging and volume rendering artifacts introduced due to breathing and other kinds of body motion during imaging , (iii) motion induced artifacts during freehand scanning of 2D frames that are subsequently registered and stacked to form 3D volume .
In this paper, we have presented a fully convolutional neural network based method for segmentation of anatomical structures in ultrasound volumes using a frame-based semantic segmentation approach. We introduce symbiosis of aspects of two popular deep learning based semantic segmentation networks in the framework illustrated in Fig.1. It features (i) a SegNet 
like encoder-decoder type model with pooling index transfer from encoder for upsampling at the decoder at matched depth, (ii) encoder initialization with weights obtained from an ImageNet pre-trained VGG11, and (iii) activation transfer from encoder to decoder for concatenation as featured in U-Net . These features together help us build a model combining the strengths of the computationally efficient semantic segmentation networks for natural images , along with the one for biomedical images 
. Furthermore, we employ a weighted loss function similar to during the learning process to preserve the contour.
This paper is organized as follows. The existing techniques for ultrasound segmentation are briefly described in Sec. 2. The exposition to the solution is detailed in Sec. 3. The description of data used, experiment performed, results obtained, and their discussions are elucidated in Sec. 4 and the work is concluded in Sec. 5.
2 Prior Art
We present a comprehensive review of the prior art for segmentation in two widely used imaging methods namely (i) inside-out imaging such as with intravascular ultrasound (IVUS) and (ii) outside-in imaging such as for the thyroid.
IVUS Segmentation: Earlier approaches primarily employed non neural network based approaches such as those based on minimization of the probabilistic cost function 
. This method employed a support vector machine (SVM) trained on the samples of the first frame in a pullback to segment lumen. Baloccoet al.  reported a comprehensive evaluation of multiple approaches through the 2011 MICCAI workshop, where the methods were tested on a common data. In an earlier work  making use of backscattering physics based learning model inspired from , we had proposed an automated algorithm for segmentation of lumen and external elastic laminae which was subsequently extended to use in both outside-in and inside out imaging in . A fully convolutional network-based architecture, named IVUS-Net  is proposed for semantic segmentation of IVUS. Another fully convolutional neural network based architecture based on modified UNet  is proposed for semantic segmentation of IVUS and demonstrated only using MHz data.
Thyroid Segmentation: Prior art included use of level-set active contours model 14]. The main advantage of those models are to reduce the effect of intensity inhomogeneity. Later on, feedforward neural networks have been used for segmentation of thyroid . In , three semi-automated approaches based on the level set, graph cut, and feature classification for thyroid segmentation were proposed, while in  authors introduced a multi-organ method for segmenting thyroid gland, carotid artery, muscles, and the trachea.
These algorithms can be grouped as being semi-automatic where they need some basic information about the organ contour, or they employ a prior knowledge about the speckle pattern of the region of interest and the geometric structure of the organ. The algorithms fail to detect the organ boundary, once image quality degrades and contour size decrease. The predominant challenge is that the processing time consumed by these algorithms remains significantly high. Most of the algorithms are not currently usable for 3D volumetric analysis. The motivation of this paper is to resolve the above-mentioned limitations using a deep fully convolution neural network (FCN) for semantic segmentation which we demonstrate here for both inside-out and outside-in ultrasound imaging.
|P1 - P8 |
|2D SegNet |
|2D UNet |
|3D UNet |
The architecture of the FCN used is illustrated in Fig. 2. It consists of an encoder unit, consisting of convolution and maxpooling layers and a decoder unit consisting of convolution and unpooling layers. The encoder resembles a VGG11  barring the terminal classification layer. Feature maps are downsampled, and the number of features maps increases progressively with layer. The decoder is built to match activation map sizes in layers matched to depth in encoder with use of pooling index transferred for unpooling operation. This improves the boundary segmentation preserving conformity around small structures which is similar to the SegNet 
. However, max pooling in encoder with a SegNet typically leads to the loss in subtle information related to smaller structures which progressively vanishes across successive depths. To address these limitations, we introduce the activation concatenation concept similar to that used in the UNet
which is of relevance to biomedical image segmentation tasks. We have employed ReLU
as an activation function in all layer except for the terminal layer of decoder where we have used sigmoid activation function to force the output towards the [0,1] extremity. The network is trained with weighted binary cross-entropy (WCE)  loss function. Weights are calculated using a morphological distance transform giving higher weights to the pixels closer to the contour as illustrated for the IVUS in Fig. 3(c) and Fig. 3(j) for thyroid. The idea of using both maxpool indices and concatenating techniques helps in improving the preciseness of localization. The input is a grayscale image, and the first layer of the encoder provides an output with channel feature map to match input size expected in a VGG11.
4 Experiments, Results and Discussions
A -fold patient wise cross validation has been used with leave-one-patient-out validation.
Intravascular Ultrasound Segmentation: We have used IVUS data from the 2011 MICCAI workshop  which is composed of pullbacks of different patients acquired at MHz (Dataset B). The proposed method is compared with nine different methods, where eight approaches have been reported earlier in . In Table 1, we have included only the best results among the participants (P1-P8) in . Subsequently 2D visualization of segmented IVUS contours are presented in Fig. 3(d) - (g) and the lumen and external elastic luminae border segmentation in the whole pullback is visualized111Supplementary material. The proposed method outperforms the recent related prior art  as presented in Table 1.
|Narayan et al. |
|Chang et al. |
|Garg et al. |
|2D SegNet |
|2D UNet |
|3D UNet |
Thyroid Segmentation: The thyroid data used in this experiment has been acquired from a publicly available dataset  which includes freehand acquired thyroid US volumes from healthy human subjects imaged with a MHz probe. The proposed approach is compared with four different algorithms which has been reported by Narayan et. al. , (Table. 2). The only limitation in trying to compare commonality is that the dataset used for this comparison is not the same as the dataset used in prior art . 2D visualization of segmentation is shown in Fig. 3(k) - (n) and the thyroid segmentation in the whole volume is visualized 11footnotemark: 1. Table 2 shows that our proposed frame work obtain better results than the prior art.
We have experimentally trained the network on a PC with Intel Core i5-8600K CPU, 32GB of system RAM, Nvidia Tesla K40c GPU with 12GB DDR5 RAM. The codes are implemented in Python 3.6 and PyTorch 0.4, Nvidia CUDA 9.0 and cuDNN 5.1 on Ubuntu 16.04 LTS OS. The proposed network is trained with a learning rate of, and batch size of 14. Adam optimizer  has been used to learn the network weights during training. A CPU platform was used for 3D CNN experiments with large scale memory requirements (Table 3), consists of Intel Xeon 8160 CPU with GB DDR4 ECC Regd. RAM, Ubuntu 16.04 LTS OS, PyTorch 0.4 on Intel Distribution of Python 3.6 with Intel MKL acceleration.
The proposed network performs better featuring max-unpooling which helps preserve local speckle information, better than transposed convolution where loss of local speckle information due to 4 padding.
|Method||Target||Input size||Batch size||
|3D UNet ||CPU||1||14||154||29.40||0.145|
In this paper, we have presented a frame work for automated segmentation of ultrasound volumes using a convolutional neural network. The performance of the proposed architecture is compared with recent prior art, and it is observed that the proposed method outperforms prior art. This network has an advantages to segment boundary region precisely due to the use of WCE as loss function. Activation concatenation concept similar to UNet has been introduced to overcome the critical information loss which happens due to pooling operation and the network is computationally less expensive.
-  N. S. Narayan, P. Marziliano, J. Kanagalingam, and C. GL. Hobbs, “Speckle patch similarity for echogenicity-based multiorgan segmentation in ultrasound images of the thyroid gland,” IEEE J. Biomed. Health Inform., vol. 21, no. 1, pp. 172–183, 2017.
-  D. Sheet, A. Karamalis, A. Eslami, P. Noël, J. Chatterjee, A. K. Ray, A. F. Laine, S. G. Carlier, and A. Navab, N.and Katouzian, “Joint learning of ultrasonic backscattering statistical physics and signal confidence primal for characterizing atherosclerotic plaques using intravascular ultrasound,” Med. Image Anal., vol. 18, no. 1, pp. 103–117, 2014.
-  T. Wunderling, B. Golla, P. Poudel, C. Arens, M. Friebe, and C. Hansen, “Comparison of thyroid segmentation techniques for 3d ultrasound,” in Proc. Med. Imaging Image Process., 2017, vol. 10133, p. 1013317.
-  V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” IEEE Trans. Patt. Anal. Mach. Intell., vol. 39, no. 12, pp. 2481–2495, 2017.
-  K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” Proc. Int. Conf. Learn. Represent., 2015.
-  O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Proc. Int. Conf. Med. Image comput. Comput. Assist. Interv., 2015, pp. 234–241.
-  E. G. Mendizabal-Ruiz, M. Rivera, and I. A. Kakadiaris, “Segmentation of the luminal border in intravascular ultrasound b-mode images using a probabilistic approach,” Med. Image Anal., vol. 17, no. 6, pp. 649–670, 2013.
-  S. Balocco, C. Gatta, F. Ciompi, A. Wahle, P. Radeva, S. Carlier, G. Unal, E. Sanidas, J. Mauri, X. Carillo, et al., “Standardized evaluation methodology and reference database for evaluating ivus image segmentation,” Comput. Med. Imaging Graph., vol. 38, no. 2, pp. 70–90, 2014.
-  D. China, P. Mitra, and D. Sheet, “On the fly segmentation of intravascular ultrasound images powered by learning of backscattering physics,” in Classification in BioApps, pp. 351–380. 2018.
-  D. China, A. Illanes, P. Poudel, M. Friebe, P. Mitra, and D. Sheet, “Anatomical structure segmentation in ultrasound volumes using cross frame belief propagating iterative random walks.,” IEEE J. Biomed. Health Inform., 2018.
-  J. Yang, L. Tong, M. Faraji, and A. Basu, “Ivus-net: An intravascular ultrasound segmentation network,” arXiv preprint arXiv:1806.03583, 2018.
-  S. Kim, Y. Jang, B. Jeon, Y. Hong, H. Shim, and H. Chang, “Fully automatic segmentation of coronary arteries based on deep neural network in intravascular ultrasound images,” in Intravascular Imaging, Comp. Assist. Stenting, Large-Scale Anno., Biomed. Data, Expert Label Syn., pp. 161–168. 2018.
-  M. A. Savelonas, D. K. Iakovidis, I. Legakis, and D. Maroulis, “Active contours guided by echogenicity and texture for delineation of thyroid nodules in ultrasound images,” IEEE Trans. Inf. Technol. Biomed., vol. 13, no. 4, pp. 519–527, 2009.
-  C. Chang, Y. Lei, C. Tseng, and S. Shih, “Thyroid segmentation and volume estimation in ultrasound images,” IEEE Trans. Biomed. Engg., vol. 57, no. 6, pp. 1348–1357, 2010.
-  Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3d u-net: learning dense volumetric segmentation from sparse annotation,” in Proc. Int. Conf. Med. Image Comput. Comput. Assist. Interv., 2016, pp. 424–432.
-  I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, Deep learning, vol. 1, 2016.