Scoliosis is an abnormal condition defined by spinal curvature towards the left or right. Early detection is key and, when accurate, it can lead to better treatment planning 
. Radiography (X-Ray) is the preferred imaging technique for clinical analysis and measurement of scoliosis as it is highly available, inexpensive, and yields quick results. Conventional spine image analysis tasks involve tedious manual labor with hand-crafted feature extraction for the measurement of scoliosis. Cobb angle, the standard quantification of scoliosis is estimated by calculating the angle between the two tangents of the upper and lower end plates of the upper and lower vertebrae. A person with aor greater Cobb angle is usually considered for scoliosis diagnosis . Fig. 1 illustrates the calculation of the Cobb angle and the labeling of relevant vertebrae in an X-Ray image.
Conventionally, measurement and assessment, which requires the identification and labeling of specific vertebral structures, is manually performed by clinicians. However, the manual measurement of scoliosis faces several difficulties. First, large anatomical variation between patients and low tissue contrast in spinal X-Ray images make it challenging to accurately and reliably assess the severity of scoliosis , and effects on the spine and body as a whole, as well as on individual vertebra, pose extra difficulty in the quantification of scoliosis . Second, measurement error is prevalent in the routine clinical assessment of scoliosis due to instrumentation, vertebral rotation, and patient positioning , and – intra- or greater inter-observer variation has commonly been reported in measuring the Cobb angle [2, 14].
Therefore, an automatic technique for the accurate measurement of scoliosis is desirable. Our specific contributions in this paper are the following:
A fully automatic and efficient pipeline for the measurement and analysis of scoliosis.
A novel segmentation network for accurately segmenting vertebrae from spine X-Ray images.
Fully automatic and accurate identification and labeling of individual vertebrae merely based on binary segmentation.
Accurate diagnostic classification of the severity of scoliosis, which is crucial for treatment planning.
Ii Related Work
While several methods for vertebrae segmentation and scoliosis measurement are available, this approach is still under-explored in the literature. Existing vertebrae segmentation methods rely on manual interaction , hand-crafted feature engineering limited to customized parameters [18, 1], follow patch-based approaches that lose full spatial context [15, 5], are limited in scope and fail to consider all the required vertebrae at a time , etc. For Cobb angle estimation, a minimum bounding rectangle was used for the patch-wise segmented vertebrae , an approach that relies on pre-processing steps including spinal region isolation and vertebrae detection. Kusuma et al. 
proposed a K-means and curve-fitting approach for Cobb angle measurement that requires a set of pre-processing steps. Other Cobb angle estimation methods have been proposed based on directly finding vertebrae corners as a form of regression task [21, 17, 8, 20]. Although promising, these supervised methods are less viable for clinical applications because of low accuracy, due to the loss of fine details in the process, and the lack of explainability.
As a departure from prior segmentation-based methods, our model is fully automatic, involving no manual intervention end-to-end, and eschews any kind of pre-processing or post-processing steps.
Iii-a Vertebrae Segmentation and Labeling
We perform binary segmentation of the spine with a well-distinguishable number of vertebrae relevant to scoliosis analysis. To formulate the problem, we assume an unknown data distribution over images and vertebrae segmentation labels . The model has access to the labeled training set sampled i.i.d. from . As illustrated in Algorithm 1, the segmentation prediction network is trained with a set of learnable parameters . We specify the objective as , where is the reference vertebrae mask and is the model prediction in each of the training iterations.
Following the progressive dense V-net model [6, 7], we propose a progressive U-Net with some careful adjustments in the U-Net . As shown in Fig. 2, our model has an encoder and a decoder with skip connections. In each encoder layer, two
convolutions are followed by instance normalization, ReLU activation, and amax-pooling. A dropout is applied in every encoder and decoder stage of the network. We generate side-outputs in every stage of the decoder. Progressively adding one side-output to the next, the segmentation performance is improved compared to collecting the final output from the final decoder stage in a U-Net. However, one key difference with  is that our model is trained without side-supervision. Only the side-outputs are generated and added progressively, yielding an improved segmentation at the final output. A convolution operation is performed to generate the side-output from each decoder stage. The progressive side-outputs also ensure that micro-structure is not lost from any level of the decoder through the convolutional operations. We generate side outputs at , , and resolutions before the final output at resolution. Therefore, the side output at resolution is added to the next decoder stage, and so on.
Iii-B Measurement of Scoliosis
Our pipeline makes use of the vertebrae segmentation in estimating Cobb angles. Algorithm 2 automatically calculates the Cobb angle by analyzing the contours from the segmented mask. When well-separated from others, each of the contours represents a vertebra relevant to the measurement of scoliosis. To verify if a contour is actually associated to a relevant vertebra, we impose a minimum size on the number of contour pixels (). After the extraction and ordering of corners, the most tilted upper vertebra and the most tilted lower vertebra are determined from the relevant vertebrae (Fig. 3). Then the Cobb angle is calculated from the slopes of the upper edge of the upper vertebra and the lower edge of the lower vertebra.
Moreover, the severity of scoliosis can be categorized and appropriate treatment planning is performed depending on the calculated Cobb angle from the spine X-Ray of a patient. In our pipeline, we therefore perform an automatic diagnostic classification following the clinically recognized scoliosis severity classes, as shown in Table I. Active treatment is typically not needed when it is mild and rigid braces can stop the progression of scoliosis when it is in moderate stage. Surgery is the last resort for severe cases, but it can be delayed for the adolescent period .
|Cobb Angle ()||Severity||Treatment Recommendation|
|mild||Check in every 2 years|
|moderate||Wear a brace for 16–23 hours/day|
|severe||Revision surgery in 20–30 years|
Iv Experimental Evaluation
Iv-a Implementation Details
Data: We use a dataset of 100 high-resolution spine X-Ray images of children with evidence of scoliosis to various extents. The dataset contains manual annotation by experts of 18 relevant vertebrae (cervical C7, thoracic T1–T12, lumbar L1–L5). We split the dataset into training (80), testing (15), and validation (5) sets. Baselines:
As baselines, we use a regular U-Net model with a choice of binary cross-entropy (XE) and Dice as loss function. For simplicity, we denote the models as UD (UNet with Dice loss), UX (UNet with XE loss), PUD (Progressive UNet with Dice loss), and PUX (Progressive UNet with XE loss).Training: The models are trained on the training set while their performances were evaluated on the testing set. The validation set is used for hyper-parameter tuning and model selection. Inputs: All the images are resized and normalized to before feeding them to the network. Hyperparameters:
We use the Adam optimizer with adaptive learning rate starting with an initial rate of 0.01 and decreasing 10 times after every 20 epochs. We apply dropout with a rate of 0.25.Machine Configuration: We implemented Algorithm 1
in TensorFlow running on a Tesla P40 GPU in a system with a 64-bit Intel(R) Xeon(R) 440G CPU.Segmentation Evaluation: For segmentation evaluation, along with qualitative visualization of masks and edges, we use the Dice index (DI), structural similarity index (SSIM), average Hausdorff distance (HD), and F1 score (F1). Scoliosis Evaluation: For the evaluation of scoliosis, we measure Cobb angles, the indices of upper and lower tilted vertebrae, and severity classification. Since the expert annotations include only the vertebrae labels for segmentation reference, we follow the same scoliosis measurement procedure for both the reference measurements and for our progressive U-Net-based approach.
Iv-B Segmentation Results
Experimental results based on both qualitative and quantitative evaluations confirm the superiority of our model, which consistently provides improved segmentation with different losses (Dice and XE). Visualizations of the segmented vertebrae (Fig. 4 and Fig. 5) depict better distinctions of the individual vertebrae merely with binary segmentation. In all four quantitative measures, our models achieve better scores than the baseline models (Table II). The superiority of our models is further confirmed by the whisker-box plots in Fig. 6. Our end-to-end vertebrae segmentation achieves a better Dice similarity score than the recently published patch-wise segmentation method  (0.993 vs 0.952). While superior DI and F1 justifies the progressive addition of the side-outputs in pixel-wise predictions, better SSIM and HD depict the model’s ability to learn the intrinsic shape and structure of the segmented vertebrae.
Iv-C Scoliosis Results
For the evaluation of scoliosis, we compare the performance of our PUX model-based measurement against the reference measurement obtained by processing the expert’s annotations. As reported in Table III, our segmentation-based pipeline achieves very accurate Cobb angles. Good agreement is observed between our model and the reference measurement in each of the X-Rays in the test set with a mean angle difference of just 2.41 degrees, which is well below the acceptable error limit recommended by the experts . Comparing with some of the existing Cobb angle measurement techniques, our method achieves lower measurement error than those reported in  and . Moreover, the categorization of scoliosis  indicates 100% diagnostic accuracy of our approach relative to the reference.
|Test||Reference Measurement||Predicition of our PUX Model|
|ID||Upper Vert||Lower Vert||Cobb Angle||Severity||Upper Vert||Lower Vert||Cobb Angle||Severity|
The accurate and reliable segmentation of vertebrae is a prerequisite for the effective measurement of scoliosis. To this end, we have established a new state-of-the-art in fully automatic vertebrae segmentation in spinal X-Ray images. Our novel framework for accurately assessing scoliosis from anterior-posterior spine radiographs makes use of an end-to-end model that can accurately and reliably segments spinal vertebrae, outputting a vertebrae segmentation mask that enables the accurate measurement of scoliosis through calculation of the Cobb angle. Our pipeline promises to be an effective tool for the clinical diagnosis of scoliosis as well as for decision support in treatment planning. We envision combining the measurement of scoliosis with the training phase such that our model can make more intelligent predictions.
-  (2012) Automatic quantification of spinal curvature in scoliotic radiograph using image processing. Journal of Medical Systems 36 (3), pp. 1943–1951. Cited by: §II.
-  (1993) Diurnal variation of cobb angle measurement in adolescent idiopathic scoliosis.. Spine 18 (12), pp. 1581–1583. Cited by: §I.
-  (2002) Imaging in scoliosis: what, why and how?. Clinical radiology 57 (7), pp. 543–562. Cited by: §IV-C.
-  (2012) School screening for scoliosis: can surface topography replace examination with scoliometer?. Scoliosis 7 (1), pp. 9. Cited by: §IV-C.
Cobb angle measurement of spine from x-ray images using convolutional neural network. Computational and Mathematical Methods in Medicine 2019. Cited by: §II, §IV-B, §IV-C.
-  (2018) Automatic segmentation of pulmonary lobes using a progressive dense V-network. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 282–290. Cited by: §III-A.
-  (2019) Fast and automatic segmentation of pulmonary lobes from chest ct using a progressive dense V-network. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization 0 (0), pp. 1–10. Cited by: §III-A.
-  (2019-12) Bipartite distance for shape-aware landmark detection in spinal X-ray images. In Medical Imaging Meets NeurIPS Workshop, Vancouver, Canada. Cited by: §II.
-  (1997) Scoliosis quantification: an overview. The Journal of the Canadian Chiropractic Association 41 (3), pp. 137. Cited by: §I.
-  (2010) Scoliosis imaging: what radiologists should know. Radiographics 30 (7), pp. 1823–1842. Cited by: §I, §I.
-  (2017) Determination of spinal curvature from scoliosis X-ray images using K-means and curve fitting for early detection of scoliosis disease. ICITISEE. Cited by: §II, §IV-C.
-  (2019) Iterative fully convolutional neural networks for automatic vertebra segmentation and identification. Medical Image Analysis 53, pp. 142–155. Cited by: §II.
-  (2019) Semi-automatic spine segmentation method of ct data. In Mechatronics 2019: Recent Advances Towards Industry 4.0, Cited by: §II.
-  (1994) Variation in cobb angle measurements in scoliosis. Skeletal radiology 23 (7), pp. 517–520. Cited by: §I.
Vertebrae segmentation via stacked sparse autoencoder from computed tomography images. In Eleventh International Conference on Digital Image Processing (ICDIP 2019), Vol. 11179, pp. 111794K. Cited by: §II.
-  (2015) U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241. Cited by: §III-A.
-  (2017) Direct estimation of spinal cobb angles by structured multi-output regression. In International Conference on Information Processing in Medical Imaging, pp. 529–540. Cited by: §II.
-  (2019) Automated ct bone segmentation using statistical shape modelling and local template matching. Computer Methods in Biomechanics and Biomedical Engineering 22 (16), pp. 1303–1310. Cited by: §II.
-  (2008) Adolescent idiopathic scoliosis. The Lancet 371 (9623), pp. 1527–1537. Cited by: §I.
-  (2017) Automatic landmark estimation for adolescent idiopathic scoliosis assessment using boostnet. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 127–135. Cited by: §I, §II.
-  (2018) Automated comprehensive adolescent idiopathic scoliosis assessment using MVC-Net. Medical image analysis 48, pp. 1–11. Cited by: §II.
-  (2016) Early-onset scoliosis: a review of history, current treatment, and future directions. Pediatrics 137 (1), pp. e20150709. Cited by: §III-B.