Analysis of Scoliosis From Spinal X-Ray Images

04/15/2020 ∙ by Abdullah-Al-Zubaer Imran, et al. ∙ 7

Scoliosis is a congenital disease in which the spine is deformed from its normal shape. Measurement of scoliosis requires labeling and identification of vertebrae in the spine. Spine radiographs are the most cost-effective and accessible modality for imaging the spine. Reliable and accurate vertebrae segmentation in spine radiographs is crucial in image-guided spinal assessment, disease diagnosis, and treatment planning. Conventional assessments rely on tedious and time-consuming manual measurement, which is subject to inter-observer variability. A fully automatic method that can accurately identify and segment the associated vertebrae is unavailable in the literature. Leveraging a carefully-adjusted U-Net model with progressive side outputs, we propose an end-to-end segmentation model that provides a fully automatic and reliable segmentation of the vertebrae associated with scoliosis measurement. Our experimental results from a set of anterior-posterior spine X-Ray images indicate that our model, which achieves an average Dice score of 0.993, promises to be an effective tool in the identification and labeling of spinal vertebrae, eventually helping doctors in the reliable estimation of scoliosis. Moreover, estimation of Cobb angles from the segmented vertebrae further demonstrates the effectiveness of our model.



There are no comments yet.


page 2

page 3

page 4

page 5

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Scoliosis is an abnormal condition defined by spinal curvature towards the left or right. Early detection is key and, when accurate, it can lead to better treatment planning [19]

. Radiography (X-Ray) is the preferred imaging technique for clinical analysis and measurement of scoliosis as it is highly available, inexpensive, and yields quick results. Conventional spine image analysis tasks involve tedious manual labor with hand-crafted feature extraction for the measurement of scoliosis. Cobb angle, the standard quantification of scoliosis is estimated by calculating the angle between the two tangents of the upper and lower end plates of the upper and lower vertebrae. A person with a

or greater Cobb angle is usually considered for scoliosis diagnosis [10]. Fig. 1 illustrates the calculation of the Cobb angle and the labeling of relevant vertebrae in an X-Ray image.

Conventionally, measurement and assessment, which requires the identification and labeling of specific vertebral structures, is manually performed by clinicians. However, the manual measurement of scoliosis faces several difficulties. First, large anatomical variation between patients and low tissue contrast in spinal X-Ray images make it challenging to accurately and reliably assess the severity of scoliosis [20], and effects on the spine and body as a whole, as well as on individual vertebra, pose extra difficulty in the quantification of scoliosis [9]. Second, measurement error is prevalent in the routine clinical assessment of scoliosis due to instrumentation, vertebral rotation, and patient positioning [10], and intra- or greater inter-observer variation has commonly been reported in measuring the Cobb angle [2, 14].

Fig. 1:

Overview of our framework for calculating the Cobb angle in a spine X-Ray through segmentation, labeling, and identification of the relevant vertebrae. After determining the most tilted vertebrae above and below the apex, tangents are drawn by extending the upper edge of the upper vertebra and lower edge of the lower vertebra. From these tangents, the Cobb angles are calculated and the scoliosis can be classified.

Therefore, an automatic technique for the accurate measurement of scoliosis is desirable. Our specific contributions in this paper are the following:

  1. A fully automatic and efficient pipeline for the measurement and analysis of scoliosis.

  2. A novel segmentation network for accurately segmenting vertebrae from spine X-Ray images.

  3. Fully automatic and accurate identification and labeling of individual vertebrae merely based on binary segmentation.

  4. Accurate diagnostic classification of the severity of scoliosis, which is crucial for treatment planning.

Ii Related Work

While several methods for vertebrae segmentation and scoliosis measurement are available, this approach is still under-explored in the literature. Existing vertebrae segmentation methods rely on manual interaction [13], hand-crafted feature engineering limited to customized parameters [18, 1], follow patch-based approaches that lose full spatial context [15, 5], are limited in scope and fail to consider all the required vertebrae at a time [12], etc. For Cobb angle estimation, a minimum bounding rectangle was used for the patch-wise segmented vertebrae [5], an approach that relies on pre-processing steps including spinal region isolation and vertebrae detection. Kusuma et al. [11]

proposed a K-means and curve-fitting approach for Cobb angle measurement that requires a set of pre-processing steps

[11]. Other Cobb angle estimation methods have been proposed based on directly finding vertebrae corners as a form of regression task [21, 17, 8, 20]. Although promising, these supervised methods are less viable for clinical applications because of low accuracy, due to the loss of fine details in the process, and the lack of explainability.

As a departure from prior segmentation-based methods, our model is fully automatic, involving no manual intervention end-to-end, and eschews any kind of pre-processing or post-processing steps.

Fig. 2: Architecture of our segmentation network (Progressive U-Net): Side outputs at three different stages of the decoder are generated and progressively added to the next stage side-output. The output from the third side-output is added to the last stage before the final convolution to generate the final segmentation output.

Iii Methods

Iii-a Vertebrae Segmentation and Labeling

We perform binary segmentation of the spine with a well-distinguishable number of vertebrae relevant to scoliosis analysis. To formulate the problem, we assume an unknown data distribution over images and vertebrae segmentation labels . The model has access to the labeled training set sampled i.i.d. from . As illustrated in Algorithm 1, the segmentation prediction network is trained with a set of learnable parameters . We specify the objective as , where is the reference vertebrae mask and is the model prediction in each of the training iterations.

Input: X-Ray images and reference vertebra masks.
Output: Predicted vertebra masks.
  Training data including spine X-Ray images and reference vertebra segmentation masks
  Model architecture with learnable parameters
  for each step over  do
     Sample minibatch
     Compute model outputs for the minibatch:
     Calculate loss for the model predictions
     Update the model along its gradient
  end for
Algorithm 1 Training for vertebrae segmentation from spine X-Ray images.

Following the progressive dense V-net model [6, 7], we propose a progressive U-Net with some careful adjustments in the U-Net [16]. As shown in Fig. 2, our model has an encoder and a decoder with skip connections. In each encoder layer, two

convolutions are followed by instance normalization, ReLU activation, and a

max-pooling. A dropout is applied in every encoder and decoder stage of the network. We generate side-outputs in every stage of the decoder. Progressively adding one side-output to the next, the segmentation performance is improved compared to collecting the final output from the final decoder stage in a U-Net. However, one key difference with [6] is that our model is trained without side-supervision. Only the side-outputs are generated and added progressively, yielding an improved segmentation at the final output. A convolution operation is performed to generate the side-output from each decoder stage. The progressive side-outputs also ensure that micro-structure is not lost from any level of the decoder through the convolutional operations. We generate side outputs at , , and resolutions before the final output at resolution. Therefore, the side output at resolution is added to the next decoder stage, and so on.

Input: Vertebra mask .
Output: Cobb angle .
  From the predicted mask , get all the contours
  for each contour in contours do
     if Number of pixels  then
        //to remove any noisy patches
        Remove contour
     end if
  end for
  This will give contours of well-separated vertebrae
  Extract four corner points for the contour
  Order the corners from bottom to top by comparing the coordinates of the extracted corner points
  Find the two vertebrae (upper and lower) with at least 2 vertebrae gap between them
  Calculate the Cobb angle, , where and are the upper and lower vertebrae slopes.
Algorithm 2 Cobb angle calculation
Raw Mask Measurement
Fig. 3: From input X-ray image (left), to segmentation mask prediction (middle), to vertebrae identification and scoliosis measurement (right) in our pipeline.

Iii-B Measurement of Scoliosis

Our pipeline makes use of the vertebrae segmentation in estimating Cobb angles. Algorithm 2 automatically calculates the Cobb angle by analyzing the contours from the segmented mask. When well-separated from others, each of the contours represents a vertebra relevant to the measurement of scoliosis. To verify if a contour is actually associated to a relevant vertebra, we impose a minimum size on the number of contour pixels (). After the extraction and ordering of corners, the most tilted upper vertebra and the most tilted lower vertebra are determined from the relevant vertebrae (Fig. 3). Then the Cobb angle is calculated from the slopes of the upper edge of the upper vertebra and the lower edge of the lower vertebra.

Moreover, the severity of scoliosis can be categorized and appropriate treatment planning is performed depending on the calculated Cobb angle from the spine X-Ray of a patient. In our pipeline, we therefore perform an automatic diagnostic classification following the clinically recognized scoliosis severity classes, as shown in Table I. Active treatment is typically not needed when it is mild and rigid braces can stop the progression of scoliosis when it is in moderate stage. Surgery is the last resort for severe cases, but it can be delayed for the adolescent period [22].

Cobb Angle () Severity Treatment Recommendation
mild Check in every 2 years
moderate Wear a brace for 16–23 hours/day
severe Revision surgery in 20–30 years
TABLE I: Clinically accepted classification and treatment planning for adolescent scoliosis based on measured Cobb angles

Iv Experimental Evaluation

Iv-a Implementation Details

Data: We use a dataset of 100 high-resolution spine X-Ray images of children with evidence of scoliosis to various extents. The dataset contains manual annotation by experts of 18 relevant vertebrae (cervical C7, thoracic T1–T12, lumbar L1–L5). We split the dataset into training (80), testing (15), and validation (5) sets. Baselines:

As baselines, we use a regular U-Net model with a choice of binary cross-entropy (XE) and Dice as loss function. For simplicity, we denote the models as UD (UNet with Dice loss), UX (UNet with XE loss), PUD (Progressive UNet with Dice loss), and PUX (Progressive UNet with XE loss).

Training: The models are trained on the training set while their performances were evaluated on the testing set. The validation set is used for hyper-parameter tuning and model selection. Inputs: All the images are resized and normalized to before feeding them to the network. Hyperparameters:

We use the Adam optimizer with adaptive learning rate starting with an initial rate of 0.01 and decreasing 10 times after every 20 epochs. We apply dropout with a rate of 0.25.

Machine Configuration: We implemented Algorithm 1

in TensorFlow running on a Tesla P40 GPU in a system with a 64-bit Intel(R) Xeon(R) 440G CPU.

Segmentation Evaluation: For segmentation evaluation, along with qualitative visualization of masks and edges, we use the Dice index (DI), structural similarity index (SSIM), average Hausdorff distance (HD), and F1 score (F1). Scoliosis Evaluation: For the evaluation of scoliosis, we measure Cobb angles, the indices of upper and lower tilted vertebrae, and severity classification. Since the expert annotations include only the vertebrae labels for segmentation reference, we follow the same scoliosis measurement procedure for both the reference measurements and for our progressive U-Net-based approach.

Iv-B Segmentation Results

Experimental results based on both qualitative and quantitative evaluations confirm the superiority of our model, which consistently provides improved segmentation with different losses (Dice and XE). Visualizations of the segmented vertebrae (Fig. 4 and Fig. 5) depict better distinctions of the individual vertebrae merely with binary segmentation. In all four quantitative measures, our models achieve better scores than the baseline models (Table II). The superiority of our models is further confirmed by the whisker-box plots in Fig. 6. Our end-to-end vertebrae segmentation achieves a better Dice similarity score than the recently published patch-wise segmentation method [5] (0.993 vs 0.952). While superior DI and F1 justifies the progressive addition of the side-outputs in pixel-wise predictions, better SSIM and HD depict the model’s ability to learn the intrinsic shape and structure of the segmented vertebrae.

Fig. 4: Boundary visualization of the predicted vertebrae masks in a spine X-Ray shows consistent improvement by our model over all other models.
UD 0.970 0.961 5.246 0.896
UX 0.956 0.955 6.767 0.868
PUD 0.993 0.966 4.597 0.919
PUX 0.993 0.970 4.677 0.922
TABLE II: Performance comparison of the vertebrae segmentation models
Fig. 5: Visualization (zoomed) of the predicted vertebrae masks in a spine X-Ray shows consistent improvement by our model over all other models.

Iv-C Scoliosis Results

For the evaluation of scoliosis, we compare the performance of our PUX model-based measurement against the reference measurement obtained by processing the expert’s annotations. As reported in Table III, our segmentation-based pipeline achieves very accurate Cobb angles. Good agreement is observed between our model and the reference measurement in each of the X-Rays in the test set with a mean angle difference of just 2.41 degrees, which is well below the acceptable error limit recommended by the experts [3]. Comparing with some of the existing Cobb angle measurement techniques, our method achieves lower measurement error than those reported in [11] and [5]. Moreover, the categorization of scoliosis [4] indicates 100% diagnostic accuracy of our approach relative to the reference.

Fig. 6: Whisker-Box plots of all four models showing consistent performance of our model with varying losses in segmenting 18 scoliosis-relevant vertebrae from the spine X-ray test set.
Test Reference Measurement Predicition of our PUX Model
ID Upper Vert Lower Vert Cobb Angle Severity Upper Vert Lower Vert Cobb Angle Severity
1 T10 L3 mild T10 L3 mild
2 T12 L4 normal T6 L3 normal
3 T11 L3 mild T11 L3 mild
4 T5 T10 mild T5 T11 mild
5 T6 T10 mild T10 L4 mild
6 T11 L4 mild T11 L4 mild
7 T10 L2 moderate T11 L3 moderate
8 T6 T12 mild T6 T12 mild
9 T5 T11 mild T6 T11 mild
10 T6 T10 mild T1 L3 mild
11 T12 L4 moderate T12 L4 moderate
12 T10 L1 mild T9 L1 mild
13 T12 L3 mild T10 L3 mild
14 T12 L4 mild T8 T12 mild
15 T12 L3 mild T11 L3 mild
TABLE III: Performance of our method for calculating Cobb angle and scoliosis severity in the test set relative to reference measurements

V Conclusions

The accurate and reliable segmentation of vertebrae is a prerequisite for the effective measurement of scoliosis. To this end, we have established a new state-of-the-art in fully automatic vertebrae segmentation in spinal X-Ray images. Our novel framework for accurately assessing scoliosis from anterior-posterior spine radiographs makes use of an end-to-end model that can accurately and reliably segments spinal vertebrae, outputting a vertebrae segmentation mask that enables the accurate measurement of scoliosis through calculation of the Cobb angle. Our pipeline promises to be an effective tool for the clinical diagnosis of scoliosis as well as for decision support in treatment planning. We envision combining the measurement of scoliosis with the training phase such that our model can make more intelligent predictions.


  • [1] H. Anitha and G. Prabhu (2012) Automatic quantification of spinal curvature in scoliotic radiograph using image processing. Journal of Medical Systems 36 (3), pp. 1943–1951. Cited by: §II.
  • [2] M. Beauchamp, H. Labelle, G. Grimard, C. Stanciu, B. Poitras, and J. Dansereau (1993) Diurnal variation of cobb angle measurement in adolescent idiopathic scoliosis.. Spine 18 (12), pp. 1581–1583. Cited by: §I.
  • [3] V. Cassar-Pullicino and S. Eisenstein (2002) Imaging in scoliosis: what, why and how?. Clinical radiology 57 (7), pp. 543–562. Cited by: §IV-C.
  • [4] J. Chowanska, T. Kotwicki, K. Rosadzinski, and Z. Sliwinski (2012) School screening for scoliosis: can surface topography replace examination with scoliometer?. Scoliosis 7 (1), pp. 9. Cited by: §IV-C.
  • [5] M. Horng, C. Kuok, M. Fu, C. Lin, and Y. Sun (2019)

    Cobb angle measurement of spine from x-ray images using convolutional neural network

    Computational and Mathematical Methods in Medicine 2019. Cited by: §II, §IV-B, §IV-C.
  • [6] A. Imran, A. Hatamizadeh, S. P. Ananth, X. Ding, D. Terzopoulos, and N. Tajbakhsh (2018) Automatic segmentation of pulmonary lobes using a progressive dense V-network. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 282–290. Cited by: §III-A.
  • [7] A. Imran, A. Hatamizadeh, S. P. Ananth, X. Ding, N. Tajbakhsh, and D. Terzopoulos (2019) Fast and automatic segmentation of pulmonary lobes from chest ct using a progressive dense V-network. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization 0 (0), pp. 1–10. Cited by: §III-A.
  • [8] A. Imran, C. Huang, H. Tang, W. Fan, K. M.C. Cheung, M. To, Z. Qian, and D. Terzopoulos (2019-12) Bipartite distance for shape-aware landmark detection in spinal X-ray images. In Medical Imaging Meets NeurIPS Workshop, Vancouver, Canada. Cited by: §II.
  • [9] G. Kawchuk and R. McArthur (1997) Scoliosis quantification: an overview. The Journal of the Canadian Chiropractic Association 41 (3), pp. 137. Cited by: §I.
  • [10] H. Kim, H. S. Kim, E. S. Moon, C. Yoon, T. Chung, H. Song, J. Suh, Y. H. Lee, and S. Kim (2010) Scoliosis imaging: what radiologists should know. Radiographics 30 (7), pp. 1823–1842. Cited by: §I, §I.
  • [11] B. A. Kusuma (2017) Determination of spinal curvature from scoliosis X-ray images using K-means and curve fitting for early detection of scoliosis disease. ICITISEE. Cited by: §II, §IV-C.
  • [12] N. Lessmann, B. van Ginneken, P. A. de Jong, and I. Išgum (2019) Iterative fully convolutional neural networks for automatic vertebra segmentation and identification. Medical Image Analysis 53, pp. 142–155. Cited by: §II.
  • [13] M. Mateusiak and K. Mikolajczyk (2019) Semi-automatic spine segmentation method of ct data. In Mechatronics 2019: Recent Advances Towards Industry 4.0, Cited by: §II.
  • [14] J. Pruijs, M. Hageman, W. Keessen, R. Van Der Meer, and J. Van Wieringen (1994) Variation in cobb angle measurements in scoliosis. Skeletal radiology 23 (7), pp. 517–520. Cited by: §I.
  • [15] S. F. Qadri, Z. Zhao, D. Ai, M. Ahmad, and Y. Wang (2019)

    Vertebrae segmentation via stacked sparse autoencoder from computed tomography images

    In Eleventh International Conference on Digital Image Processing (ICDIP 2019), Vol. 11179, pp. 111794K. Cited by: §II.
  • [16] O. Ronneberger, P. Fischer, and T. Brox (2015) U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241. Cited by: §III-A.
  • [17] H. Sun, X. Zhen, C. Bailey, P. Rasoulinejad, Y. Yin, and S. Li (2017) Direct estimation of spinal cobb angles by structured multi-output regression. In International Conference on Information Processing in Medical Imaging, pp. 529–540. Cited by: §II.
  • [18] E. Taghizadeh, A. Terrier, F. Becce, A. Farron, and P. Büchler (2019) Automated ct bone segmentation using statistical shape modelling and local template matching. Computer Methods in Biomechanics and Biomedical Engineering 22 (16), pp. 1303–1310. Cited by: §II.
  • [19] S. L. Weinstein, L. A. Dolan, J. C. Cheng, A. Danielsson, and J. A. Morcuende (2008) Adolescent idiopathic scoliosis. The Lancet 371 (9623), pp. 1527–1537. Cited by: §I.
  • [20] H. Wu, C. Bailey, P. Rasoulinejad, and S. Li (2017) Automatic landmark estimation for adolescent idiopathic scoliosis assessment using boostnet. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 127–135. Cited by: §I, §II.
  • [21] H. Wu, C. Bailey, P. Rasoulinejad, and S. Li (2018) Automated comprehensive adolescent idiopathic scoliosis assessment using MVC-Net. Medical image analysis 48, pp. 1–11. Cited by: §II.
  • [22] S. Yang, L. M. Andras, G. J. Redding, and D. L. Skaggs (2016) Early-onset scoliosis: a review of history, current treatment, and future directions. Pediatrics 137 (1), pp. e20150709. Cited by: §III-B.