The combination of positron emission tomography and X-ray computed tomography (PET/CT) is used extensively both in both clinical and research environments where device up-time is a key operational metric for imaging organizations. During a PET scanner’s life, however, one or more block detectors in the imaging array will likely malfunction causing the system to lose coincidence events along the corresponding lines of response (LORs). As illustrated in Fig. 1, this creates gaps in the sinogram data which leads to artifacts and image degradation, as well as downtime for the scanner (Zito et al., 2007; Elhami et al., 2010; Voert et al., 2016). This scenario is likely to become even more common as technological advances lead to scanners with longer axial fields-of-view such as the development and commercialization of whole-body PET scanners (Cherry et al., 2018)
. This paper presents a deep learning based technique to mitigate single and multiple failed block detectors, while restoring reconstructed image quality.
Previous research on mitigating missing PET data from either intentional gaps due to the design of the PET scanner or unintentional regions of missing data due to the failure one or more block detectors largely focused on interpolation, model based or transform based methods. Sinogram repair using interpolation with linear, bilinear and bicubic methods along the radial, angular or axial directions is computationally efficient and reduces reconstructed image artifacts considerably(de Jong et al., 2003), but has been shown to have lower image quality than model (Baghaei et al., 2008) and transform (van Velden et al., 2008) based methods. A more advanced interpolation method comparable to the model based and transform methods involves collapsing the transradial slices where data is missing followed by bicubic interpolation based resizing (Peltonen et al., 2013). The model based methods use either a statistical (Kinahan et al., 1997; Baghaei et al., 2008; van Velden et al., 2008) or analytical (Baghaei et al., 2000)
reconstruction algorithm to create an initial image estimate from a sinogram with missing data and then forward project these images using the PET system model to estimate the missing data repeating the process iteratively. Transform based repair methods convert a sinogram from the spatial domain into the frequency domain and then filter the frequency domain coefficients corresponding to the missing data. The filtered data is then transformed back into the spatial domain and the process is repeated until a stopping criterion is satisfied. The constrained Fourier space (CFS) method(Karp et al., 1988), which employs a bow-tie shaped frequency filter, was the first transform based PET sinogram repair method widely studied (van Velden et al., 2008; Baghaei et al., 2008; Lehnert and Meikle, 2006; Buchert et al., 1999). The discrete cosine transform (DCT) was also explored for this purpose (Tuna et al., 2008) demonstrating promising results using a wedge shaped frequency filter.
There is little published research on deep learning based PET sinogram repair methods, but techniques similar to the methods proposed in this study are used in medical imaging for limited angle / low dose x-ray CT (Kan et al., 2016; Anirudh et al., 2018; Xie et al., 2018; Zhao et al., 2018; Li, 2019) and artifact mitigation (Zhang and Yu, 2018). Inpainting is a general topic of interest in the wider deep learning literature beginning with the work of Xie, Xu and Chen (Xie et al., 2012) who applied a stacked sparse auto-encoder to the problem and this was followed by numerous additional works that primarily applied generative adversarial networks (GANs) (Goodfellow et al., 2014) to the inpainting problem such as (Yeh et al., 2016; Yang et al., 2017; Wang et al., 2018; Yu et al., 2018). While we considered using a GAN for our proposed network, we decided the additional complexity was not necessary and opted for a simpler approach.
In a deep learning context, sinogram repair can be framed as mapping input sinograms with missing LORs to repaired sinograms by learning an inpainting neural network function parameterized by weights
and trained by minimizing a loss functionover a training data set with back-propagation and gradient descent (LeCun et al., 1988). The solution is given by
The performance of the Sinogram Repair Network presented in this paper for implementing was evaluated with whole-body patient data for one, two and four missing block detectors. Results were quantitatively compared in sinogram space and image space to the previously published sinogram repair methods of interpolation, the constrained Fourier space method and a model based method utilizing iterative reconstruction.
2 Materials and methods
2.1 Sinogram Repair Network
. The network consists of 23 convolutional layers with a contracting segment (8 layers) that extracts positional features and twice performs spatial down-sampling, a bottle neck segment (6 layers) that learns a compressed representation, and an expansive segment (8 layers) that spatially up-samples the compressed representation and extracts contextual information. The contracting and expansive segments are connected by skip connections that allow the network to combine the positional and contextual information to predict the missing sinogram information. The final sigmoid activation function helps provide stability during training.
All convolutional kernels are 5x5 except for the up-sampling layers that use 4x4 kernels. Padding is employed to ensure that the dimensions remain consistent after each convolution. Down-sampling to half-size in each dimension is accomplished by using a kernel stride of 2, and up-sampling is accomplished by using a fractionally strided convolution (also known as transposed convolution). After each spatial down-sampling, the number of kernels in a convolutional layer is doubled, and with each up-sampling the number of kernels in a convolutional layer is halved. Each convolutional layer is followed by batch normalization and a Leaky-ReLU activation function with a negative slope of 0.2.
2.2 Experimental Data Sets
The training and test data was acquired using a Siemens Biograph mCT scanner from 20 PET/CT patient scans using a whole-body non-time-of-flight protocol of normal clinical duration with approximately 2-3 minutes per bed position. Due to the varying distribution of the radioactive source and attenuating structures of the body, these types of PET scans produce a wide distribution of sinograms from sparse and noisy containing less than 20,000 counts up to densely packed sinograms with rich structures containing well over 300,000 counts. The resulting data set contained 66,798 raw (uncorrected) sinogram slices counting all direct and oblique planes. From this data set 2 randomly selected patients totalling 7,164 sinograms became the test set with the remaining slices used to train the Sinogram Repair Network. To augment the data set even further during training, the input slices were randomly flipped horizontally and vertically with a probability of 0.5. Since the original data set does not contain missing information, defective block detectors were simulated by generating a set of sinogram masks, one for each of the 48 block detectors in the PET gantry and randomly applying them during training. Although based on the original 59,634 training sinograms, this dynamic augmentation and random removal of LORs meant the neural network rarely, if ever, saw the same data sample during training, which significantly aided in generalization and preventing overfitting. With the 7,164 slices in the test set, 10 realizations of missing blocks were created for each of one, two, and four missing blocks with no additional augmentation.
Since the range of potential values in a sinogram are only limited by the size of the integer representing a given bin, which in this case is 16 bit, instability can occur during neural network training. To control this unbounded parameter each sinogram is scaled down to a range prior to input to the network and the output is scaled back up using the same scaling factor. Scaling is an important aspect to consider when working with sinogram data sets since measurements such as Standardized Uptake Values (SUV) are important quantitative diagnostic measures taken from the resulting images.
2.3 Network Training
The proposed Sinogram Repair Network was implemented using PyTorch(Paszke et al., 2017)
and trained for 4,500 epochs with each epoch utilizing 2,048 sinogram slices randomly sampled from the training set without replacement in mini-batches of 32. At runtime, randomly selected masks were applied to the sampled sinograms to create missing data. The Adam optimizer(Kingma and Ba, 2014)
was employed, which is an extension to classic stochastic gradient descent that maintains a learning rate for each network parameter separately, with an initial learning rate of 0.0001, a first momentum of 0.5 and a second momentum of 0.999. After eachepochs, a warm restart (Loshchilov and Hutter, 2017) was performed where the learning rate was reset to its initial value and the optimizer’s momentum buffers were cleared creating a temporary period of instability allowing the optimizer to escape sub-optimal local minimums. The value for was incremented by a value at each reset to increase the duration between warm restarts. We set the initial value of to 300 and to 50. Various loss functions were explored and we chose to follow previous research dedicated to loss functions for image repair (Zhao et al., 2017) and use a weighted combination of an element-wise L1 and multi-scale structural similarity (MS-SSIM) (Wang et al., 2003) based loss. Specifically,
where each sinogram and contains bins and the SSIM components are calculated on a sliding 11 x 11 window between sinograms and where is the mean,
is the variance andis the covariance of and .
where , and are constants given by
and is the dynamic range of values (1 in our case), and . The luminance component () is only evaluated at the highest scale whereas the contrast () and structure () measures are evaluated at five scales each half the dimension of the previous. The values of and were explored empirically with the results reported below based on and .
2.4 Evaluation Methods
The Sinogram Repair Network (SRN) performance is evaluated in sinogram, image, and quantitative accuracy domains. Comparisons are made against each of the top performing previously established repair methods, namely, interpolation, model based and transform based. For interpolation, the method described in (Peltonen et al., 2013) is used. The model based method uses OSEM reconstruction from the TomoPy library (Gürsoy et al., 2014) with 10 iterations, and the transform based method uses the constrained Fourier spaced (CFS) method (Karp et al., 1988) with a bow-tie filter and 100 iterations.
In the sinogram domain, ground truth samples were compared to repaired sinograms for each of the repair methods across all 7,164 slices in the test data set with one, two and four missing blocks. Since all repair methods fill in the missing information in the original sinogram, the evaluated region of interest was limited to the repaired area to avoid diluting the performance measurement. Additionally we applied a 3x3 Gaussian kernel [0.5 1 0.5] x [0.5 1 0.5] to both sinograms before comparison to filter out high frequency noise thereby providing a more realistic comparison. The normalized mean squared error (NMSE), which is an absolute measure, was calculated between the sinograms. That is,
In the reconstructed image domain MS-SSIM, which uses a perception-based model, was used to compare the similarity between reference images and images reconstructed from sinograms with missing data and sinograms corrected with each repair method. In our experiment, all 876 images from the two test patient data sets were reconstructed from sinograms containing one, two, and four missing blocks in each slice and compared with images from ground truth sinograms. All reconstructions were performed using the standard Biograph mCT scanner’s reconstruction software utilizing the Ordered Subset Expectation Maximization (OSEM) protocol with 3 iterations, 21 subsets and a 5x5 Gaussian filter.
In the quantitative accuracy domain, three regions of interest were selected in the test data corresponding to the liver, uptake near the kidney and a small lesion. The mean tracer concentrations were measured across images based on ground truth, repaired and not repaired sinograms.
3.1 Sinogram Comparison
Figure 3 provides a sample of repaired sinogram slices for each method along with the ground truth and the missing data. A visual comparison across all methods of repairing missing sinogram data show the established methods performing in accordance with previously published results, but the Sinogram Repair Network results most closely resemble the ground truth.
Figure 4 shows the normalized mean squared error from equation 7 grouped into sinogram ranges 25,000 counts wide measured for each repair method averaged across ten random realizations of missing test set data for one, two and four missing blocks. Additionally the plots include bars to indicate the single slice maximum and minimum error for each method. Each of the repair methods has the largest error at low count densities. On inspection of these sinograms, the reason for the lower performance is the low signal to noise ratio in these regions, which tend to originate from either end of the whole-body scan where sensitivity is lower and in the case of the upper legs, the tracer distribution is more sparse. With this in mind, extra care must be taken when relying on sinogram repair in regions of high noise. As the count density increases however, all repair methods perform better and experience decreasing error and a narrowing of the difference between maximum and minimum error with the Sinogram Repair Network improving faster than the other methods with a widening performance gap especially as the amount of missing data increases.
3.2 Reconstructed Image Comparison
Table 1 and Fig. 5 compare sinogram repair method performance in image space against ground truth using MS-SSIM for one, two and four missing blocks. The averaged performance across all test slices presented in Table 1 shows that the previously published methods along with the Sinogram Repair Network all perform relatively well with only one missing block detector. However, as the amount of missing data increases, the performance gap between the previously established methods and the proposed neural network grew, which highlights the ability of this new method to handle higher quantities of failed block detectors. When the individual slices of a single patient from the test set are averaged over 10 realizations of missing block detectors, as shown in Fig. 5, the Sinogram repair network outperforms the other methods both across the range of whole-body images and quantities of missing data.
Another observation of note is the lower performance of all repair methods on the start and end image planes, which we found to be consistent across data sets. These regions of a PET acquisition suffer from inherently low sensitivity and high noise due to a scanner’s geometric constraints and thus produce both sparsely populated sinograms with fewer counts (oblique planes) and sinograms high noise components (direct planes). A visual inspection of repaired sinograms from this region confirms the relatively poor performance. Further research could be conducted to better understand the repair of sparse and noisy sinograms and develop more consistent techniques in these areas, but is beyond the scope of this study.
|1 Missing Block||0.923||0.970||0.964||0.950||0.984|
|2 Missing Blocks||0.871||0.940||0.932||0.914||0.963|
|4 Missing Blocks||0.794||0.899||0.902||0.853||0.947|
Coronal slices of reconstructed image volumes with four missing blocks from the test data set are presented in Figure 6 along with their difference image from ground truth. Visual inspection shows the Sinogram Repair Network reconstructed image is most similar to ground truth and has the least pronounced difference image. The images based on previous repair methods all show visual evidence of streaking as noted by the red arrow. They also contain areas of artificially low activity next to high activity as noted by the blue arrows. Both of these artifacts are confirmed by inspecting the difference image, but in contrast the Sinogram Repair Network image lacks these undesirable artifacts. The results from one and two missing blocks are not shown since differences in a single missing block are hard to distinguish visually and while images with two missing blocks do contain similar artifacts to the images shown, the visual differences are less pronounced.
3.3 Quantitative Measurements
The use of quantitative measurements of tracer concentration in PET such as SUV (Kinahan and Fletcher, 2010) taken directly from the reconstructed image are used in treatment planning, clinical research, drug trials, and assessing response to therapy over multiple studies. It is therefore important when repairing sinograms to accurately represent the tracer concentration to avoid adverse affects on the analysis of patient images.
Figure 7(a) shows the regions of interest (ROIs) examined in this study. The yellow ROI of the kidney measures 5,495 mm. The red ROI of a small lesion measures 929 mm while the green ROI measures 2,892 mm and is of the liver, which is commonly used as a reference point in quantitative PET measurements (Kinahan and Fletcher, 2010). Fig. 7(b) gives a quantitative comparison of the repair methods across ROIs for 1, 2, and 4 missing blocks. Measurements from the original image reconstructed from the ground truth sinograms is shown in black and is considered the ideal for this comparison. As expected, the repair methods generally exhibit lower tracer concentration measurements than the original and this gap tends to widen as the number of simulated failed blocks increases. The Sinogram Repair Network maintains more consistent measurements across varying amounts of missing data and generally provides measurements closer to the ideal than other methods. Although the Sinogram Repair Network performs well, this study only examined quantitative measurements from two patients due to the limited data set and further work is necessary to draw strong and confident conclusions about quantitative reliability.
One application of the Sinogram Repair Network could be to continue clinical operations with full diagnostic quality in the event one or more block detectors fail the daily quality control procedure. This mitigation may become even more important as the number of block detectors and axial field of view of PET scanners continues to increase. However, not all detector failures have equal impact on the diagnostic quality of patient images; the number of bad block detectors and their relative location make a difference. The clinical relevance of bad blocks was first studied in 1999 (Buchert et al., 1999), and many of their suggestions are still prudent even with newer and more effective repair techniques. A key suggestion is to validate the diagnostic quality of the scanner by creating a mask of the failed detectors and then executing an extended quality control procedure by simulating the missing data on previously stored reference data sets representative of the clinical environment. These data sets are then repaired, reconstructed and scored using a structural similarity metric. The results of this metric along with a visual inspection and quantitative measurement comparison to the ground truth should determine if the diagnostic quality of the scanner is compromised and whether imaging should confidently continue or cease until a repair is made. For the most robust results the reference data sets should contain at least some small lesions at various distances from the center of the transaxial field of view.
The proposed Sinogram Repair Network is an effective method of repairing missing data in PET whole-body sinograms with one or more malfunctioning block detectors. A wide range of aspects were examined including sinogram and image space analysis and quantitative measurement accuracy and when compared with the existing repair methods, the Sinogram Repair Network was found to perform better in all cases.
Even with the superior performance of the sinogram repair method proposed in this paper, there are a number of avenues of further research that could provide improvements over the current performance level. One direction is to split the data set into smaller distributions based on some classifying feature such as the total counts in a sinogram. Then multiple networks could each be trained on a subset of the broader distribution and would be required to generalize to a much smaller portion of the global distribution of sinograms. Another promising direction is to consider 3D sinogram volumes when filling in missing data with the expectation that a neural network could learn the relationships between the various sinogram segments and utilize information from segments without missing data to fill gaps in affected sinograms. An expansion of this idea could also include the time-of-flight components present in most modern PET scanners. Other avenues for further exploration include using alternative neural network structures such as the Wavelet Residual Network(Kang et al., 2018) or the utilization of other loss functions such as those found in Generative Adversarial Networks (Goodfellow et al., 2014).
Additionally, there are a number of other potential applications where the techniques in this research could be applied. One natural application for the proposed method is to fix normalization errors. We briefly experimented with failed detector blocks during continuous bed motion acquisitions, which causes artifacts similar to normalization errors, and found the sinogram repair method to work well. Perhaps the most impactful extension to our work is designing a PET/CT scanner with intentional gaps in the detector array (both transaxial and axial) and using techniques similar to those presented here to fill in the missing data creating a scanner with performance characteristics similar to a system without gaps in the detector array, but at a significantly lower cost, longer axial field of view and higher acquired data efficiency. This approach would enable all downstream image formation components to remain unchanged in contrast to other intentional gap techniques that require special reconstruction methods to effectively mitigate the gaps.
- Lose the views: limited angle CT reconstruction via implicit sinogram completion. , pp. 6343–6352. Cited by: §1.
- Compensation of missing projection data for MDAPET camera. In 2000 IEEE Nuclear Science Symposium. Conference Record (Cat. No.00CH37149), Vol. 3, pp. 17/41–17/45 vol.3. External Links: Cited by: §1.
- An evaluation of missing data compensation methods for a PET camera by comparing to no-gap data. In 2008 IEEE Nuclear Science Symposium Conference Record, Vol. , pp. 5273–5276. External Links: Cited by: §1.
- Quality assurance in PET: evaluation of the clinical relevance of detector defects. Journal of Nuclear Medicine 40, pp. 1657–65. Cited by: §1, §4.
- Total-Body PET: maximizing sensitivity to create new opportunities for clinical research and patient care. JNM 59 (1), pp. 3–12. External Links: Cited by: §1.
- Correction methods for missing data in sinograms of the HRRT PET scanner. IEEE Transactions on Nuclear Science 50 (5), pp. 1452–1456. External Links: Cited by: §1.
- On the significance of defective block detectors in clinical 18F-FDG PET/CT imaging. Molecular Imaging and Biology 13, pp. 265–274. Cited by: §1.
- Generative adversarial nets. In Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (Eds.), pp. 2672–2680. External Links: Cited by: §1, §5.
- TomoPy: a framework for the analysis of synchrotron tomographic data. Journal of Synchrotron Radiation 21 (5), pp. 1188–1193. External Links: Cited by: §2.4.
- WaveNet: a deep convolutional neural network using directional wavelets for low-dose X-ray CT reconstruction. Medical Physics 44, pp. . External Links: Cited by: §1.
- Deep convolutional framelet denosing for low-dose CT via wavelet residual network. IEEE Transactions on Medical Imaging 37 (6), pp. 1358–1369. External Links: Cited by: §5.
- Constrained Fourier space method for compensation of missing data in emission computed tomography. IEEE Transactions on Medical Imaging 7 (1), pp. 21–25. External Links: Cited by: §1, §2.4.
- Statistical image reconstruction in PET with compensation for missing data. IEEE Transactions on Nuclear Science 44 (4), pp. 1552–1557. External Links: Cited by: §1.
- Positron emission tomography-computed tomography standardized uptake values in clinical practice and assessing response to therapy. In Seminars in ultrasound, CT and MR, Vol. 31,6, pp. 496–505. Cited by: §3.3, §3.3.
- Adam: A method for stochastic optimization. International Conference on Learning Representations ICLR(’14). Cited by: §2.3.
- A theoretical framework for back-propagation. In Proceedings of the 1988 connectionist models summer school, Vol. 1, pp. 21–28. Cited by: §1.
- Impact of detector defects on image quality and quantification for the micropet focus 220 scanner. In 2006 IEEE Nuclear Science Symposium Conference Record, Vol. 5, pp. 3032–3036. External Links: Cited by: §1.
- A sinogram inpainting method based on generative adversarial network for limited-angle computed tomography. In 15th International Meeting on Fully Three-Dimensional Image Reconstruction in Radiology and Nuclear Medicine, 1107220 (28 May 2019), Vol. 11072. External Links: Cited by: §1.
- SGDR: stochastic gradient descent with restarts. In International Conference on Learning Representations ICLR(’07), Cited by: §2.3.
- Automatic differentiation in pytorch. In NIPS-W, Cited by: §2.3.
- 1-D interpolation method for the HRRT PET sinogram gap-filling. In 2013 IEEE Nuclear Science Symposium and Medical Imaging Conference (2013 NSS/MIC), Vol. , pp. 1–4. External Links: Cited by: §1, §2.4.
- U-net: convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI), LNCS, Vol. 9351, pp. 234–241. Note: (available on arXiv:1505.04597 [cs.CV]) External Links: Cited by: §2.1.
- Data estimation for the ECAT HRRT sinograms by utilizing the DCT domain. In 2008 IEEE Nuclear Science Symposium Conference Record, Vol. , pp. 5076–5080. External Links: Cited by: §1.
- Gap filling strategies for 3D-FBP reconstructions of high-resolution research tomograph scans. IEEE Transactions on Medical Imaging 27 (7), pp. 934–942. External Links: Cited by: §1.
- The effect of defective pet detectors in clinical simultaneous 18F-FDG time-of-flight PET/MR imaging. Molecular Imaging and Biology 19, pp. 626–635. Cited by: §1.
Image reconstruction is a new frontier of machine learning. IEEE Transactions on Medical Imaging 37 (6), pp. 1289–1296. External Links: Cited by: §1.
- Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems Computers, 2003, Vol. 2, pp. 1398–1402 Vol.2. External Links: Cited by: §2.3.
- Image denoising and inpainting with deep neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, NIPS’12, USA, pp. 341–349. External Links: Cited by: §1.
- Artifact removal using improved googlenet for sparse-view CT reconstruction. In Scientific Reports, Cited by: §1.
High-resolution image inpainting using multi-scale neural patch synthesis. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , pp. 4076–4084. External Links: Cited by: §1.
- Semantic image inpainting with deep generative models. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6882–6890. Cited by: §1.
- Generative image inpainting with contextual attention. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5505–5514. Cited by: §1.
- Convolutional neural network based metal artifact reduction in X-ray computed tomography. IEEE Transactions on Medical Imaging 37 (6), pp. 1370–1381. External Links: Cited by: §1.
- Loss functions for image restoration with neural networks. IEEE Transactions on Computational Imaging 3 (1), pp. 47–57. External Links: Cited by: §2.3.
- Unsupervised Learnable Sinogram Inpainting Network (SIN) for Limited Angle CT reconstruction. arXiv e-prints, pp. arXiv:1811.03911. External Links: Cited by: §1.
- Analysis of different detector and electronics defects on F18-FDG images. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 571 (1), pp. 493 – 497. Note: Proceedings of the 1st International Conference on Molecular Imaging Technology External Links: Cited by: §1.