Joint Reconstruction and Calibration using Regularization by Denoising

by   Mingyang Xie, et al.

Regularization by denoising (RED) is a broadly applicable framework for solving inverse problems by using priors specified as denoisers. While RED has been shown to provide state-of-the-art performance in a number of applications, existing RED algorithms require exact knowledge of the measurement operator characterizing the imaging system, limiting their applicability in problems where the measurement operator has parametric uncertainties. We propose a new method, called Calibrated RED (Cal-RED), that enables joint calibration of the measurement operator along with reconstruction of the unknown image. Cal-RED extends the traditional RED methodology to imaging problems that require the calibration of the measurement operator. We validate Cal-RED on the problem of image reconstruction in computerized tomography (CT) under perturbed projection angles. Our results corroborate the effectiveness of Cal-RED for joint calibration and reconstruction using pre-trained deep denoisers as image priors.


Async-RED: A Provably Convergent Asynchronous Block Parallel Stochastic Method using Deep Denoising Priors

Regularization by denoising (RED) is a recently developed framework for ...

Infusing Learned Priors into Model-Based Multispectral Imaging

We introduce a new algorithm for regularized reconstruction of multispec...

Online Regularization by Denoising with Applications to Phase Retrieval

Regularization by denoising (RED) is a powerful framework for solving im...

Unsupervised PET Reconstruction from a Bayesian Perspective

Positron emission tomography (PET) reconstruction has become an ill-pose...

Block Coordinate Regularization by Denoising

We consider the problem of estimating a vector from its noisy measuremen...

Automatic calibration of time of flight based non-line-of-sight reconstruction

Time of flight based Non-line-of-sight (NLOS) imaging approaches require...

Towards Generating Benchmark Datasets for Worm Infection Studies

Worm origin identification and propagation path reconstruction are among...

1 Introduction

The reconstruction of an unknown image from a set of noisy measurements is crucial in imaging inverse problems such as tomography [1, 2], phase retrieval [3, 4], and compressive sensing [5, 6]. When solving such ill-posed inverse problems, a widely used approach is to regularize the solution by using an image prior, such as the Tikhonov regularization [7] or total variation regularization (TV) [8].

Traditional image reconstruction methods assume accurate knowledge of the measurement operator characterizing the imaging system. In some cases, however, this operator is not known to sufficient accuracy, and can be modelled as depending on some parameters that have to be calibrated to obtain an accurate characterization of the imaging system. For example, calibration is essential in computerized tomography (CT) [9] when there is uncertainty in the projection angles used in data collection. In such settings, it is common to either manually calibrate the imaging system by using a known phantom [10, 11, 12, 13] or embed a calibration step into the reconstruction method [14, 15, 16].

There has been considerable recent interest in plug-and-play priors (PnP) [17] and regularization by denoising (RED) [18], as frameworks for exploiting image denoisers as priors for image reconstruction. These methods achieve state-of-the-art performance in a variety of imaging applications by integrating advanced denoisers that are not necessarily expressible in the form of a regularization function [19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]. However, current PnP/RED algorithms assume exact knowledge of the measurement operator, limiting their ability to solve problems involving parametric uncertainties in the data acquisition.

We propose a new calibrated RED (Cal-RED) method that integrates an automatic parameter calibration procedure within the RED reconstruction. Each iteration of Cal-RED performs a -update step, followed by a traditional -update step, leading to the joint recovery of the unknown image and parameters . One of the key benefits of Cal-RED is that it can leverage powerful deep denoisers, such as DnCNN [30], for regularizing the inverse problem. We validate Cal-RED on CT by showing its ability to: (a) reduce the root mean squared error (RMSE) of the projection angles from to ; and (b) improve the imaging signal to noise ratio (SNR) from 10.16 dB (uncalibrated) to 21.13 dB (calibrated), with only a 0.95 dB gap relative to the oracle RED that knows the true angles. Note that while our focus in this letter is on RED, our calibration strategy is also fully compatible with PnP.

2 Background

2.1 Inverse Problems in Imaging

Consider the imaging problem specified by the linear model


where denotes the uknown image, denotes the measurements, is the measurement operator characterizing the response of the imaging system, denotes the unknown parameters of the measurement operator, and is the noise. When the true value of the parameters is known, image reconstruction can be formulated as an optimization problem


where is the data-fidelity term that uses the measurement operator to ensure the consistency with the measurements and is the regularizer that imposes prior knowledge onto . For example, consider the smooth -norm data-fidelity term


and the nonsmooth TV regularizer , where is the regularization parameter and is the image gradient [8]. Alternating direction method of multipliers (ADMM) [31] and fast iterative shrinkage/thresholding algorithm (FISTA) [32] are two common optimization algorithms that address the nonsmoothness of in image reconstruction via the proximal operator


which can be interpreted as a

maximum a prior probability (MAP)

estimator for AWGN denoising.

Deep learning has recently gained popularity for solving imaging inverse problems. An extensive review of deep learning in this context can be found in [33, 34, 35]. Instead of explicitly defining an optimization problem, the traditional deep-learning approach is based on training an existing network architecture, such as UNet [36], to invert the measurement operator by exploiting the natural redundancies in the imaging data [37, 38]

. It is common to first bring the measurements to the image domain and train the network to map the corresponding low-quality images to their clean target versions via supervised learning 

[37, 38].

2.2 Regularization by Denoising

RED is a recently introduced image recovery framework [18, 29] that seeks an images that satisfy


where denotes the gradient of the data-fidelity term using the true calibration parameters , is the regularization parameter, is a denoiser for the input noise level . RED thus computes an equilibrium point that balances the data fidelity against the fixed points of the denoiser. When the denoiser is locally homogeneous and has a symmetric Jacobian [29], the term corresponds to the gradient of the regularization function


RED has been extensively applied in computational imaging, including in image restoration [28], phase retrieval [27], and tomographic imaging [1]. Later works have developed scalable variants of RED [26, 39] and further replace the AWGN denoising with a general artifact-removal operator [40].

2.3 Calibrating the Measurement Operator

The target-free calibration of the measurement operator has been an emerging topic in recent years. For CT scanners with slightly misaligned projection angles, existing calibration methods include entropy-based correction [41], frequency-based correction [42], and a technique that combined gradient descent and multi-range testing [14]. A popular method to calibrate projection angles of Cryo-EM is called projection-matching [43, 44], which relies on a prior estimate of the density map of the unknown single particle. It seeks the angle of every projection by pairing it with a clean template of the projection of the estimated density map from a known angle. Malhotraz et al. [16]

propose a three-step pipeline for Cryo-EM to recover the projection angles and the target density map in sequence: (a) denoise the projections by the Principal Component Analysis technique, (b) estimate the angles through iterative coordinate gradient descent, and (c) reconstruct the density map by

filtered back-projection (FBP). The key contribution of this paper is the investigation of the calibration problem in the context of PnP/RED algorithms, which has not been done in the prior work.

3 Proposed Method

The proposed Cal-RED algorithm is summarized in Algorithm 1. It alternates between the updates of and . We initialize with their corresponding nominal values, , which are assumed to be known from the design of the imaging system. We additionally initialize the image with a direct inversion from measurements , which for CT reduces to the traditional FBP [9].

For the -optimization, as shown in lines 5-7 in Algorithm 1, we minimize the objective function


by performing the gradient descent


In Equation 7, we consider the Tikhonov penalty as the prior for . The partial derivative is calculated by automatic differentiation [45].

We update by following the RED framework, as shown in lines 9-11 of Algorithm 1. For the denoising prior , we choose a 17-layer DnCNN architecture in which all the convolution filters are of size and every feature map has 64 channels. Note that Cal-RED does not modify the update step of RED, which implies that the -update step of Cal-RED could also be integrated within PnP or other variants of RED, such as BC-RED [39].

Cal-RED relies on Nesterov acceleration [46] when computing both and updates by adding the acceleration sequence


When for all , the algorithms reverts to the usual gradient method without acceleration. In Algorithm 1, the variables and store the intermediate values used for acceleration of and , respectively.

3:  for  do
4:     # -update step
8:     # -update step
12:  end for
Algorithm 1 Cal-RED
Figure 1: Illustration of the convergence of Cal-RED with projection angles corrupted by AWGN of SD 1°and 5°. The top figure corresponds to a sinogram input SNR of 40 dB, while the bottom figure corresponds to that of 30 dB. In each figure, the values of SNR and RMSE are plotted against the number of iterations using solid and dotted curves, respectively. The left axis represents the scale of SNR while the right axis represents that of RMSE. The optimal SNR values (obtained by RED using the true angles) is plotted as the horizontal dashed line for reference. Note the significant reduction of angular errors and the nearly optimal SNR results.
Figure 2: Recovery of a knee image from a sinogram with noise corresponding to input SNR of 40 dB and projection angles corrupted by AWGN of SD 1°and 5°. Both the ground truth and the image reconstructed by oracle RED given the true angles are placed at the leftmost column for reference. The details are highlighted in the yellow and red boxes attached below each image.

4 Experiments

In this section, we validate the performance of Cal-RED on simulated parallel-beam CT reconstruction, where the 2D target is rotated around an axis to acquire projections from different angles and form a sinogram. We assume that the CT machine is designed to project from nominal angles , an array of 90 projection angles that are evenly-distributed on a half circle. However, due to alignment errors, the CT machine actually projects from , another array of 90 angles that is assumed to be the result of corrupting

by an unknown AWGN vector with standard deviation (SD) of 1° and 5°, respectively. In our experiments, we synthesize the sinograms by using the ground-truth

and further corrupt these sinograms by AWGN corresponding to the input SNR of 40 dB and 30 dB, respectively.

We quantify the image quality by using SNR (dB)


and the angular error by RMSE


All numerical results are averaged over six medical knee images discretized to the size of pixels. Note that the measurement operator in this case corresponds to the Radon transform.

We compare Cal-RED with several baseline methods, including RED, U-Net, FISTA and the traditional least-square method (LSM). For both Cal-RED and RED, we train DnCNN denoisers for AWGN removal at three noise levels and pick the one that achieves the best performance for evaluation. We generate the training data for U-Net by filtered-back-projecting the sinogram using the mismatched angles . We train an individual U-Net for each combination of . In order to better demonstrate the benefit of using deep denoising prior, we run FISTA with the TV regularizer for comparison. We also incorporate the -update step into both FISTA and LSM to construct the angle-calibrating FISTA (Cal-FISTA) and the angle-calibrating LSM (Cal-LSM), respectively.

Figure 1 illustrates the convergence of Cal-RED in terms of both image SNR and angular RMSE. In the cases where is severely mismatched, Cal-RED is able to reduce the angular RMSE by a factor of more than 7 — from 5°to around 0.65°; moreover, Cal-RED can achieve a good reconstruction comparable to that of RED using the true with respect to SNR. Table 1 and Table 2 summarize the numerical results under different combinations of input sinogram SNR and angular error. It is worth noting that Cal-RED and Cal-FISTA significantly outperform RED and FISTA, respectively, which highlights the effectiveness of the -update step. Despite being designed for the angle calibration purpose, when Cal-RED is applied to a problem without calibration errors, it achieves an SNR only 0.03 dB less than that of the original RED. When the input SNR is 30 dB, the angles recovered by Cal-RED and Cal-FISTA are far more accurate than those recovered by Cal-LSM, which demonstrates how denoising priors can also contribute to measurement operator calibration.

Figure 2 compares the visual results of FBP, U-Net, Cal-FISTA, Cal-RED, and RED. Both the reconstructions of FBP and uncalibrated RED suffer from obvious line artifacts, which demonstrates their vulnerability to a mismatched measurement operator. Although Cal-FISTA achieves an SNR close to that of Cal-RED, its reconstruction contains visible blocking artifacts. The U-Net removes most of the artifacts that are contained in the FBP reconstruction, but it also has an undesirable effect of oversmoothing small features. While the traditional deep learning alone fails to balance the preservation of features and the removal of artifacts, deep denoising priors like DnCNN help Cal-RED to reconstruct high-resolution images with minimal artifacts. For this particular image, when the projection angles are corrupted by AWGN of SD 5°, Cal-RED is able to achieve a remarkable SNR of 21.40 dB, while uncalibrated RED only achieves SNR of 10.01 dB.

Input SNR 30 dB 40 dB
Angular Error
FBP 14.17 12.69 9.09 19.42 15.68 10.14
U-Net 19.84 18.01 14.49 22.01 18.58 14.55
FISTA 19.40 16.92 11.55 21.20 17.19 11.56
RED 19.99 15.46 10.06 22.08 15.84 10.16
Cal-LSM 7.80 5.07 4.18 16.73 14.88 13.34
Cal-FISTA 19.35 19.14 18.69 21.17 20.84 20.29
Cal-RED 19.96 19.74 19.52 22.07 21.78 21.13
Table 1: Average SNRs of Reconstructed CT Images (dB)
Input SNR 30 dB 40 dB
Angular Error
Cal-RED 0.073 0.650 0.024 0.651
Cal-FISTA 0.078 0.649 0.029 0.648
Cal-LSM 0.281 0.797 0.026 0.649
Table 2: Average RMSE of Calibrated CT Angles (°)

5 Conclusion

In this letter, we propose Cal-RED, a method that takes advantage of advanced denoising priors to jointly recover the measurement operator parameters and the target image. We validate its robustness to severely mismatched measurement operator parameters and noisy measurements via simulated CT reconstruction. Future work includes extending the proposed calibration strategy to other applications and image reconstruction frameworks.


Research presented in this article was supported by the Laboratory Directed Research and Development program of Los Alamos National Laboratory under project number 20200061DR.


  • [1] Z. Wu, Y. Sun, A. Matlock, J. Liu, L. Tian, and U. S. Kamilov, “SIMBA: Scalable inversion in optical tomography using deep denoising priors,” IEEE Journal of Selected Topics in Signal Process., vol. 14, no. 6, pp. 1163–1175, Mar. 2020.
  • [2]

    S. Sreehari, S. V. Venkatakrishnan, B. Wohlberg, G. T. Buzzard, L. F. Drummy, J. P. Simmons, and C. A. Bouman, “Plug-and-play priors for bright field electron tomography and sparse interpolation,”

    IEEE Trans. Comput. Imaging, vol. 2, no. 4, pp. 408–423, Dec. 2016.
  • [3] Z. Wu, Y. Sun, J. Liu, and U. Kamilov, “Online regularization by denoising with applications to phase retrieval,” in

    IEEE Int. Conf. Computer Vision (ICCV) Workshops

    , Oct. 2019.
  • [4] E. J. Candès, Y. C. Eldar, T. Strohmer, and V. Voroninski, “Phase retrieval via matrix completion,” SIAM J. Imaging Sci., vol. 6, no. 1, Feb. 2013.
  • [5] H.-Y. Liu, U. S. Kamilov, D. Liu, H. Mansour, and P. T. Boufounos, “Compressive imaging with iterative forward models,” in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Process. (ICASSP), New Orleans, LA, USA, Mar. 2017.
  • [6] C. A. Metzler, A. Maleki, and R. G. Baraniuk, “From denoising to compressed sensing,” IEEE Trans. Inf. Theory, vol. 62, no. 9, pp. 5117–5144, Sep. 2016.
  • [7] A. Ribés and F. Schmitt, “Linear inverse problems in imaging,” IEEE Signal Process. Mag., vol. 25, no. 4, pp. 84–99, Jul. 2008.
  • [8] L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Physica D, vol. 60, no. 1–4, pp. 259–268, Nov. 1992.
  • [9] A. C. Kak and M. Slaney, Principles of Computerized Tomographic Imaging.   IEEE, 1988.
  • [10] C. E. Cann, “Quantitative CT for determination of bone mineral density: a review.” Radiology, vol. 166, no. 2, pp. 509–522, 1988.
  • [11] X. Wang, J. G. Mainprize, M. P. Kempston, G. E. Mawdsley, and M. J. Yaffe, “Digital breast tomosynthesis geometry calibration,” in Medical Imaging 2007: Physics of Medical Imaging, vol. 6510, 2007.
  • [12] H. Miao, X. Wu, H. Zhao, and H. Liu, “A phantom-based calibration method for digital X-ray tomosynthesis,” Journal of X-ray Science and Technology, vol. 20, pp. 17–29, Jan. 2012.
  • [13] J. Zou, X. Hu, H. Lv, and X. Hu, “An investigation of calibration phantoms for CT scanners with tube voltage modulation,” Int. Journal of Biomedical Imaging, vol. 2013, Dec. 2013.
  • [14] C. Cheng, Y. Ching, P. Ko, and Y. Hwu, “Correction of center of rotation and projection angle in synchrotron X-ray computed tomography,” Scientific Reports, vol. 8, Dec. 2018.
  • [15] D. Lee, P. Hoffmann, D. Kopperdhal, and T. Keaveny, “Phantomless calibration of CT scans for measurement of bmd and bone strength-inter-operator reanalysis precision,” Bone, vol. 103, Aug. 2017.
  • [16]

    E. Malhotra and A. Rajwade, “Tomographic reconstruction from projections with unknown view angles exploiting moment-based relationships,” in

    2016 IEEE Int. Conf. Image Process. (ICIP), 2016.
  • [17] S. V. Venkatakrishnan, C. A. Bouman, and B. Wohlberg, “Plug-and-play priors for model based reconstruction,” in Proc. IEEE Global Conf. Signal and Inf. Process. (GlobalSIP), 2013.
  • [18] Y. Romano, M. Elad, and P. Milanfar, “The little engine that could: Regularization by denoising (RED),” SIAM J. Imaging Sci., vol. 10, no. 4, pp. 1804–1844, 2017.
  • [19] S. H. Chan, X. Wang, and O. A. Elgendy, “Plug-and-play ADMM for image restoration: Fixed-point convergence and applications,” IEEE Trans. Comput. Imaging, vol. 3, no. 1, pp. 84–98, Mar. 2017.
  • [20] A. M. Teodoro, J. M. Bioucas-Dias, and M. Figueiredo, “A convergent image fusion algorithm using scene-adapted Gaussian-mixture-based denoising,” IEEE Trans. Image Process., vol. 28, no. 1, pp. 451–463, Jan. 2019.
  • [21] A. Brifman, Y. Romano, and M. Elad, “Turning a denoiser into a super-resolver using plug and play priors,” in Proc. IEEE Int. Conf. Image Proc. (ICIP), Phoenix, AZ, USA, Sep. 2016.
  • [22] A. M. Teodoro, J. M. Biocas-Dias, and M. A. T. Figueiredo, “Image restoration and reconstruction using variable splitting and class-adapted image priors,” in Proc. IEEE Int. Conf. Image Proc. (ICIP), Phoenix, AZ, USA, Sep. 2016.
  • [23] K. Zhang, W. Zuo, S. Gu, and L. Zhang, “Learning deep CNN denoiser prior for image restoration,” in

    Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)

    , 2017.
  • [24] T. Meinhardt, M. Moeller, C. Hazirbas, and D. Cremers, “Learning proximal operators: Using denoising networks for regularizing inverse imaging problems,” in Proc. IEEE Int. Conf. Computer Vision (ICCV), Venice, Italy, Oct. 2017.
  • [25] U. S. Kamilov, H. Mansour, and B. Wohlberg, “A plug-and-play priors approach for solving nonlinear imaging inverse problems,” IEEE Signal Process. Lett., vol. 24, no. 12, pp. 1872–1876, Dec. 2017.
  • [26] Y. Sun, B. Wohlberg, and U. S. Kamilov, “An online plug-and-play algorithm for regularized image reconstruction,” IEEE Trans. Comput. Imaging, vol. 5, no. 3, pp. 395–408, 2019.
  • [27] C. A. Metzler, P. Schniter, A. Veeraraghavan, and R. G. Baraniuk, “prDeep: Robust phase retrieval with a flexible deep network,” in

    Proc. 35th Int. Conf. Machine Learning (ICML)

    , 2018.
  • [28] G. Mataev, M. Elad, and P. Milanfar, “DeepRED: Deep image prior powered by RED,” in Proc. IEEE Int. Conf. Computer Vision Workshops (ICCVW), Seoul, South Korea, Oct. 2019.
  • [29] E. T. Reehorst and P. Schniter, “Regularization by denoising: Clarifications and new interpretations,” IEEE Trans. Comput. Imaging, vol. 5, no. 1, pp. 52–67, Mar. 2019.
  • [30] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising,” IEEE Trans. Image Process., vol. 26, no. 7, pp. 3142–3155, Feb. 2017.
  • [31] M. V. Afonso, J. M.Bioucas-Dias, and M. A. T. Figueiredo, “Fast image recovery using variable splitting and constrained optimization,” IEEE Trans. Image Process., vol. 19, no. 9, pp. 2345–2356, Sep. 2010.
  • [32] A. Beck and M. Teboulle, “Fast gradient-based algorithm for constrained total variation image denoising and deblurring problems,” IEEE Trans. Image Process., vol. 18, no. 11, pp. 2419–2434, November 2009.
  • [33]

    M. T. McCann, K. H. Jin, and M. Unser, “Convolutional neural networks for inverse problems in imaging: A review,”

    IEEE Signal Process. Mag., vol. 34, no. 6, pp. 85–95, 2017.
  • [34]

    A. Lucas, M. Iliadis, R. Molina, and A. K. Katsaggelos, “Using deep neural networks for inverse problems in imaging: Beyond analytical methods,”

    IEEE Signal Process. Mag., vol. 35, no. 1, pp. 20–36, Jan. 2018.
  • [35] F. Knoll, K. Hammernik, C. Zhang, S. Moeller, T. Pock, D. K. Sodickson, and M. Akcakaya, “Deep-learning methods for parallel magnetic resonance imaging reconstruction: A survey of the current approaches, trends, and issues,” IEEE Signal Process. Mag., vol. 37, no. 1, pp. 128–140, Jan. 2020.
  • [36] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015.
  • [37] Y. Han, J. Kim, and J. C. Ye, “Differentiated backprojection domain deep learning for conebeam artifact removal,” IEEE Trans. Medical Imaging, 2020.
  • [38] Y. Sun, Z. Xia, and U. S. Kamilov, “Efficient and accurate inversion of multiple scattering with deep learning,” Opt. Express, vol. 26, no. 11, pp. 14 678–14 688, May 2018.
  • [39] Y. Sun, J. Liu, and U. S. Kamilov, “Block coordinate regularization by denoising,” in Advances in Neural Inf. Process. Systems 33, Vancouver, BC, Canada, Dec. 8-14, 2019.
  • [40] J. Liu, Y. Sun, C. Eldeniz, W. Gan, H. An, and U. S. Kamilov, “RARE: Image reconstruction using deep priors learned without ground truth,” IEEE J. Select. Topics Signal Process., 2020.
  • [41] T. Donath, F. Beckmann, and A. Schreyer, “Automated determination of the center of rotation in tomography data,” Journal of the Optical Society of America. A, Optics, Image science, and Vision, vol. 23, pp. 1048–57, Jun. 2006.
  • [42] N. T. Vo, M. Drakopoulos, R. Atwood, and C. Reinhard, “Reliable method for calculating the center of rotation in parallel-beam tomography,” Optics Express, vol. 22, pp. 19 078–19 086, Jul. 2014.
  • [43] P. A. Penczek, R. A. Grassucci, and J. Frank, “The ribosome at improved resolution: New techniques for merging and orientation refinement in 3D cryo-electron microscopy of biological particles,” Ultramicroscopy, vol. 53, no. 3, pp. 251 – 270, 1994.
  • [44] T. S. Baker and R. H. Cheng, “A model-based approach for determining orientations of biological macromolecules imaged by cryoelectron microscopy,” Journal of Structural Biology, vol. 116, no. 1, pp. 120 – 130, 1996.
  • [45] A. Baydin, B. Pearlmutter, A. Radul, and J. Siskind, “Automatic differentiation in machine learning: A survey,” Journal of Machine Learning Research, vol. 18, pp. 1–43, Apr. 2018.
  • [46] Y. Nesterov, Introductory Lectures on Convex Optimization: A Basic Course.   Kluwer Academic Publishers, 2004.