share
I Introduction
Compressive sensing (CS), an emerging sampling and reconstructing strategy, can recover original signal from dramatically fewer measurements with a sub-Nyquist sampling rate [CS]. As CS has the potentials of significantly improving the sampling speed and sensor energy efficiency, it has been applied in many practical applications, including single pixel imaging [singlecamera], fast magnetic resonance imaging [MRI], high-speed video cameras [Video] and image encryption [cqli:meet:JISA19]. To deal with high-dimensional natural images efficiently, block-based CS is proposed as a lightweight CS approach [BCS, SPL, BCS-1]
. In such strategy, a scene under view is partitioned into some small blocks, which are then sampled and reconstructed independently. Meaningful information is usually not uniformly distributed in an image, so the block partition benefits more fair allocation of the sensing resources for the whole image
[BCS-Salie].Although block-based CS enjoys the advantages of low-cost sampling, lightweight reconstruction, and capability of adaptively assigning sensing resources, it also usually suffers from reduced quality of image reconstruction due to blocking artifacts [BCS-1, BCS-2]. To address the issue, some methods using an iterative block-based CS algorithm (BCS) are proposed [BCS, BCS-1]. In each iteration, the projection operation is used to build an approximation of each block, while denoising operation acts on the full image reassembled by the approximative blocks. Results demonstrate that the recovered image blocks can be improved, while blocking artifacts can also be ameliorated as the iterations progress. This approach, however, may increase the reconstructing time, since small blocks still need to be concatenated into large-size full images to remove blocking artifacts.
Inspired by the powerful learning ability of deep neural networks in image representation [DNN-C, DNN-S], several network-based CS methods are proposed [Icassp, Reconnet, Ldamp, Im-recon, ISTA], which are significantly faster than the traditional CS reconstruction algorithms. Using a fully connected layer to mimic the CS sampling, the network models can jointly optimize the sampling matrix and the reconstruction process, improving the qualities of recovered images. Although the CS network models are carefully constructed to enhance learning capabilities, several specific models have to been trained with various sampling rates, ignoring the mutual relationships among them. Consequently, blocking artifacts still exist in the existing deep network methods [Reconnet, Im-recon, ISTA], especially when the employed sampling rates are very low. Moreover, most network-based image CS methods are trained as a black box, ignoring structural insight of CS reconstruction algorithms. Consequently, the reconstruction accuracy is decreased.
In this paper, we propose a multi-channel deep learning architecture for casting BCS algorithm into a learning network. It can benefit from the speed and learning capacities of deep networks while retaining the advantages of the previous BCS algorithms. To facilitate description, we term the multi-channel deep architecture as BCS-Net, which consists of a channel-specific sampling network and a unified deep reconstruction network. The channel-specific architecture is specifically designed to handle block-based allocation of sensing resources. The blocks with various sampling rates can then be fed into the same deep reconstruction network to exploit inter-block correlation for removing blocking artifacts. We further divide the reconstruction network into a fixed number of residual layers, each of which corresponds to an iteration of the BCS algorithm. To enable training, a modified version of the famous DnCNN network designed in
[DnCNN] is used to replace the denoising operation in traditional BCS approach, which easily propagates gradients.Our contributions of the paper are summarized below:
-
A multi-channel sampling architecture specifically for block-based image CS is designed. Using this multi-channel architecture, block-wise CS measuring processed with a variety of sampling rates can be integrated into a single model to utilize the correlation among the blocks with different CS sampling rates.
-
A deep reconstruction architecture based on the BCS algorithm is proposed using block-wise approximation and full-image-based denoising.
-
Performances of the proposed approaches are verified by extensive experiments on three widely used benchmark datasets. The results show that the proposed multi-channel deep network can significantly outperform the state-of-the-art CS methods and network-based ones in terms of both subjective and objective metrics.
The remainder of this paper is organized as follows. In Sec. II, the related work on CS methods and network-based methods are reviewed. Section III introduces the idea of block-wise approximation and full-image-based denoising in BCS algorithm, and presents an extended version of the well-known Damp algorithm. The proposed multi-channel deep architecture is presented in Sec. IV and test results on its performance are given in Sec. V. The last section concludes the paper.
Ii The related work
In this section, we present the background of CS theory, then review the representative work on block-based image CS and deep network approaches.
Ii-a Preliminary of CS theory
Compressive sensing consists of two main steps: sampling/measuring process and reconstructing process. Let and denote a sparse signal of size and an measurement/sampling matrix, respectively. Then, the sampling process can be presented as
| (1) |
where is the
-length measurement vector sampled from
. If signal is not sparse but compressible, the sampling process has to be further deduced from Eq. (1). That is, , where represents measurement noise, denotes the sparse transform, is the coefficient vector in the corresponding transform domain, , and is considered as a sparse vector approximating to . In general, a natural image is not strictly sparse signal, but often approximately sparse in some spare transform domain.The process of reconstructing signal needs much more computational complexity than the sampling process. It has been proven in [CS] that if the sampling matrix obeys the restricted isometry property (RIP), it is possible to recover by solving an -norm optimization problem: , subjecting to , even if . Here is a small constant, and one has when is a compressible signal.
In the past two decades, a number of CS reconstruction algorithms have been developed, including basis pursuit [BP], orthogonal matching pursuit [OMP], and the latest iteration-based Damp algorithm [Damp]. Although these algorithms enjoy solid mathematical foundations, they usually need long reconstructing time due to high computational complexity.
Ii-B Block-based image CS
Block-based CS is more effective for processing natural images because of increased dimensionality of such signals [BCS, BCS-1, BCS-Small]. The scene under view is partitioned into relatively smaller non-overlapping blocks. The measurement matrixes corresponding to the small-size blocks are observed. Then, the image is sampled and reconstructed on a block-by-block basis [BCS-1]. This block-independent approach results in a reduced computational complexity for reconstruction with a much simplified sampling process.
As shown in Fig. 1, the meaningful information in different blocks of an image is non-uniform. Depending on the volume of information contained in each block, different sampling rates are adopted to reduce the overall sampling rate. Taking the two images in Fig. 1 as an example, lower CS sampling rates can be allocated for the block marked “C” in the image “Cameraman” and block “G” in image “Parrot”.
As a consequence, sensing resources should be reasonably allocated to each block, instead of equally assigned. In [BCS-Salie], less sampling rates are allocated to non-salient image blocks but more to salient ones using the characteristics of human visual system. Concretely, a low-resolution sensor is used produce an initially sampled image, such that the adaptive sampling can be achieved for the input scene. In [adaptive], the CS procedure is initialized with a low fixed-rate pre-sampling and an initial recovery. Then, the important regions are extracted by computing the saliency map of the initially recovered image. The adaptive CS strategy is further validated on some real video sequences [adaptive-tip]. In [Asymmetric], we also propose an asymmetric approach to ensure fairer allocation of sampling rates among image blocks.
 |
 |
Due to the fact that block partition breaks the global correlation of the whole image, block-based CS prone to generate reconstructed image of low quality. In [BCS], an iterative BCS reconstruction approach is proposed to remove the blocking artifacts. The recovered images via BCS algorithm are approximated on a block-by-block basis, but hard-thresholding denoising in each iteration is imposed on the full image, not image blocks. As a result, the artifacts incurred by block partition can be smoothed as the iteration progress. However, this can result in substantially increasing reconstruction complexity because of full-image denoising, which violates the motivation of lightweight design.
Ii-C Deep network approach for image CS
The tremendous success of deep learning in computer vision as shown in
[DNN-C, DNN-S] attracted application of deep neural networks in image CS [Ldamp, Reconnet, ISTA, CSnet]. When an imaging system acquires CS measurements, the reconstructing process is performed with a deep network. Compared with the traditional CS, deep network approaches generally enjoy much faster reconstruction speed, while still achieving high-quality recovered images owing to their significant learning capabilities.In [Reconnet], a network-based ReconNet approach is introduced to learn the inverse mapping from block-wise CS measurements to their desired image blocks. It is further improved in [Im-recon] that the measurement matrix and the reconstruction process are jointly learned. In [ISTA, Ldamp], traditional CS algorithms and deep networks are blended by treating parameters of the algorithm as weights to be learned. Unfortunately, full-image reconstruction is prone to overfitting due to the potentially overwhelming number of parameters of the sampling layer. So block-independent image recovery has to be performed instead of reconstructing the full image directly. As a consequence, blocking artifacts can be observed in the recovered images. To address the issue, the recovered images are fed into the BM3D denoiser designed in [BM3D] to remove blocking artifact [Reconnet, Im-recon]. However, the benefit of using an traditional off-the-shelf denoiser is not convincingly demonstrated.
For each sampling rate, the corresponding network model has to be trained to learn the inverse CS mapping. This may be not desirable for block-based CS, since a large number of model parameters need to be stored. In [Ldamp]
, a neural network architecture is applied to a variety of measurement matrices. Unfortunately, its performance improvement is less significant than traditional CS methods because the measurement matrices cannot participate in the network training. Inspired by the multi-scale super-resolution method given in
[Enhanced], [Multi] introduce a multi-scale CS approach, where the main network is shared across multiple sampling rates. However, their method only reuses a portion of parameters, and a CS sampling rate still correspond to a specific network model. In other words, they do not consider block partition and the corresponding problem of blocking artifacts.Iii Block-wise approximation and full-image-based denoising
In this section, we first introduce the key idea of the popular BCS algorithm [BCS], i.e., block-wise approximation and full-image-based denoising. Then we propose an extended version of the well known Damp approach [Damp] by exploiting this idea.
Iii-a Block-wise approximation and full-image-based denoising in BCS
The BCS algorithm is an iterative reconstruction approach specialized for block-based image CS. It solves the image reconstruction problem by using approximation with projection onto the convex set and hard thresholding denoising in the iteration process, as shown in
| (2) |
where is the measurement matrix corresponding to a block and is its pseudo-inverse, denotes the approximation of the recovered block , and all are reassembled into a full image . Here we assume that the original image are divided into blocks, is the measurement sampled from block . The reconstruction starts from some initial approximation and forms the recovered image at iteration. In the BCS approach, represents hard thresholding, which is widely used in removing Gaussian noise. In the special case when is an orthonormal matrix, we can deduce that , where is the transpose of .
We can conclude that, the key idea of BCS algorithm is block-wise approximation, as illustrated in the first equation in (2). The denoising operation, on the contrary, is imposed on the full image, not each block, as shown in the second equation in (2). In this way, blocking artifacts can be removed while still maintaining block-wise CS sampling.
Iii-B BCS-Damp algorithm
Inspired by BCS algorithm, in this subsection, we propose an extension for Damp algorithm, called BCS-Damp, to reconstruct the images with block partition.
Damp is a state-of-the-art CS reconstruction algorithm, which is also an iterative approach like BCS. Let be the measurement matrix of image , and is the corresponding measurements. Damp algorithm takes the form
| (3) |
where . The part in Eq. (3) can be written as , where is the original image and can be regarded as a Gaussian noise at iteration .
is an estimate of the standard deviation of that Gaussian noise.
is the number of CS measurements. div denotes the operation of partial derivative, and div represents the divergence of the denoiser.However, Damp is not specially designed for block-based CS, and correspondingly it does not consider blocking artifacts as a result. With the idea of BCS in mind, that is, block-wise approximation is employed to decrease computation complexity, while full-image-based denoising is imposed to remove blocking artifacts, we propose an extension of Damp for block-based image CS. Our proposed BCS-Damp algorithm is illustrated as
| (4) |
where represents the block in an image. Here we maintain the operation of in Damp algorithm. The modification we have to introduce is that, does not run on the blocks, but on the full image , obtained by concatenating all approximated blocks , as shown in the last equation in (4).
In the following section, block-wise approximation but full-image-based denoising, along with the iterative structure of BCS algorithm, will be further casted into a carefully designed deep network for removing artifacts and improving the recovered image.
Iv Multi-channel deep network architecture
In this section, we propose a multi-channel deep network architecture, termed BCS-Net, to reconstruct the images acquired by block-wise CS sampling. The proposed BCS-Net is composed of a multi-channel sampling network and a deep reconstructing network, as shown in Fig. 2. These two networks consist of an integrated end-to-end model, of which the learnable parameters are jointly trained by our proposed two-stage training strategy.
Iv-a Multi-channel sampling architecture
This subsection investigates a -channel network architecture, named , to mimic the adaptive sampling process of block-based CS, as shown in Fig. 3.
The proposed has channels, each of which corresponds to a sampling rate assigned to a specific image block. A higher value of indicates a more detailed division of sampling rates of image blocks, thereby block partition can benefit the fairer allocation of the sensing resources, but resulting in more complex sampling architecture. The full scene under view, , is partitioned into non-overlapping image blocks of size . Using the schematic image in Fig. 3 as an example, is set to 9. According to CS theory, for image block , the sampling process can be represented as . Here () is the measurement matrix of block , , , and is the corresponding measurements. We have , where is the sampling rate assigned to block . With our sampling network, a channel is related to a sampling rate, and channels can then correspond to a number of target sampling rates by employing linear combination of those sampling rates.
For the channel in , we use a convolutional layer, in which we do not set bias and activation, to mimic the sampling process. This convolutional layer is defined as , where denotes convolution kernels of size . In other words, corresponds to measurement matrix . Note that the measurements among different channels are different from each other, since each channel corresponds to its own convolutional kernel. However, the blocks within the same image are spatially correlated. As a consequence, the measurements of those blocks among different channels are related to each other. If is fed into channel, we have
| (5) |
The size of the convolution kernel depends on the sampling rate and the size of image blocks. We should note that, compared with traditional measurement matrix, the weights of are learnable. In this perspective, it is more rigorous that network-based CS approaches should be referred to as the ones inspired by CS, instead of being CS.
We note that channels in our sampling network corresponds to sampling rates, respectively, and may be less than , the total number of the blocks in an image. With our multi-channel model, the blocks have to be measured via their respective channels, and the number of the channels, , is then related to the number of blocks of an image, . Ideally, a block corresponds to a unique channel. In this case, we have . If several blocks within an image are similar to each other, such as those blocks belonging to the background, they can be considered to assign the same sampling rates. At this point, we have . Fortunately, all channels can be trained sufficiently in the training process, since we have enough training images, and thus each of these channels will receive enough training blocks regardless of the allocation strategies employed.
Obviously, if the full image is not partitioned into blocks, too many parameters may be needed to store the weights and it will be easily prone to overfitting. This may also be the main reason that most existing deep network approaches reconstruct the full image with block-by-block strategy. In this paper, we are going to further investigate the problem of blocking artifacts due to block partition.
Iv-B Deep reconstructing architecture
In this subsection, we construct a deep reconstruction architecture to cast the idea of iterative block-wise approximation but full-image-based denoising into the network, achieving model-level removal of blocking artifacts. Our deep architecture is composed of an initial reconstruction network and a deep reconstruction network , as shown in Fig. 4.
The initial reconstruction network, , has inputs, each of which corresponds to a sampling channel in as illustrated in Section IV-A. The () input is connected to the corresponding convolutional layer, which uses kernels of size and generates values by convolving them with . Here is the measurement of block entering channel. All these values are combined into one feature map, , and we refer to it as the initially reconstructed result of block . From the network’s point of view, we have
| (6) |
where denotes the above mentioned convolutional kernel corresponding to the channel in the sampling network. As we can see from Fig. 4, our initial reconstruction network includes only one convolutional layer for simplification reason, and the initially recovered images is going to be improved by our deep reconstruction network.
The proposed deep reconstruction network, , is further divided into phases, so that the iterative BCS algorithm can be unrolled along with these phases. In , each phase corresponds to one iteration in BCS algorithm consisting of approximation and denoising operations. At phase the block-wise approximation is implemented by using a formula slightly different from (2) in Sec. III-A, as shown in
| (7) |
for each block , where is the measurement matrix specialized for block , if is fed into the network via channel in our multi-channel sampling model. Note that the matrix
is learnable, and it may be not an orthogonal matrix. Thus, its pseudo-inverse,
, can not be simplified into as in the traditional BCS or Damp algorithm.We then reassemble all approximated blocks to build a full, approximated image,
, for further denoising processing. To enable training of the deep reconstruction network, we modify the famous DnCNN network to implement full-image denoising. Traditional denoising methods, such as hard thresholding in BCS and BM3D in Damp algorithm, will not work in deep network architecture, since they cannot propagate gradients. This restricts us to focus on feed-forward convolutional neural networks. DnCNN is our choice, which fortunately offers improved performance on image deblocking and Gaussian denoising. Our deep reconstruction network is composed of
phases. Each phase has 5 convolutional layers, and the configuration is designed by referring to the DnCNN network. The first layer generates feature maps with the kernels of size , and last layer generates 1 feature map with one kernel. All three other layers employ kernels, each of which is of the size of. It should be noted that all conventional layers explore the RELU activation function except the last layer. In DnCNN, 20 convolutional layers are employed to form a deep network for image denoising. In our network model, 5 convolutional layers form a phase, and
phases are employ to deal with both image denoising and image approximation. In our experiments, , and are set to 64, 3 and 10, respectively. Let be the parameters of convolutional kernels in phase. Then one has| (8) |
where is the approximated image in the first phase in the deep network.
Iv-C Two-stage training
We propose to divide the training process into two stages to improve the recovered images by training sampling matrix while being able of utilizing in deep reconstruction process.
As illustrated in Section IV-A, in our network sampling matrixes are implemented by employing convolution operations. That is, the elements in and have to be taken from the convolution kernel in the channel in the sampling network. However, we find that, if both and participate in the training process, we cannot achieve a desired recovered image. This is because, in the training process, has to be updated in real time along with each back propagation. Unfortunately, back propagation is based on the gradient decent rule, which will be hindered due to the real-time computing of in the deep reconstruction network.
In view of this, in the first stage, we aim to obtain the training parameters of the sampling network. That is, we have to achieve optimal sampling matrices and the corresponding matrices . It is observed that, our deep architecture consists of an initial reconstruction network without including and a deep reconstruction network where is utilized to improve the recovered images. In this way, we combine the sampling network and the initial reconstruction part of our reconstruction network into a training network, i.e., , which is used to train the sampling matrix . Given the training images , the cost function is
| (9) |
where is the number of images in the training dataset.
In the second stage, we further train the reconstructing network consisting of an initial reconstruction part and a deep reconstruction part, i.e., , where the parameters in come from . That is, the sampling weights are fixed while the parameters are updated in the training process. Our reconstructing network
directly learns the mapping between the CS measurements and the ground truth, and the loss function minimizing the error between the input and the output is on the basis of full images instead of image blocks. Mean square error is adopted to design an end-to-end cost function
| (10) |
where denotes the CS measurement vector of the block in training image.
V Performance evaluation
In this section, we conduct extensive experiments to evaluate the performance of the proposed BCS-Net and BCS-Damp schemes, and compare them with state-of-the-art methods, including traditional BCS [BCS], Damp [Damp], network-based Ista [ISTA], ReconNet [Reconnet] and its improved version, I-Recon [Im-recon], in terms of reconstruction quality, time complexity and visual effect.
V-a Training and testing
V-A1 Constructing a training set
The training images are from the training set (200 images) and testing set (200 images) of the BSDS500 database [BSD500], in which we randomly crop 89600 images with the size of as the training set. Each training image, , is further partitioned into 9 image blocks of size , . That is, there are a total of 806400 blocks in our training set. Visual saliency of the scene was exploited in [saliency, BCS-Salie]. In the experiments, the methods in reference [BCS-Salie] is used to compute the saliency map of training images. Suppose that represents the amount of the saliency information embodied in image . Then we have , where is the total number of pixels on image , denotes the saliency map of , and is the saliency value of location on . Let be the saliency information of image block , and denotes the proportion of the saliency information of block . We can then construct the training data pair for our network, as shown in
| (11) |
where and there are a total of training image pairs.
For three existing network-based approaches, Ista, ReconNet and I-Recon, 806400 image blocks are randomly cropped. These blocks and themselves consists of 806400 training block pairs for training, since these approaches are all based on block-independent image recovery.
V-A2 Training details
We set for our -channel network model, and the sampling rates are in the range of . Each image pair is further processed in order to find out the most appropriate channels in the network. For a given target sampling rate (), we calculate sub-rate of block as
| (12) |
where is defined in Section V-A1, and are the sizes of an image and its block, respectively. In the training process, the space of sampling rates is divided into seven intervals, each of which corresponds to a channel. If falls within the interval corresponding to channel, then block is pushed into the network via channel .
In the experiment, we train the network with 50 epoches. The batch size is set to 1, since each image have to be partitioned into 9 blocks and these blocks are reassembled in our multi-channel network. The mean square error between the original image and the output of the network is calculated as the loss for back-gradient propagation. Adam optimization
[adam]with a learning rate of 0.0001 is adopted to optimize the parameters. We use TensorFlow 1.4
[Tensor] to train the proposed multi-channel network at a desktop platform configured with one NVIDIA 1060 GPU, one CPU @ 4.00 GHz of Intel(R) Core (TM) i7-4790K and 32GB of memory. The training processes takes about 3 hours for one epoch.V-A3 Testing set
We test our multi-channel networks with three widely used benchmark datasets, Set5, Set11 and BSD100, where Set5 and Set11 are shown in Fig. 5 and Fig. 6, respectively.
Set5 consists of 5 gray images, where the sizes of “Bird” and “Head” are , “Baby”, “Butterfly” and “Woman” are with the size of , and , respectively. Set11 has 11 gray images, where the sizes of “Fingerprint” and “Flintstones” are , and the other 9 images are all with the size of . BSD100 includes 100 images with the size of or . These test images are with a various types of spatial distribution of key visual information. For example, the main meaningful information in images “Cameraman” and “Parrot” in Set11 is located in the single connected region. In contract, the visual information of “Bird” in Set5, “Fingerprint”, “Flintstones” and “Peppers” in Set11 uniformly distributes in the whole images. Note that all those test images are strictly separate from the training datasets.
a)
b)
c)
d)
e)
V-B Results and analysis
V-B1 Comparisons with the state-of-the-art methods
In this subsection, we evaluate the performance of the proposed BCS-Net with adaptive allocation of sampling rate and BCS-Damp, and compare them with the existing methods.
For our BCS-Net, all test images are reprocessed to simulate the initial CS sampling by conforming to the following simulations. Each original test image is first resized to one percent of its original size, which mimics the scene under view pre-sampled by a low-resolution imaging sensor. After that, the saliency of the pre-sampled image is computed. This small-size saliency map is further normalized and bilinearly interpolated to a map with the original size. The saliency information of the original test images,
, is then estimated. As a consequence, the sampling resources can be allocated to all blocks by using (12), instead of being equally allocated.To guarantee the average rate of the sampling rates corresponding to each channel being equal or close to the target rate, , the following procedure is employed. As with the training process, the space of sampling rates is first divided into seven intervals. The internal corresponds to channel, and is represented as , where and is the total number of channels in our multi-channel model. Then we compute the sampling rate for block by using (12), where and is the total number of blocks. If falls into the interval , the value of is changed to . Here is the sampling rate corresponding to channel and . If , then the average sampling rate of all block is exactly equal to the target sampling rate. Otherwise, we carry out fine tune by changing some blocks to the higher or lower channel according to the positive or negative difference, i.e., . We should note that the average sampling rates can be very close to the target rates, and thus we ignore the differences between them in the experiments.
The average PSNR (peak signal-to-noise ratios) and SSIM (structural similarity index) with BCS-Net are reported in Table I. The comparison of running times of reconstructing the images in Set5 and Set11 is shown in Table II. Here the running times are the average values of all 16 test images in Set5 and Set11 with the sampling rate of 0.1. We should note that, 0.1 is the target sampling rate of an image, and our running time contains the time of reconstructing all blocks with different channels in the multi-channel architecture.
| Set5 (PSNR/SSIM) | |||||||
|---|---|---|---|---|---|---|---|
| [width=10em]AlgorithmSampling rate | 0.01 | 0.03 | 0.05 | 0.1 | 0.2 | 0.3 | 0.4 |
| BCS [BCS] | 16.20/0.3613 | 20.80/0.5088 | 22.54/0.5974 | 24.87/0.7280 | 28.50/0.8130 | 30.54/0.8539 | 32.32/0.8899 |
| Damp [Damp] | 6.51/0.0311 | 19.95/0.4821 | 21.56/0.5667 | 24.24/0.6997 | 28.49/0.8527 | 32.29/0.9137 | 34.25/0.9395 |
| BCS-Damp | 19.97/0.4827 | 22.28/0.6267 | 24.10/0.6962 | 27.95/0.8247 | 32.47/0.9105 | 36.46/0.9397 | 39.53/0.9565 |
| ReconNet [Reconnet] | 18.46/0.4492 | 21.54/0.5699 | 23.33/0.6462 | 25.70/0.7422 | 28.16/0.8197 | 30.03/0.8620 | 31.00/0.8793 |
| I-Recon [Im-recon] | 21.49/0.5571 | 25.00/0.7113 | 26.97/0.7908 | 28.49/0.8329 | 30.53/0.8823 | 34.51/0.9403 | 35.30/0.9465 |
| Ista [ISTA] | 18.06/0.4589 | 21.50/0.5624 | 25.15/0.7307 | 28.89/0.8405 | 33.21/0.9152 | 36.00/0.9456 | 38.14/0.9622 |
| BCS-Net (WA) | 22.98/0.6103 | 26.69/0.7702 | 28.72/0.8371 | 31.86/0.9034 | 35.43/0.9488 | 37.87/0.9681 | 39.88/0.9785 |
| BCS-Net | 22.98/0.6103 | 27.09/0.7699 | 28.81/0.8237 | 32.71/0.9030 | 36.12/0.9483 | 38.64/0.9694 | 39.88/0.9785 |
| Set11 (PSNR/SSIM) | |||||||
| [width=10em]AlgorithmSampling rate | 0.01 | 0.03 | 0.05 | 0.1 | 0.2 | 0.3 | 0.4 |
| BCS [BCS] | 15.65/0.3973 | 19.40/0.5146 | 20.89/0.5768 | 23.15/0.6836 | 25.74/0.7778 | 27.78/0.8331 | 29.75/0.8783 |
| Damp [Damp] | 5.49/0.0582 | 18.47/0.4986 | 20.14/0.5588 | 22.73/0.6873 | 26.85/0.8335 | 30.09/0.8994 | 32.93/0.9339 |
| BCS-Damp | 18.53/0.4692 | 21.03/0.5795 | 22.61/0.6471 | 26.20/0.7919 | 30.75/0.8976 | 34.21/0.9415 | 37.09/0.9648 |
| ReconNet [Reconnet] | 16.99/0.4145 | 19.80/0.5110 | 21.14/0.5935 | 23.28/0.6896 | 25.54/0.7719 | 27.11/0.8155 | 28.32/0.8411 |
| I-Recon [Im-recon] | 19.80/0.5018 | 22.59/0.6540 | 24.54/0.7442 | 25.97/0.7888 | 27.92/0.8457 | 31.45/0.9135 | 32.26/0.9243 |
| Ista [ISTA] | 16.55/0.4139 | 19.74/0.5154 | 22.83/0.6792 | 26.49/0.8010 | 30.79/0.8950 | 33.76/0.9345 | 36.03/0.9547 |
| BCS-Net (WA) | 20.88/0.5505 | 24.05/0.7048 | 25.89/0.7851 | 28.63/0.8628 | 32.08/0.9220 | 34.65/0.9506 | 36.70/0.9662 |
| BCS-Net | 20.88/0.5505 | 24.47/0.7807 | 26.04/0.7723 | 29.43/0.8676 | 33.06/0.9283 | 35.60/0.9554 | 36.70/0.9662 |
| BSD100 (PSNR/SSIM) | |||||||
| [width=10em]AlgorithmSampling rate | 0.01 | 0.03 | 0.05 | 0.1 | 0.2 | 0.3 | 0.4 |
| BCS [BCS] | 18.53/0.4213 | 20.93/0.4797 | 21.54/0.4996 | 23.21/0.5843 | 25.04/0.6742 | 26.48/0.7384 | 27.75/0.7875 |
| Damp [Damp] | 7.00/0.0495 | 19.57/0.4260 | 20.54/0.4706 | 21.92/0.5333 | 24.29/0.6286 | 26.00/0.6970 | 27.57/0.7541 |
| BCS-Damp | 19.90/0.4255 | 21.58/0.4896 | 22.47/0.5216 | 24.10/0.5842 | 26.48/0.6745 | 28.79/0.7490 | 31.44/0.8184 |
| ReconNet [Reconnet] | 18.74/0.3960 | 20.56/0.4647 | 21.52/0.5120 | 23.00/0.5837 | 24.68/0.6694 | 25.83/0.7204 | 26.71/0.7611 |
| I-Recon [Im-recon] | 21.15/0.4654 | 23.09/0.5662 | 24.19/0.6320 | 25.35/0.7098 | 26.87/0.7872 | 29.24/0.8593 | 30.07/0.8812 |
| Ista [ISTA] | 17.86/0.3957 | 20.47/0.4699 | 22.65/0.5692 | 24.79/0.6726 | 27.64/0.7906 | 29.86/0.8580 | 31.70/0.9003 |
| BCS-Net (WA) | 22.03/0.4997 | 24.10/0.6085 | 25.16/0.6709 | 26.97/0.7651 | 29.52/0.8609 | 31.59/0.9107 | 33.44/0.9405 |
| BCS-Net | 22.03/0.4997 | 24.43/0.6108 | 25.58/0.6669 | 27.84/0.7709 | 30.59/0.8672 | 32.64/0.9160 | 33.44/0.9405 |
The best performance is labeled in bold, the second best is italic, and the second best is underlined.
| Algorithm | BCS | Damp | BCS-Damp | ReconNet | I-Recon | Ista | BCS-Net(WA) | BCS-Net |
|---|---|---|---|---|---|---|---|---|
| Time | 100s | 228s | 611s | 0.87s | 0.89s | 1.05s | 2.02s |
As shown in Table I, our BCS-Net yields higher-quality recovered image in terms of both PSNR and SSIM than other existing methods, including BCS, Damp, ReconNet, I-Recon, and Ista, for Set5, Set11, and BSD100, respectively. From Table I, we can easily observe a significant performance improvement of, for instance, 3.82 dB and 2.98 dB on Set5 and Set11 with the sampling rate of 0.1, and 2.95 dB on BSD 100 with the sampling rate of 0.2, respectively. We notice that, along with the increase of the sampling rate, the improvement of our scheme slows down. The possible cause is that, when the sampling rate is as high as 0.1 or 0.2, the recovered images for the competing approach are of relatively high quality. And there is not too much space for improvement with even more sampling rates.
We can also see from Table I that, in existing methods I-Recon has relatively better reconstruction quality at extremely low sampling rates of 001, 0.03 and 0.05, while Ista usually performs better at the sampling rates of 0.2 and above. However, our BCS-Net always outperforms the traditional BCS and Damp algorithms, as well as the network-based ReconNet, I-Recon and Ista algorithms. We think that, the performance improvement of our scheme is mainly due to the following two factors, i.e., adaptive allocation of sampling rate in our multi-channel sampling network, and block-wise approximation and full-image denoising in our deep reconstruction network. We notice that BCS-Net has slightly longer running time than network-based ReconNet, I-Recon and Ista because of our multi-channel sampling and block reassembling, but it runs significantly far faster than optimization-based BCS and Damp reconstruction algorithm.
The proposed BCS-Damp approach also outperforms Damp and BCS algorithm, as shown in Table I. This is because, compared with Damp algorithm, in our BCS-Damp, BM3D denoising is imposed on the full image instead of each block, and blocking artifacts can then ameliorated. And compared with BCS approach, our BCS-Damp has better denoising performance, since BM3D denoising outperforms the hard thresholding employed in BCS approach. We notice that the proposed BCS-Damp even outperforms BCS-Net in terms of PSNR for Set11 at very high sampling rate of 0.4. We think this is another indication that network-based approaches generally offer more advantages over relatively lower sampling rates. Note that our BCS-Damp has much longer running time than BCS and Damp due to our full-image denoising strategy in the iteration process and relatively higher computation complexity of BM3D denoising algorithm.
| Images | SR | BCS | Damp | BCS-Damp | ReconNet | I-Recon | Ista | BCS-Net | BCS-Net(WA) |
| Baby | 0.01 | 18.20/0.4838 | 4.46/0.0288 | 24.29/0.6615 | 21.19/0.5582 | 24.22/0.6430 | 20.87/0.5778 | 26.35/0.6988 | 26.35/0.6988 |
| 0.03 | 24.60/0.7020 | 23.37/0.6554 | 25.50/0.7403 | 24.24/0.6482 | 27.88/0.7614 | 23.67/0.6347 | 30.66/0.8274 | 29.59/0.8013 | |
| 0.05 | 26.10/0.7646 | 25.30/0.7265 | 27.34/0.7850 | 25.77/0.7084 | 29.80/0.8294 | 27.68/0.7671 | 32.07/0.8689 | 31.27/0.8493 | |
| 0.1 | 28.81/0.8684 | 27.37/0.8034 | 29.90/0.8746 | 28.03/0.7749 | 30.86/0.8605 | 30.23/0.8381 | 34.72/0.9137 | 33.63/0.9075 | |
| 0.2 | 31.57/0.9328 | 30.10/0.8968 | 33.73/0.9502 | 30.56/0.8459 | 32.58/0.8997 | 33.25/0.9055 | 37.74/0.9579 | 36.52/0.9525 | |
| 0.3 | 33.31/0.9583 | 32.95/0.9508 | 37.38/0.9760 | 31.93/0.8770 | 36.12/0.9451 | 35.52/0.9410 | 39.94/0.9755 | 38.59/0.9715 | |
| 0.4 | 34.89/0.9730 | 34.91/0.9732 | 40.56/0.9872 | 32.84/0.8911 | 36.65/0.9461 | 37.41/0.9611 | 40.52/0.9822 | 40.52/0.9822 | |
| Bird | 0.01 | 17.90/0.3885 | 8.33/0.0225 | 19.98/0.4757 | 19.38/0.4609 | 22.14/0.5726 | 19.00/0.4628 | 23.35/0.6391 | 23.35/0.6391 |
| 0.03 | 20.72/0.4550 | 20.87/0.4850 | 22.35/0.6164 | 22.25/0.6085 | 26.07/0.7543 | 22.49/0.5930 | 27.60/0.7871 | 27.98/0.8142 | |
| 0.05 | 23.70/0.6481 | 21.67/0.5280 | 24.43/0.7029 | 24.32/0.6827 | 28.22/0.8325 | 26.32/0.7657 | 28.96/0.8286 | 30.55/0.8832 | |
| 0.1 | 26.67/0.7584 | 25.27/0.7030 | 29.40/0.8570 | 27.02/0.7964 | 30.25/0.8942 | 31.20/0.8852 | 34.88/0.9369 | 34.40/0.9454 | |
| 0.2 | 30.41/0.8583 | 31.50/0.8963 | 37.51/0.9644 | 29.71/0.8740 | 32.70/0.9350 | 36.83/0.9576 | 39.60/0.9770 | 39.20/0.9790 | |
| 0.3 | 33.22/0.9085 | 36.89/0.9623 | 42.26/0.9862 | 32.12/0.9148 | 37.65/0.9745 | 40.50/0.9785 | 42.69/0.9889 | 42.29/0.9895 | |
| 0.4 | 35.50/0.9362 | 37.21/0.9753 | 45.56/0.9928 | 33.22/0.9308 | 38.62/0.9788 | 43.10/0.9867 | 44.67/0.9937 | 44.67/0.9937 | |
| Butterfly | 0.01 | 11.98/0.2601 | 5.23/0.0006 | 13.08/0.3317 | 13.36/0.2766 | 15.47/0.3802 | 12.97/0.2851 | 15.68/0.4078 | 15.68/0.4078 |
| 0.03 | 14.88/0.3615 | 13.10/0.3215 | 15.28/0.4779 | 15.77/0.4041 | 19.36/0.6211 | 15.71/0.4032 | 21.01/0.6988 | 20.62/0.7046 | |
| 0.05 | 16.32/0.4330 | 14.98/0.4477 | 17.41/0.5913 | 17.52/0.5319 | 21.58/0.7326 | 19.31/0.6666 | 23.56/0.7947 | 23.55/0.8211 | |
| 0.1 | 18.71/0.5545 | 17.65/0.6308 | 22.21/0.7999 | 20.11/0.6581 | 22.78/0.7511 | 24.56/0.8479 | 27.94/0.8968 | 27.46/0.9015 | |
| 0.2 | 21.44/0.6622 | 22.80/0.8348 | 26.84/0.8994 | 22.74/0.7596 | 24.85/0.8200 | 30.17/0.9355 | 30.96/0.9386 | 31.24/0.9504 | |
| 0.3 | 23.50/0.7233 | 26.46/0.9052 | 29.99/0.9375 | 24.80/0.8227 | 29.54/0.9286 | 33.66/0.9604 | 33.98/0.9680 | 33.67/0.9698 | |
| 0.4 | 25.26/0.7698 | 29.77/0.9387 | 34.24/0.9648 | 25.81/0.8401 | 30.34/0.9362 | 36.50/0.9741 | 35.60/0.9791 | 35.60/0.9791 | |
| Head | 0.01 | 17.60/0.3134 | 9.44/0.0964 | 24.62/0.4831 | 21.33/0.5025 | 25.30/0.6133 | 21.08/0.5199 | 28.19/0.6603 | 28.19/0.6603 |
| 0.03 | 24.79/0.5102 | 24.42/0.4616 | 27.37/0.6304 | 25.09/0.6025 | 28.21/0.6939 | 25.54/0.6128 | 30.73/0.7368 | 30.54/0.7351 | |
| 0.05 | 25.97/0.5512 | 26.65/0.5626 | 28.62/0.6692 | 26.97/0.6408 | 29.71/0.7469 | 19.31/0.6831 | 31.96/0.7813 | 31.74/0.7792 | |
| 0.1 | 29.86/0.7232 | 28.50/0.6380 | 30.31/0.7282 | 28.90/0.7115 | 30.88/0.8091 | 30.82/0.7561 | 33.89/0.8426 | 33.60/0.8400 | |
| 0.2 | 31.74/0.7876 | 31.17/0.7610 | 32.37/0.7926 | 30.41/0.7694 | 32.46/0.8574 | 33.34/0.8352 | 35.99/0.8996 | 35.81/0.8977 | |
| 0.3 | 33.10/0.8320 | 32.67/0.8036 | 33.50/0.8238 | 32.08/0.8054 | 35.51/0.8975 | 35.07/0.8819 | 37.71/0.9311 | 37.43/0.9285 | |
| 0.4 | 34.27/0.8650 | 33.92/0.8380 | 34.55/0.8513 | 32.53/0.8276 | 36.02/0.9100 | 36.38/0.9117 | 38.85/0.9485 | 38.85/0.9485 | |
| Woman | 0.01 | 15.29/0.3608 | 5.10/0.0074 | 17.89/0.4613 | 17.06/0.4475 | 20.30/0.5762 | 16.41/0.4490 | 21.30/0.6457 | 21.30/0.6457 |
| 0.03 | 19.01/0.5153 | 18.01/0.4867 | 20.92/0.6684 | 20.32/0.5861 | 23.47/0.7258 | 20.08/0.5684 | 25.44/0.7995 | 24.71/0.7958 | |
| 0.05 | 20.63/0.5902 | 19.19/0.5684 | 22.72/0.7325 | 22.06/0.6674 | 25.55/0.8125 | 23.97/0.7711 | 27.50/0.8450 | 26.48/0.8529 | |
| 0.1 | 24.13/0.7354 | 22.41/0.7235 | 27.95/0.8640 | 24.45/0.7702 | 27.67/0.8497 | 27.62/0.8753 | 32.12/0.9247 | 30.22/0.9226 | |
| 0.2 | 27.36/0.8242 | 26.88/0.8747 | 31.88/0.9456 | 27.36/0.8497 | 30.06/0.8994 | 32.46/0.9421 | 36.30/0.9684 | 34.37/0.9645 | |
| 0.3 | 29.55/0.8724 | 32.50/0.9465 | 39.19/0.9751 | 29.23/0.8898 | 33.77/0.9557 | 35.24/0.9661 | 38.90/0.9835 | 37.38/0.9811 | |
| 0.4 | 31.64/0.9053 | 35.46/0.9722 | 42.74/0.9865 | 30.59/0.9068 | 34.90/0.9615 | 37.32/0.9773 | 39.74/0.9888 | 39.74/0.9888 |
| Images | SR | BCS | Damp | BCS-Damp | ReconNet | I-Recon | Ista | BCS-Net | BCS-Net(WA) |
| House | 0.01 | 17.70/0.5555 | 5.01/0.0704 | 21.76/0.6336 | 19.10/0.5401 | 22.16/0.6125 | 19.30/0.5487 | 23.65/0.6767 | 23.65/0.6767 |
| 0.03 | 22.04/0.6457 | 20.59/0.6651 | 25.74/0.7618 | 22.47/0.6046 | 25.51/0.7173 | 22.48/0.6034 | 29.76/0.8128 | 28.28/0.7914 | |
| 0.05 | 23.70/0.6766 | 24.38/0.7092 | 28.53/0.8049 | 24.00/0.6678 | 27.70/0.7814 | 26.68/0.7607 | 31.21/0.8413 | 30.29/0.8281 | |
| 0.1 | 26.36/0.7372 | 27.25/0.7951 | 32.45/0.8525 | 26.55/0.7398 | 29.06/0.8016 | 30.55/0.8366 | 33.62/0.8795 | 32.75/0.8734 | |
| 0.2 | 29.72/0.8104 | 32.12/0.8540 | 35.48/0.8828 | 28.90/0.7927 | 30.96/0.8430 | 35.10/0.8921 | 37.30/0.9183 | 36.23/0.9134 | |
| 0.3 | 31.49/0.8390 | 35.05/0.8833 | 38.12/0.9326 | 30.40/0.8220 | 34.42/0.8984 | 37.17/0.9198 | 38.62/0.9386 | 38.16/0.9360 | |
| 0.4 | 33.92/0.8921 | 37.08/0.9177 | 40.15/0.9532 | 31.57/0.8379 | 35.24/0.9057 | 38.67/0.9400 | 39.71/0.9557 | 39.71/0.9557 | |
| Cameraman | 0.01 | 16.56/0.4689 | 6.33/0.1057 | 17.98/0.4725 | 17.01/0.4502 | 19.65/0.5543 | 16.33/0.4624 | 20.38/0.6148 | 20.38/0.6148 |
| 0.03 | 18.70/0.5384 | 17.64/0.5119 | 20.52/0.6318 | 19.12/0.5225 | 21.65/0.6490 | 19.44/0.5151 | 23.56/0.7430 | 22.64/0.7117 | |
| 0.05 | 20.05/0.5863 | 18.95/0.5920 | 21.68/0.6760 | 20.08/0.5774 | 23.00/0.7143 | 20.82/0.6598 | 25.43/0.7935 | 23.82/0.7666 | |
| 0.1 | 21.71/0.6478 | 20.92/0.6640 | 24.44/0.7635 | 21.76/0.6553 | 23.81/0.7380 | 23.50/0.7559 | 28.02/0.8612 | 26.06/0.8376 | |
| 0.2 | 24.30/0.7380 | 23.67/0.7711 | 28.54/0.8463 | 23.27/0.7194 | 25.43/0.7948 | 27.46/0.8556 | 30.74/0.9142 | 29.05/0.9004 | |
| 0.3 | 25.97/0.7880 | 26.33/0.8266 | 30.97/0.8805 | 24.60/0.7618 | 28.12/0.8739 | 30.20/0.9025 | 32.44/0.9392 | 31.11/0.9285 | |
| 0.4 | 27.90/0.8334 | 27.86/0.8570 | 34.50/0.9435 | 25.53/0.7853 | 29.03/0.8862 | 32.25/0.9274 | 32.62/0.9463 | 32.62/0.9463 | |
| Parrot | 0.01 | 17.58/0.5604 | 5.85/0.0690 | 20.09/0.6576 | 17.26/0.5491 | 20.75/0.6509 | 16.75/0.5325 | 22.23/0.7164 | 22.23/0.7164 |
| 0.03 | 20.46/0.6447 | 20.16/0.6586 | 22.14/0.7185 | 20.53/0.6197 | 22.79/0.7377 | 20.41/0.6269 | 25.35/0.8229 | 24.23/0.8014 | |
| 0.05 | 22.30/0.7160 | 21.23/0.6935 | 23.45/0.7705 | 21.68/0.6868 | 24.53/0.8031 | 23.32/0.7585 | 27.87/0.8638 | 25.50/0.8405 | |
| 0.1 | 24.05/0.7771 | 22.83/0.7693 | 28.12/0.8563 | 23.52/0.7562 | 26.03/0.8293 | 26.72/0.8499 | 31.14/0.9126 | 28.39/0.8957 | |
| 0.2 | 26.21/0.8375 | 25.60/0.8418 | 32.90/0.9073 | 25.84/0.8171 | 28.00/0.8731 | 29.54/0.9101 | 34.36/0.9496 | 31.67/0.9389 | |
| 0.3 | 28.18/0.8754 | 28.19/0.8878 | 35.55/0.9351 | 27.20/0.8453 | 31.35/0.9286 | 32.92/0.9398 | 36.90/0.9668 | 34.71/0.9602 | |
| 0.4 | 30.00/0.9057 | 30.82/0.9169 | 38.17/0.9574 | 28.38/0.8648 | 31.85/0.9344 | 35.41/0.9579 | 37.23/0.9736 | 37.23/0.9736 | |
| Peppers | 0.01 | 14.53/0.4051 | 5.78/0.0621 | 17.37/0.5129 | 16.57/0.4309 | 19.20/0.5369 | 15.94/0.4263 | 20.67/0.6088 | 20.67/0.6088 |
| 0.03 | 17.96/0.4620 | 18.58/0.5333 | 19.93/0.6371 | 19.36/0.5086 | 22.28/0.6969 | 19.47/0.5328 | 23.69/0.7542 | 23.90/0.7607 | |
| 0.05 | 20.99/0.5872 | 19.87/0.5936 | 21.58/0.6839 | 20.63/0.6023 | 24.15/0.7776 | 22.49/0.6901 | 24.87/0.7996 | 25.46/0.8254 | |
| 0.1 | 23.76/0.6694 | 22.60/0.7055 | 25.97/0.7939 | 22.91/0.7002 | 25.26/0.7986 | 27.28/0.8170 | 28.35/0.8897 | 28.30/0.8868 | |
| 0.2 | 24.86/0.6751 | 28.10/0.8371 | 31.10/0.8854 | 25.02/0.7691 | 26.91/0.8459 | 32.52/0.9075 | 31.99/0.9305 | 32.08/0.9337 | |
| 0.3 | 26.83/0.7374 | 31.01/0.8925 | 34.85/0.9262 | 26.73/0.8167 | 31.33/0.9141 | 35.45/0.9383 | 34.59/0.9506 | 34.38/0.9502 | |
| 0.4 | 29.88/0.8303 | 34.02/0.9213 | 37.08/0.9435 | 28.00/0.8349 | 31.27/0.9214 | 37.47/0.9537 | 36.10/0.9617 | 36.10/0.9617 | |
| Boats | 0.01 | 16.46/0.3843 | 5.46/0.0593 | 19.42/0.4605 | 18.29/0.4089 | 20.85/0.4837 | 17.54/0.4058 | 22.05/0.5402 | 22.05/0.5402 |
| 0.03 | 20.66/0.4870 | 17.82/0.4902 | 21.55/0.5623 | 20.97/0.5073 | 23.95/0.6375 | 20.95/0.5030 | 25.67/0.7043 | 25.27/0.6946 | |
| 0.05 | 21.31/0.5552 | 20.78/0.5396 | 23.21/0.6315 | 22.48/0.5823 | 25.80/0.7234 | 23.93/0.6708 | 27.54/0.7777 | 27.27/0.7782 | |
| 0.1 | 22.65/0.6355 | 23.46/0.6514 | 26.27/0.7502 | 24.55/0.6790 | 27.55/0.7891 | 27.46/0.7966 | 30.39/0.8679 | 30.01/0.8672 | |
| 0.2 | 24.54/0.7426 | 27.76/0.8147 | 31.41/0.8873 | 26.93/0.7698 | 29.60/0.8565 | 31.94/0.9057 | 33.67/0.9244 | 33.49/0.9326 | |
| 0.3 | 28.59/0.8024 | 31.11/0.8973 | 35.87/0.9469 | 28.61/0.8256 | 33.04/0.9234 | 35.27/0.9479 | 36.14/0.9546 | 35.69/0.9550 | |
| 0.4 | 30.49/0.8506 | 35.55/0.9471 | 39.14/0.9702 | 29.76/0.8509 | 34.02/0.9335 | 37.75/0.9668 | 37.63/0.9686 | 37.63/0.9686 |
| Images | SR | BCS | Damp | BCS-Damp | ReconNet | I-Recon | Ista | BCS-Net | BCS-Net(WA) |
| Monarch | 0.01 | 13.38/0.3399 | 6.25/0.0462 | 14.94/0.3961 | 14.69/0.3612 | 17.43/0.4803 | 14.35/0.3708 | 17.82/0.4981 | 17.82/0.4981 |
| 0.03 | 16.68/0.4536 | 15.51/0.4449 | 17.77/0.5305 | 17.92/0.5005 | 21.11/0.6781 | 17.40/0.4917 | 23.30/0.7287 | 22.72/0.7409 | |
| 0.05 | 18.43/0.5281 | 17.05/0.5321 | 20.01/0.6324 | 19.06/0.5875 | 23.47/0.7758 | 21.13/0.6930 | 24.65/0.7718 | 25.54/0.8295 | |
| 0.1 | 21.08/0.6369 | 20.09/0.6519 | 24.07/0.7821 | 21.82/0.7020 | 24.61/0.8016 | 25.53/0.8366 | 29.32/0.8937 | 28.86/0.9065 | |
| 0.2 | 24.14/0.7480 | 24.87/0.8413 | 28.76/0.9005 | 24.45/0.7960 | 26.67/0.8620 | 30.98/0.9335 | 33.23/0.9527 | 32.77/0.9563 | |
| 0.3 | 26.79/0.8144 | 28.49/0.9181 | 32.79/0.9499 | 26.24/0.8468 | 31.03/0.9396 | 34.87/0.9640 | 36.11/0.9743 | 35.30/0.9742 | |
| 0.4 | 28.49/0.8527 | 31.76/0.9519 | 37.98/0.9750 | 27.54/0.8685 | 31.91/0.9481 | 37.73/0.9766 | 37.35/0.9826 | 37.35/0.9826 | |
| Foreman | 0.01 | 18.06/0.5834 | 4.92/0.1082 | 23.52/0.7136 | 19.71/0.5991 | 23.68/0.6997 | 19.00/0.5895 | 26.22/0.7565 | 26.22/0.7565 |
| 0.03 | 24.22/0.7102 | 23.81/0.7271 | 28.49/0.8314 | 23.77/0.6807 | 27.15/0.8056 | 23.5/0.6740 | 30.68/0.8442 | 30.70/0.8579 | |
| 0.05 | 25.95/0.7359 | 26.06/0.7846 | 30.96/0.8664 | 25.73/0.7466 | 29.33/0.8534 | 28.49/0.8272 | 32.35/0.8752 | 32.27/0.8886 | |
| 0.1 | 29.04/0.8149 | 30.64/0.8518 | 33.78/0.9066 | 27.79/0.8039 | 29.90/0.8549 | 33.18/0.8995 | 35.85/0.9325 | 35.18/0.9293 | |
| 0.2 | 32.01/0.8674 | 34.74/0.9193 | 36.62/0.9325 | 30.03/0.8551 | 31.94/0.8892 | 37.75/0.9456 | 39.00/0.9619 | 38.29/0.9608 | |
| 0.3 | 34.48/0.9019 | 37.28/0.9396 | 38.80/0.9484 | 31.81/0.8802 | 35.96/0.9421 | 40.32/0.9647 | 40.99/0.9753 | 40.23/0.9738 | |
| 0.4 | 36.40/0.9253 | 38.60/0.9497 | 41.24/0.9649 | 32.73/0.8887 | 36.20/0.9425 | 42.33/0.9757 | 41.77/0.9811 | 41.77/0.9811 | |
| Barbara | 0.01 | 16.89/0.3458 | 5.66/0.0402 | 19.38/0.4291 | 18.17/0.3727 | 20.82/0.4607 | 17.51/0.3665 | 21.75/0.4959 | 21.75/0.4959 |
| 0.03 | 20.66/0.4694 | 19.53/0.4380 | 20.90/0.4804 | 20.43/0.4642 | 22.61/0.5813 | 20.32/0.4681 | 23.24/0.5996 | 23.32/0.6216 | |
| 0.05 | 21.31/0.5193 | 20.86/0.4716 | 21.71/0.5128 | 21.13/0.5157 | 23.40/0.6388 | 21.80/0.5661 | 23.64/0.6380 | 23.75/0.6570 | |
| 0.1 | 22.65/0.5961 | 22.36/0.5683 | 23.52/0.6199 | 22.20/0.5859 | 23.88/0.6915 | 23.53/0.6842 | 24.73/0.7283 | 24.50/0.7270 | |
| 0.2 | 24.53/0.7091 | 25.12/0.7253 | 29.26/0.8764 | 23.08/0.6525 | 24.75/0.7571 | 26.76/0.8331 | 29.29/0.8805 | 26.64/0.8279 | |
| 0.3 | 26.06/0.7814 | 29.76/0.8777 | 35.40/0.9553 | 23.87/0.7019 | 27.99/0.8616 | 30.76/0.9255 | 33.03/0.9479 | 30.78/0.9225 | |
| 0.4 | 27.47/0.8356 | 34.29/0.9527 | 38.03/0.9735 | 25.14/0.7598 | 29.56/0.8986 | 34.09/0.9606 | 34.23/0.9637 | 34.23/0.9637 | |
| Lena | 0.01 | 14.79/0.3779 | 5.83/0.0700 | 19.35/0.5126 | 17.81/0.4427 | 21.02/0.5437 | 17.22/0.4371 | 22.00/0.5956 | 22.00/0.5956 |
| 0.03 | 20.55/0.5481 | 19.66/0.5570 | 22.34/0.6565 | 20.94/0.5322 | 23.89/0.6788 | 20.91/0.5453 | 25.88/0.7504 | 25.64/0.7461 | |
| 0.05 | 22.41/0.6115 | 20.96/0.6153 | 23.80/0.7082 | 22.52/0.6285 | 25.85/0.7614 | 24.69/0.7265 | 26.11/0.7824 | 27.15/0.8047 | |
| 0.1 | 24.20/0.6803 | 24.19/0.7137 | 26.32/0.7794 | 24.45/0.7079 | 26.86/0.7957 | 27.69/0.8227 | 30.62/0.8895 | 29.36/0.8737 | |
| 0.2 | 26.43/0.7736 | 27.05/0.8172 | 30.93/0.8895 | 26.55/0.7825 | 28.82/0.8581 | 31.37/0.9067 | 34.36/0.9481 | 32.29/0.9316 | |
| 0.3 | 28.87/0.8301 | 29.01/0.8677 | 34.82/0.9432 | 27.99/0.8235 | 32.18/0.9261 | 33.83/0.9419 | 36.71/0.9686 | 35.07/0.9612 | |
| 0.4 | 30.44/0.8680 | 31.31/0.9063 | 37.49/0.9655 | 29.24/0.8521 | 33.13/0.9372 | 36.25/0.9622 | 37.30/0.9755 | 37.30/0.9755 | |
| Flintstones | 0.01 | 12.23/0.2009 | 4.33/0.0060 | 14.50/0.2814 | 13.59/0.2421 | 16.13/0.3335 | 13.45/0.2500 | 16.65/0.3800 | 16.65/0.3800 |
| 0.03 | 14.96/0.3500 | 14.48/0.3242 | 16.04/0.4268 | 15.87/0.3365 | 18.72/0.4968 | 15.92/0.3458 | 19.74/0.5683 | 19.49/0.5664 | |
| 0.05 | 15.07/0.3575 | 15.56/0.4251 | 17.21/0.5738 | 17.32/0.4397 | 20.51/0.6134 | 18.99/0.5629 | 22.61/0.6708 | 21.38/0.6751 | |
| 0.1 | 18.22/0.6299 | 17.64/0.6448 | 21.62/0.8325 | 19.65/0.5663 | 22.63/0.6929 | 23.47/0.7489 | 25.29/0.8044 | 25.03/0.8065 | |
| 0.2 | 20.99/0.7820 | 22.87/0.8816 | 28.20/0.9540 | 22.74/0.7061 | 25.37/0.7852 | 28.56/0.8586 | 29.37/0.8792 | 29.23/0.8876 | |
| 0.3 | 23.26/0.8617 | 27.51/0.9512 | 30.47/0.9701 | 24.59/0.7637 | 29.00/0.8758 | 30.86/0.8920 | 31.63/0.9127 | 31.26/0.9141 | |
| 0.4 | 25.18/0.9040 | 29.92/0.9690 | 31.80/0.9761 | 25.87/0.7915 | 29.73/0.8862 | 32.31/0.9142 | 32.60/0.9300 | 32.60/0.9300 | |
| Fingerprint | 0.01 | 13.96/0.1487 | 4.93/0.0027 | 15.48/0.0913 | 14.74/0.1625 | 16.08/0.1634 | 14.64/0.1631 | 16.29/0.1722 | 16.29/0.1722 |
| 0.03 | 16.78/0.3512 | 15.37/0.1338 | 15.89/0.1378 | 16.39/0.3438 | 18.81/0.5152 | 16.38/0.3629 | 18.30/0.4676 | 18.32/0.4686 | |
| 0.05 | 17.55/0.4712 | 15.89/0.1905 | 16.61/0.2580 | 17.87/0.4935 | 22.23/0.7433 | 18.83/0.5556 | 21.18/0.6812 | 22.39/0.7430 | |
| 0.1 | 19.68/0.6941 | 18.05/0.5441 | 21.64/0.7736 | 20.93/0.6896 | 26.04/0.8843 | 22.50/0.7626 | 26.36/0.8847 | 26.49/0.8872 | |
| 0.2 | 22.77/0.8718 | 23.46/0.8654 | 25.01/0.9111 | 24.18/0.8308 | 28.68/0.9383 | 26.76/0.8903 | 30.35/0.9513 | 31.10/0.9590 | |
| 0.3 | 25.06/0.9329 | 27.28/0.9515 | 28.72/0.9687 | 26.13/0.8829 | 31.55/0.9644 | 29.69/0.9426 | 34.46/0.9806 | 34.47/0.9806 | |
| 0.4 | 27.08/0.9633 | 31.03/0.9836 | 32.39/0.9900 | 27.77/0.9180 | 32.88/0.9738 | 32.10/0.9663 | 37.19/0.9894 | 37.19/0.9894 |
V-B2 Performance evaluation without assigning sampling resources
In this subsection, we evaluate the performance of our BCS-Net scheme without assigning sampling rate (WA), since the sensing resources may not be allocated in certain scenarios. That is, all blocks in an image are assigned the same sampling rate, and accordingly, they are fed into our model from the same channel corresponding to the target rate. In this case, our multi-channel architecture becomes a unified deep network for target sampling rates. In the simulation, is set to 7, and these seven channels are in a range of . The comparison of average PSNR and SSIM of recovered Set5, Set11 and BSD100 can be observed in Table I. The detailed comparison of PSNR and SSIM for Set5 and Set11 is illustrated in Table III and IV, where we omit the results of testing set BSD100 due to limited space.
We should note that, our multi-channel model decreases large amount of storage requirements, since we use a unified deep reconstruction network to serve all sampling rates. For instance, Ista approach for seven different sampling rates has about 0.34 million (M) parameters each, totaling 2.4M, while our reconstruction network has only 1.1M parameters.
Table I shows that, the proposed WA scheme still achieves higher PSNR and SSIM than the best results of the existing BCS, Damp, ReconNet, I-Recon and Ista, thanks to the strategy of block-wise approximation and full-image based denoising. We can see from Table I that, our WA scheme always has lower PSNR than the proposed BCS-Net with adaptive allocation at the sampling rates of for Set5, Set11 and BSD100. We also notice that, for Set5, our WA scheme achieves slightly better SSIM than BCS-Net with adaptive allocation. From Table III, IV, the quality gap between BCS-Net and BCS-Net (WA) increases in some images, but decreases in other images. For example, the gap of recovered “Cameraman” and “Parrot” is about 28.02-26.06=1.96(dB) and 31.14-28.39=2.59(dB), but it drops to 25.29-25.03=0.26(dB) and 26.36-24.49=-0.13(dB) for “Flintstones” and “Fingerprint” with sampling rate of 0.1, respectively. This is because, the foreground objects of “Cameraman” and “Parrot” can be clearly distinguished from background information. In other words, the main meaning information in these images is limited to some local regions. As a consequence, the average assignment of sampling rate is obviously less efficient than adaptive allocation. However, the meaning information in “Flintstones” and “Fingerprint” is almost uniformly distributed in their respective image, and thus the effect of adaptive allocation is not so obvious. From Table I, our WA scheme has the same values of PSNR and SSIM with adaptive allocation at the target rates of 0.01 and 0.4. The reason is, in our experiment 0.01 and 0.4 are set to be minimal and maximal sampling rates respectively. According to (12) illustrated in Section V-A2, the adaptive allocation strategy degenerate to equal allocation, i.e., WA scheme.
V-B3 Comparison of visual effects
Using test image “Parrot” as an example, this section gives the visual effect of recovered images with our BCS-Net, BCS-Damp and other existing methods.
As shown in Fig. 8(a), in the case of very low sampling rate of 0.03, the proposed BCS-Net has obviously better visual effect than other approaches. We can observe significant blocking artifacts in recovered “Parrot” with traditional Damp and network-based Ista, ReconNet and I-Recon. This is because all these algorithms reconstruct “Parrot” block by block, without consideration of blocking artifacts due to block partition. We also notice that the block artifacts in BCS and our proposed BCS-Damp are not so obvious as other competing methods. The reason is that, in BCS and BCS-Damp, block-wise approximation is interleaved with full-image-based denoising, and the artifacts can then be gradually ameliorated as iterations progress. Our BCS-Net synthesizes the common merits of BCS algorithm and deep network approach, and thus achieves the best performance.
As we can see from Fig. 8(b), when the sampling rate increases to 0.3, the recovered “Parrot” with seven approaches are all improved. However, we can still observe obvious blocking artifacts for Damp and ReconNet algorithms. With I-Recon and Ista, some weak blocking artifacts are also noted. Compared with those competing approaches, our BCS-Net is capable of reconstructing more details and sharper edges, and has not more blocking artifacts.
Vi Conclusion
In this paper, we further studied the problem of block-based image compressive sensing, and proposed a multi-channel deep neural network architecture, termed ‘BCS-Net’. The proposed architecture originates from the popular block-based CS algorithm, where block-wise iterative approximation together with full-image-based denoising is key for improving the recovered image. We then cast this idea into a carefully designed deep network, so that our proposed BCS-Net is capable of benefitting both from the learning capacities of deep network and from the hand-designed structure of BCS algorithm. Extensive experimental results show that our BCS-Net with adaptive sensing resource allocation achieves far better reconstruction quality and superb visual effect compared with state-of-the-art methods. At the same time, BCS-Net with WA approach also has excellent reconstruction performance with significantly reduced number of network parameters.