Multi-Channel Deep Networks for Block-Based Image Compressive Sensing

by   Siwang Zhou, et al.

Incorporating deep neural networks in image compressive sensing (CS) receives intensive attentions recently. As deep network approaches learn the inverse mapping directly from the CS measurements, a number of models have to be trained, each of which corresponds to a sampling rate. This may potentially degrade the performance of image CS, especially when multiple sampling rates are assigned to different blocks within an image. In this paper, we develop a multi-channel deep network for block-based image CS with performance significantly exceeding the current state-of-the-art methods. The significant performance improvement of the model is attributed to block-based sampling rates allocation and model-level removal of blocking artifacts. Specifically, the image blocks with a variety of sampling rates can be reconstructed in a single model by exploiting inter-block correlation. At the same time, the initially reconstructed blocks are reassembled into a full image to remove blocking artifacts within the network by unrolling a hand-designed block-based CS algorithm. Experimental results demonstrate that the proposed method outperforms the state-of-the-art CS methods by a large margin in terms of objective metrics, PSNR, SSIM, and subjective visual quality.



page 1

page 2

page 6

page 8

page 10


ConvCSNet: A Convolutional Compressive Sensing Framework Based on Deep Learning

Compressive sensing (CS), aiming to reconstruct an image/signal from a s...

ISTA-Net++: Flexible Deep Unfolding Network for Compressive Sensing

While deep neural networks have achieved impressive success in image com...

Compressive Closeness in Networks

Distributed algorithms for network science applications are of great imp...

AMP-Net: Denoising based Deep Unfolding for Compressive Image Sensing

Most compressive sensing (CS) reconstruction methods can be divided into...

Smoothness-Constrained Image Recovery from Block-Based Random Projections

In this paper we address the problem of visual quality of images reconst...

Adaptive and Cascaded Compressive Sensing

Scene-dependent adaptive compressive sensing (CS) has been a long pursui...

Near-lossless L-infinity constrained Multi-rate Image Decompression via Deep Neural Network

Recently a number of CNN-based techniques were proposed to remove image ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Compressive sensing (CS), an emerging sampling and reconstructing strategy, can recover original signal from dramatically fewer measurements with a sub-Nyquist sampling rate [CS]. As CS has the potentials of significantly improving the sampling speed and sensor energy efficiency, it has been applied in many practical applications, including single pixel imaging [singlecamera], fast magnetic resonance imaging [MRI], high-speed video cameras [Video] and image encryption [cqli:meet:JISA19]. To deal with high-dimensional natural images efficiently, block-based CS is proposed as a lightweight CS approach [BCS, SPL, BCS-1]

. In such strategy, a scene under view is partitioned into some small blocks, which are then sampled and reconstructed independently. Meaningful information is usually not uniformly distributed in an image, so the block partition benefits more fair allocation of the sensing resources for the whole image


Although block-based CS enjoys the advantages of low-cost sampling, lightweight reconstruction, and capability of adaptively assigning sensing resources, it also usually suffers from reduced quality of image reconstruction due to blocking artifacts [BCS-1, BCS-2]. To address the issue, some methods using an iterative block-based CS algorithm (BCS) are proposed [BCS, BCS-1]. In each iteration, the projection operation is used to build an approximation of each block, while denoising operation acts on the full image reassembled by the approximative blocks. Results demonstrate that the recovered image blocks can be improved, while blocking artifacts can also be ameliorated as the iterations progress. This approach, however, may increase the reconstructing time, since small blocks still need to be concatenated into large-size full images to remove blocking artifacts.

Inspired by the powerful learning ability of deep neural networks in image representation [DNN-C, DNN-S], several network-based CS methods are proposed [Icassp, Reconnet, Ldamp, Im-recon, ISTA], which are significantly faster than the traditional CS reconstruction algorithms. Using a fully connected layer to mimic the CS sampling, the network models can jointly optimize the sampling matrix and the reconstruction process, improving the qualities of recovered images. Although the CS network models are carefully constructed to enhance learning capabilities, several specific models have to been trained with various sampling rates, ignoring the mutual relationships among them. Consequently, blocking artifacts still exist in the existing deep network methods [Reconnet, Im-recon, ISTA], especially when the employed sampling rates are very low. Moreover, most network-based image CS methods are trained as a black box, ignoring structural insight of CS reconstruction algorithms. Consequently, the reconstruction accuracy is decreased.

In this paper, we propose a multi-channel deep learning architecture for casting BCS algorithm into a learning network. It can benefit from the speed and learning capacities of deep networks while retaining the advantages of the previous BCS algorithms. To facilitate description, we term the multi-channel deep architecture as BCS-Net, which consists of a channel-specific sampling network and a unified deep reconstruction network. The channel-specific architecture is specifically designed to handle block-based allocation of sensing resources. The blocks with various sampling rates can then be fed into the same deep reconstruction network to exploit inter-block correlation for removing blocking artifacts. We further divide the reconstruction network into a fixed number of residual layers, each of which corresponds to an iteration of the BCS algorithm. To enable training, a modified version of the famous DnCNN network designed in

[DnCNN] is used to replace the denoising operation in traditional BCS approach, which easily propagates gradients.

Our contributions of the paper are summarized below:

  • A multi-channel sampling architecture specifically for block-based image CS is designed. Using this multi-channel architecture, block-wise CS measuring processed with a variety of sampling rates can be integrated into a single model to utilize the correlation among the blocks with different CS sampling rates.

  • A deep reconstruction architecture based on the BCS algorithm is proposed using block-wise approximation and full-image-based denoising.

  • Performances of the proposed approaches are verified by extensive experiments on three widely used benchmark datasets. The results show that the proposed multi-channel deep network can significantly outperform the state-of-the-art CS methods and network-based ones in terms of both subjective and objective metrics.

The remainder of this paper is organized as follows. In Sec. II, the related work on CS methods and network-based methods are reviewed. Section III introduces the idea of block-wise approximation and full-image-based denoising in BCS algorithm, and presents an extended version of the well-known Damp algorithm. The proposed multi-channel deep architecture is presented in Sec. IV and test results on its performance are given in Sec. V. The last section concludes the paper.

Ii The related work

In this section, we present the background of CS theory, then review the representative work on block-based image CS and deep network approaches.

Ii-a Preliminary of CS theory

Compressive sensing consists of two main steps: sampling/measuring process and reconstructing process. Let and denote a sparse signal of size and an measurement/sampling matrix, respectively. Then, the sampling process can be presented as


where is the

-length measurement vector sampled from

. If signal is not sparse but compressible, the sampling process has to be further deduced from Eq. (1). That is, , where represents measurement noise, denotes the sparse transform, is the coefficient vector in the corresponding transform domain, , and is considered as a sparse vector approximating to . In general, a natural image is not strictly sparse signal, but often approximately sparse in some spare transform domain.

The process of reconstructing signal needs much more computational complexity than the sampling process. It has been proven in [CS] that if the sampling matrix obeys the restricted isometry property (RIP), it is possible to recover by solving an -norm optimization problem: , subjecting to , even if . Here is a small constant, and one has when is a compressible signal.

In the past two decades, a number of CS reconstruction algorithms have been developed, including basis pursuit [BP], orthogonal matching pursuit [OMP], and the latest iteration-based Damp algorithm [Damp]. Although these algorithms enjoy solid mathematical foundations, they usually need long reconstructing time due to high computational complexity.

Ii-B Block-based image CS

Block-based CS is more effective for processing natural images because of increased dimensionality of such signals [BCS, BCS-1, BCS-Small]. The scene under view is partitioned into relatively smaller non-overlapping blocks. The measurement matrixes corresponding to the small-size blocks are observed. Then, the image is sampled and reconstructed on a block-by-block basis [BCS-1]. This block-independent approach results in a reduced computational complexity for reconstruction with a much simplified sampling process.

As shown in Fig. 1, the meaningful information in different blocks of an image is non-uniform. Depending on the volume of information contained in each block, different sampling rates are adopted to reduce the overall sampling rate. Taking the two images in Fig. 1 as an example, lower CS sampling rates can be allocated for the block marked “C” in the image “Cameraman” and block “G” in image “Parrot”.

As a consequence, sensing resources should be reasonably allocated to each block, instead of equally assigned. In [BCS-Salie], less sampling rates are allocated to non-salient image blocks but more to salient ones using the characteristics of human visual system. Concretely, a low-resolution sensor is used produce an initially sampled image, such that the adaptive sampling can be achieved for the input scene. In [adaptive], the CS procedure is initialized with a low fixed-rate pre-sampling and an initial recovery. Then, the important regions are extracted by computing the saliency map of the initially recovered image. The adaptive CS strategy is further validated on some real video sequences [adaptive-tip]. In [Asymmetric], we also propose an asymmetric approach to ensure fairer allocation of sampling rates among image blocks.

Fig. 1: Dividing two images for allocation of the sensing resources.

Due to the fact that block partition breaks the global correlation of the whole image, block-based CS prone to generate reconstructed image of low quality. In [BCS], an iterative BCS reconstruction approach is proposed to remove the blocking artifacts. The recovered images via BCS algorithm are approximated on a block-by-block basis, but hard-thresholding denoising in each iteration is imposed on the full image, not image blocks. As a result, the artifacts incurred by block partition can be smoothed as the iteration progress. However, this can result in substantially increasing reconstruction complexity because of full-image denoising, which violates the motivation of lightweight design.

Ii-C Deep network approach for image CS

The tremendous success of deep learning in computer vision as shown in

[DNN-C, DNN-S] attracted application of deep neural networks in image CS [Ldamp, Reconnet, ISTA, CSnet]. When an imaging system acquires CS measurements, the reconstructing process is performed with a deep network. Compared with the traditional CS, deep network approaches generally enjoy much faster reconstruction speed, while still achieving high-quality recovered images owing to their significant learning capabilities.

In [Reconnet], a network-based ReconNet approach is introduced to learn the inverse mapping from block-wise CS measurements to their desired image blocks. It is further improved in [Im-recon] that the measurement matrix and the reconstruction process are jointly learned. In [ISTA, Ldamp], traditional CS algorithms and deep networks are blended by treating parameters of the algorithm as weights to be learned. Unfortunately, full-image reconstruction is prone to overfitting due to the potentially overwhelming number of parameters of the sampling layer. So block-independent image recovery has to be performed instead of reconstructing the full image directly. As a consequence, blocking artifacts can be observed in the recovered images. To address the issue, the recovered images are fed into the BM3D denoiser designed in [BM3D] to remove blocking artifact [Reconnet, Im-recon]. However, the benefit of using an traditional off-the-shelf denoiser is not convincingly demonstrated.

For each sampling rate, the corresponding network model has to be trained to learn the inverse CS mapping. This may be not desirable for block-based CS, since a large number of model parameters need to be stored. In [Ldamp]

, a neural network architecture is applied to a variety of measurement matrices. Unfortunately, its performance improvement is less significant than traditional CS methods because the measurement matrices cannot participate in the network training. Inspired by the multi-scale super-resolution method given in

[Enhanced], [Multi] introduce a multi-scale CS approach, where the main network is shared across multiple sampling rates. However, their method only reuses a portion of parameters, and a CS sampling rate still correspond to a specific network model. In other words, they do not consider block partition and the corresponding problem of blocking artifacts.

Iii Block-wise approximation and full-image-based denoising

In this section, we first introduce the key idea of the popular BCS algorithm [BCS], i.e., block-wise approximation and full-image-based denoising. Then we propose an extended version of the well known Damp approach [Damp] by exploiting this idea.

Iii-a Block-wise approximation and full-image-based denoising in BCS

The BCS algorithm is an iterative reconstruction approach specialized for block-based image CS. It solves the image reconstruction problem by using approximation with projection onto the convex set and hard thresholding denoising in the iteration process, as shown in


where is the measurement matrix corresponding to a block and is its pseudo-inverse, denotes the approximation of the recovered block , and all are reassembled into a full image . Here we assume that the original image are divided into blocks, is the measurement sampled from block . The reconstruction starts from some initial approximation and forms the recovered image at iteration. In the BCS approach, represents hard thresholding, which is widely used in removing Gaussian noise. In the special case when is an orthonormal matrix, we can deduce that , where is the transpose of .

We can conclude that, the key idea of BCS algorithm is block-wise approximation, as illustrated in the first equation in (2). The denoising operation, on the contrary, is imposed on the full image, not each block, as shown in the second equation in (2). In this way, blocking artifacts can be removed while still maintaining block-wise CS sampling.

Iii-B BCS-Damp algorithm

Inspired by BCS algorithm, in this subsection, we propose an extension for Damp algorithm, called BCS-Damp, to reconstruct the images with block partition.

Damp is a state-of-the-art CS reconstruction algorithm, which is also an iterative approach like BCS. Let be the measurement matrix of image , and is the corresponding measurements. Damp algorithm takes the form


where . The part in Eq. (3) can be written as , where is the original image and can be regarded as a Gaussian noise at iteration .

is an estimate of the standard deviation of that Gaussian noise.

is the number of CS measurements. div denotes the operation of partial derivative, and div represents the divergence of the denoiser.

However, Damp is not specially designed for block-based CS, and correspondingly it does not consider blocking artifacts as a result. With the idea of BCS in mind, that is, block-wise approximation is employed to decrease computation complexity, while full-image-based denoising is imposed to remove blocking artifacts, we propose an extension of Damp for block-based image CS. Our proposed BCS-Damp algorithm is illustrated as


where represents the block in an image. Here we maintain the operation of in Damp algorithm. The modification we have to introduce is that, does not run on the blocks, but on the full image , obtained by concatenating all approximated blocks , as shown in the last equation in (4).

In the following section, block-wise approximation but full-image-based denoising, along with the iterative structure of BCS algorithm, will be further casted into a carefully designed deep network for removing artifacts and improving the recovered image.

Iv Multi-channel deep network architecture

In this section, we propose a multi-channel deep network architecture, termed BCS-Net, to reconstruct the images acquired by block-wise CS sampling. The proposed BCS-Net is composed of a multi-channel sampling network and a deep reconstructing network, as shown in Fig. 2. These two networks consist of an integrated end-to-end model, of which the learnable parameters are jointly trained by our proposed two-stage training strategy.

Fig. 2: BCS-Net architecture, which is composed of a multi-channel sampling network and a reconstruction network including an initial reconstruction part and a deep reconstruction part.

Iv-a Multi-channel sampling architecture

This subsection investigates a -channel network architecture, named , to mimic the adaptive sampling process of block-based CS, as shown in Fig. 3.

Fig. 3: -channel sampling network , with which the blocks with different sampling rates are fed into the network via their respective channels.

The proposed has channels, each of which corresponds to a sampling rate assigned to a specific image block. A higher value of indicates a more detailed division of sampling rates of image blocks, thereby block partition can benefit the fairer allocation of the sensing resources, but resulting in more complex sampling architecture. The full scene under view, , is partitioned into non-overlapping image blocks of size . Using the schematic image in Fig. 3 as an example, is set to 9. According to CS theory, for image block , the sampling process can be represented as . Here () is the measurement matrix of block , , , and is the corresponding measurements. We have , where is the sampling rate assigned to block . With our sampling network, a channel is related to a sampling rate, and channels can then correspond to a number of target sampling rates by employing linear combination of those sampling rates.

For the channel in , we use a convolutional layer, in which we do not set bias and activation, to mimic the sampling process. This convolutional layer is defined as , where denotes convolution kernels of size . In other words, corresponds to measurement matrix . Note that the measurements among different channels are different from each other, since each channel corresponds to its own convolutional kernel. However, the blocks within the same image are spatially correlated. As a consequence, the measurements of those blocks among different channels are related to each other. If is fed into channel, we have


The size of the convolution kernel depends on the sampling rate and the size of image blocks. We should note that, compared with traditional measurement matrix, the weights of are learnable. In this perspective, it is more rigorous that network-based CS approaches should be referred to as the ones inspired by CS, instead of being CS.

We note that channels in our sampling network corresponds to sampling rates, respectively, and may be less than , the total number of the blocks in an image. With our multi-channel model, the blocks have to be measured via their respective channels, and the number of the channels, , is then related to the number of blocks of an image, . Ideally, a block corresponds to a unique channel. In this case, we have . If several blocks within an image are similar to each other, such as those blocks belonging to the background, they can be considered to assign the same sampling rates. At this point, we have . Fortunately, all channels can be trained sufficiently in the training process, since we have enough training images, and thus each of these channels will receive enough training blocks regardless of the allocation strategies employed.

Obviously, if the full image is not partitioned into blocks, too many parameters may be needed to store the weights and it will be easily prone to overfitting. This may also be the main reason that most existing deep network approaches reconstruct the full image with block-by-block strategy. In this paper, we are going to further investigate the problem of blocking artifacts due to block partition.

Iv-B Deep reconstructing architecture

In this subsection, we construct a deep reconstruction architecture to cast the idea of iterative block-wise approximation but full-image-based denoising into the network, achieving model-level removal of blocking artifacts. Our deep architecture is composed of an initial reconstruction network and a deep reconstruction network , as shown in Fig. 4.

Fig. 4: Deep reconstruction architecture, in which the image is approximated by block at each residual phase, while these approximated blocks are reassembled into a full image to perform denoising by employing a modified version of DnCNN phase by phase.
Fig. 5: Deep reconstruction .

The initial reconstruction network, , has inputs, each of which corresponds to a sampling channel in as illustrated in Section IV-A. The () input is connected to the corresponding convolutional layer, which uses kernels of size and generates values by convolving them with . Here is the measurement of block entering channel. All these values are combined into one feature map, , and we refer to it as the initially reconstructed result of block . From the network’s point of view, we have


where denotes the above mentioned convolutional kernel corresponding to the channel in the sampling network. As we can see from Fig. 4, our initial reconstruction network includes only one convolutional layer for simplification reason, and the initially recovered images is going to be improved by our deep reconstruction network.

The proposed deep reconstruction network, , is further divided into phases, so that the iterative BCS algorithm can be unrolled along with these phases. In , each phase corresponds to one iteration in BCS algorithm consisting of approximation and denoising operations. At phase the block-wise approximation is implemented by using a formula slightly different from (2) in Sec. III-A, as shown in


for each block , where is the measurement matrix specialized for block , if is fed into the network via channel in our multi-channel sampling model. Note that the matrix

is learnable, and it may be not an orthogonal matrix. Thus, its pseudo-inverse,

, can not be simplified into as in the traditional BCS or Damp algorithm.

We then reassemble all approximated blocks to build a full, approximated image,

, for further denoising processing. To enable training of the deep reconstruction network, we modify the famous DnCNN network to implement full-image denoising. Traditional denoising methods, such as hard thresholding in BCS and BM3D in Damp algorithm, will not work in deep network architecture, since they cannot propagate gradients. This restricts us to focus on feed-forward convolutional neural networks. DnCNN is our choice, which fortunately offers improved performance on image deblocking and Gaussian denoising. Our deep reconstruction network is composed of

phases. Each phase has 5 convolutional layers, and the configuration is designed by referring to the DnCNN network. The first layer generates feature maps with the kernels of size , and last layer generates 1 feature map with one kernel. All three other layers employ kernels, each of which is of the size of

. It should be noted that all conventional layers explore the RELU activation function except the last layer. In DnCNN, 20 convolutional layers are employed to form a deep network for image denoising. In our network model, 5 convolutional layers form a phase, and

phases are employ to deal with both image denoising and image approximation. In our experiments, , and are set to 64, 3 and 10, respectively. Let be the parameters of convolutional kernels in phase. Then one has


where is the approximated image in the first phase in the deep network.

Iv-C Two-stage training

We propose to divide the training process into two stages to improve the recovered images by training sampling matrix while being able of utilizing in deep reconstruction process.

As illustrated in Section IV-A, in our network sampling matrixes are implemented by employing convolution operations. That is, the elements in and have to be taken from the convolution kernel in the channel in the sampling network. However, we find that, if both and participate in the training process, we cannot achieve a desired recovered image. This is because, in the training process, has to be updated in real time along with each back propagation. Unfortunately, back propagation is based on the gradient decent rule, which will be hindered due to the real-time computing of in the deep reconstruction network.

In view of this, in the first stage, we aim to obtain the training parameters of the sampling network. That is, we have to achieve optimal sampling matrices and the corresponding matrices . It is observed that, our deep architecture consists of an initial reconstruction network without including and a deep reconstruction network where is utilized to improve the recovered images. In this way, we combine the sampling network and the initial reconstruction part of our reconstruction network into a training network, i.e., , which is used to train the sampling matrix . Given the training images , the cost function is


where is the number of images in the training dataset.

In the second stage, we further train the reconstructing network consisting of an initial reconstruction part and a deep reconstruction part, i.e., , where the parameters in come from . That is, the sampling weights are fixed while the parameters are updated in the training process. Our reconstructing network

directly learns the mapping between the CS measurements and the ground truth, and the loss function minimizing the error between the input and the output is on the basis of full images instead of image blocks. Mean square error is adopted to design an end-to-end cost function


where denotes the CS measurement vector of the block in training image.

V Performance evaluation

In this section, we conduct extensive experiments to evaluate the performance of the proposed BCS-Net and BCS-Damp schemes, and compare them with state-of-the-art methods, including traditional BCS [BCS], Damp [Damp], network-based Ista [ISTA], ReconNet [Reconnet] and its improved version, I-Recon [Im-recon], in terms of reconstruction quality, time complexity and visual effect.

V-a Training and testing

V-A1 Constructing a training set

The training images are from the training set (200 images) and testing set (200 images) of the BSDS500 database [BSD500], in which we randomly crop 89600 images with the size of as the training set. Each training image, , is further partitioned into 9 image blocks of size , . That is, there are a total of 806400 blocks in our training set. Visual saliency of the scene was exploited in [saliency, BCS-Salie]. In the experiments, the methods in reference [BCS-Salie] is used to compute the saliency map of training images. Suppose that represents the amount of the saliency information embodied in image . Then we have , where is the total number of pixels on image , denotes the saliency map of , and is the saliency value of location on . Let be the saliency information of image block , and denotes the proportion of the saliency information of block . We can then construct the training data pair for our network, as shown in


where and there are a total of training image pairs.

For three existing network-based approaches, Ista, ReconNet and I-Recon, 806400 image blocks are randomly cropped. These blocks and themselves consists of 806400 training block pairs for training, since these approaches are all based on block-independent image recovery.

V-A2 Training details

We set for our -channel network model, and the sampling rates are in the range of . Each image pair is further processed in order to find out the most appropriate channels in the network. For a given target sampling rate (), we calculate sub-rate of block as


where is defined in Section V-A1, and are the sizes of an image and its block, respectively. In the training process, the space of sampling rates is divided into seven intervals, each of which corresponds to a channel. If falls within the interval corresponding to channel, then block is pushed into the network via channel .

In the experiment, we train the network with 50 epoches. The batch size is set to 1, since each image have to be partitioned into 9 blocks and these blocks are reassembled in our multi-channel network. The mean square error between the original image and the output of the network is calculated as the loss for back-gradient propagation. Adam optimization


with a learning rate of 0.0001 is adopted to optimize the parameters. We use TensorFlow 1.4

[Tensor] to train the proposed multi-channel network at a desktop platform configured with one NVIDIA 1060 GPU, one CPU @ 4.00 GHz of Intel(R) Core (TM) i7-4790K and 32GB of memory. The training processes takes about 3 hours for one epoch.

V-A3 Testing set

We test our multi-channel networks with three widely used benchmark datasets, Set5, Set11 and BSD100, where Set5 and Set11 are shown in Fig. 5 and Fig. 6, respectively.

Set5 consists of 5 gray images, where the sizes of “Bird” and “Head” are , “Baby”, “Butterfly” and “Woman” are with the size of , and , respectively. Set11 has 11 gray images, where the sizes of “Fingerprint” and “Flintstones” are , and the other 9 images are all with the size of . BSD100 includes 100 images with the size of or . These test images are with a various types of spatial distribution of key visual information. For example, the main meaningful information in images “Cameraman” and “Parrot” in Set11 is located in the single connected region. In contract, the visual information of “Bird” in Set5, “Fingerprint”, “Flintstones” and “Peppers” in Set11 uniformly distributes in the whole images. Note that all those test images are strictly separate from the training datasets.

a) b) c) d) e)
Fig. 6: Five typical images owning different spatial information distribution: a) “Baby”; b) “Bird”; c) “Butterfly”; d) “Head”; e) “Woman”.
Fig. 7: Elven images with a variety of spatial information distribution in Test dataset “Set11”.

V-B Results and analysis

V-B1 Comparisons with the state-of-the-art methods

In this subsection, we evaluate the performance of the proposed BCS-Net with adaptive allocation of sampling rate and BCS-Damp, and compare them with the existing methods.

For our BCS-Net, all test images are reprocessed to simulate the initial CS sampling by conforming to the following simulations. Each original test image is first resized to one percent of its original size, which mimics the scene under view pre-sampled by a low-resolution imaging sensor. After that, the saliency of the pre-sampled image is computed. This small-size saliency map is further normalized and bilinearly interpolated to a map with the original size. The saliency information of the original test images,

, is then estimated. As a consequence, the sampling resources can be allocated to all blocks by using (12), instead of being equally allocated.

To guarantee the average rate of the sampling rates corresponding to each channel being equal or close to the target rate, , the following procedure is employed. As with the training process, the space of sampling rates is first divided into seven intervals. The internal corresponds to channel, and is represented as , where and is the total number of channels in our multi-channel model. Then we compute the sampling rate for block by using (12), where and is the total number of blocks. If falls into the interval , the value of is changed to . Here is the sampling rate corresponding to channel and . If , then the average sampling rate of all block is exactly equal to the target sampling rate. Otherwise, we carry out fine tune by changing some blocks to the higher or lower channel according to the positive or negative difference, i.e., . We should note that the average sampling rates can be very close to the target rates, and thus we ignore the differences between them in the experiments.

The average PSNR (peak signal-to-noise ratios) and SSIM (structural similarity index) with BCS-Net are reported in Table I. The comparison of running times of reconstructing the images in Set5 and Set11 is shown in Table II. Here the running times are the average values of all 16 test images in Set5 and Set11 with the sampling rate of 0.1. We should note that, 0.1 is the target sampling rate of an image, and our running time contains the time of reconstructing all blocks with different channels in the multi-channel architecture.

[width=10em]AlgorithmSampling rate 0.01 0.03 0.05 0.1 0.2 0.3 0.4
BCS [BCS] 16.20/0.3613 20.80/0.5088 22.54/0.5974 24.87/0.7280 28.50/0.8130 30.54/0.8539 32.32/0.8899
Damp [Damp] 6.51/0.0311 19.95/0.4821 21.56/0.5667 24.24/0.6997 28.49/0.8527 32.29/0.9137 34.25/0.9395
BCS-Damp 19.97/0.4827 22.28/0.6267 24.10/0.6962 27.95/0.8247 32.47/0.9105 36.46/0.9397 39.53/0.9565
ReconNet [Reconnet] 18.46/0.4492 21.54/0.5699 23.33/0.6462 25.70/0.7422 28.16/0.8197 30.03/0.8620 31.00/0.8793
I-Recon [Im-recon] 21.49/0.5571 25.00/0.7113 26.97/0.7908 28.49/0.8329 30.53/0.8823 34.51/0.9403 35.30/0.9465
Ista [ISTA] 18.06/0.4589 21.50/0.5624 25.15/0.7307 28.89/0.8405 33.21/0.9152 36.00/0.9456 38.14/0.9622
BCS-Net (WA) 22.98/0.6103 26.69/0.7702 28.72/0.8371 31.86/0.9034 35.43/0.9488 37.87/0.9681 39.88/0.9785
BCS-Net 22.98/0.6103 27.09/0.7699 28.81/0.8237 32.71/0.9030 36.12/0.9483 38.64/0.9694 39.88/0.9785
[width=10em]AlgorithmSampling rate 0.01 0.03 0.05 0.1 0.2 0.3 0.4
BCS [BCS] 15.65/0.3973 19.40/0.5146 20.89/0.5768 23.15/0.6836 25.74/0.7778 27.78/0.8331 29.75/0.8783
Damp [Damp] 5.49/0.0582 18.47/0.4986 20.14/0.5588 22.73/0.6873 26.85/0.8335 30.09/0.8994 32.93/0.9339
BCS-Damp 18.53/0.4692 21.03/0.5795 22.61/0.6471 26.20/0.7919 30.75/0.8976 34.21/0.9415 37.09/0.9648
ReconNet [Reconnet] 16.99/0.4145 19.80/0.5110 21.14/0.5935 23.28/0.6896 25.54/0.7719 27.11/0.8155 28.32/0.8411
I-Recon [Im-recon] 19.80/0.5018 22.59/0.6540 24.54/0.7442 25.97/0.7888 27.92/0.8457 31.45/0.9135 32.26/0.9243
Ista [ISTA] 16.55/0.4139 19.74/0.5154 22.83/0.6792 26.49/0.8010 30.79/0.8950 33.76/0.9345 36.03/0.9547
BCS-Net (WA) 20.88/0.5505 24.05/0.7048 25.89/0.7851 28.63/0.8628 32.08/0.9220 34.65/0.9506 36.70/0.9662
BCS-Net 20.88/0.5505 24.47/0.7807 26.04/0.7723 29.43/0.8676 33.06/0.9283 35.60/0.9554 36.70/0.9662
[width=10em]AlgorithmSampling rate 0.01 0.03 0.05 0.1 0.2 0.3 0.4
BCS [BCS] 18.53/0.4213 20.93/0.4797 21.54/0.4996 23.21/0.5843 25.04/0.6742 26.48/0.7384 27.75/0.7875
Damp [Damp] 7.00/0.0495 19.57/0.4260 20.54/0.4706 21.92/0.5333 24.29/0.6286 26.00/0.6970 27.57/0.7541
BCS-Damp 19.90/0.4255 21.58/0.4896 22.47/0.5216 24.10/0.5842 26.48/0.6745 28.79/0.7490 31.44/0.8184
ReconNet [Reconnet] 18.74/0.3960 20.56/0.4647 21.52/0.5120 23.00/0.5837 24.68/0.6694 25.83/0.7204 26.71/0.7611
I-Recon [Im-recon] 21.15/0.4654 23.09/0.5662 24.19/0.6320 25.35/0.7098 26.87/0.7872 29.24/0.8593 30.07/0.8812
Ista [ISTA] 17.86/0.3957 20.47/0.4699 22.65/0.5692 24.79/0.6726 27.64/0.7906 29.86/0.8580 31.70/0.9003
BCS-Net (WA) 22.03/0.4997 24.10/0.6085 25.16/0.6709 26.97/0.7651 29.52/0.8609 31.59/0.9107 33.44/0.9405
BCS-Net 22.03/0.4997 24.43/0.6108 25.58/0.6669 27.84/0.7709 30.59/0.8672 32.64/0.9160 33.44/0.9405
TABLE I: The average PSNR in dB and SSIM on Set5, Set11 and BSD100 with a range of sampling rates.

The best performance is labeled in bold, the second best is italic, and the second best is underlined.

Algorithm BCS Damp BCS-Damp ReconNet I-Recon Ista BCS-Net(WA) BCS-Net
Time 100s 228s 611s 0.87s 0.89s 1.05s 2.02s
TABLE II: The average time for reconstructing the images in Set5 and Set11 with the sampling rate of 0.1 (in second).

As shown in Table I, our BCS-Net yields higher-quality recovered image in terms of both PSNR and SSIM than other existing methods, including BCS, Damp, ReconNet, I-Recon, and Ista, for Set5, Set11, and BSD100, respectively. From Table I, we can easily observe a significant performance improvement of, for instance, 3.82 dB and 2.98 dB on Set5 and Set11 with the sampling rate of 0.1, and 2.95 dB on BSD 100 with the sampling rate of 0.2, respectively. We notice that, along with the increase of the sampling rate, the improvement of our scheme slows down. The possible cause is that, when the sampling rate is as high as 0.1 or 0.2, the recovered images for the competing approach are of relatively high quality. And there is not too much space for improvement with even more sampling rates.

We can also see from Table I that, in existing methods I-Recon has relatively better reconstruction quality at extremely low sampling rates of 001, 0.03 and 0.05, while Ista usually performs better at the sampling rates of 0.2 and above. However, our BCS-Net always outperforms the traditional BCS and Damp algorithms, as well as the network-based ReconNet, I-Recon and Ista algorithms. We think that, the performance improvement of our scheme is mainly due to the following two factors, i.e., adaptive allocation of sampling rate in our multi-channel sampling network, and block-wise approximation and full-image denoising in our deep reconstruction network. We notice that BCS-Net has slightly longer running time than network-based ReconNet, I-Recon and Ista because of our multi-channel sampling and block reassembling, but it runs significantly far faster than optimization-based BCS and Damp reconstruction algorithm.

The proposed BCS-Damp approach also outperforms Damp and BCS algorithm, as shown in Table I. This is because, compared with Damp algorithm, in our BCS-Damp, BM3D denoising is imposed on the full image instead of each block, and blocking artifacts can then ameliorated. And compared with BCS approach, our BCS-Damp has better denoising performance, since BM3D denoising outperforms the hard thresholding employed in BCS approach. We notice that the proposed BCS-Damp even outperforms BCS-Net in terms of PSNR for Set11 at very high sampling rate of 0.4. We think this is another indication that network-based approaches generally offer more advantages over relatively lower sampling rates. Note that our BCS-Damp has much longer running time than BCS and Damp due to our full-image denoising strategy in the iteration process and relatively higher computation complexity of BM3D denoising algorithm.

Images SR BCS Damp BCS-Damp ReconNet I-Recon Ista BCS-Net BCS-Net(WA)
Baby 0.01 18.20/0.4838 4.46/0.0288 24.29/0.6615 21.19/0.5582 24.22/0.6430 20.87/0.5778 26.35/0.6988 26.35/0.6988
0.03 24.60/0.7020 23.37/0.6554 25.50/0.7403 24.24/0.6482 27.88/0.7614 23.67/0.6347 30.66/0.8274 29.59/0.8013
0.05 26.10/0.7646 25.30/0.7265 27.34/0.7850 25.77/0.7084 29.80/0.8294 27.68/0.7671 32.07/0.8689 31.27/0.8493
0.1 28.81/0.8684 27.37/0.8034 29.90/0.8746 28.03/0.7749 30.86/0.8605 30.23/0.8381 34.72/0.9137 33.63/0.9075
0.2 31.57/0.9328 30.10/0.8968 33.73/0.9502 30.56/0.8459 32.58/0.8997 33.25/0.9055 37.74/0.9579 36.52/0.9525
0.3 33.31/0.9583 32.95/0.9508 37.38/0.9760 31.93/0.8770 36.12/0.9451 35.52/0.9410 39.94/0.9755 38.59/0.9715
0.4 34.89/0.9730 34.91/0.9732 40.56/0.9872 32.84/0.8911 36.65/0.9461 37.41/0.9611 40.52/0.9822 40.52/0.9822
Bird 0.01 17.90/0.3885 8.33/0.0225 19.98/0.4757 19.38/0.4609 22.14/0.5726 19.00/0.4628 23.35/0.6391 23.35/0.6391
0.03 20.72/0.4550 20.87/0.4850 22.35/0.6164 22.25/0.6085 26.07/0.7543 22.49/0.5930 27.60/0.7871 27.98/0.8142
0.05 23.70/0.6481 21.67/0.5280 24.43/0.7029 24.32/0.6827 28.22/0.8325 26.32/0.7657 28.96/0.8286 30.55/0.8832
0.1 26.67/0.7584 25.27/0.7030 29.40/0.8570 27.02/0.7964 30.25/0.8942 31.20/0.8852 34.88/0.9369 34.40/0.9454
0.2 30.41/0.8583 31.50/0.8963 37.51/0.9644 29.71/0.8740 32.70/0.9350 36.83/0.9576 39.60/0.9770 39.20/0.9790
0.3 33.22/0.9085 36.89/0.9623 42.26/0.9862 32.12/0.9148 37.65/0.9745 40.50/0.9785 42.69/0.9889 42.29/0.9895
0.4 35.50/0.9362 37.21/0.9753 45.56/0.9928 33.22/0.9308 38.62/0.9788 43.10/0.9867 44.67/0.9937 44.67/0.9937
Butterfly 0.01 11.98/0.2601 5.23/0.0006 13.08/0.3317 13.36/0.2766 15.47/0.3802 12.97/0.2851 15.68/0.4078 15.68/0.4078
0.03 14.88/0.3615 13.10/0.3215 15.28/0.4779 15.77/0.4041 19.36/0.6211 15.71/0.4032 21.01/0.6988 20.62/0.7046
0.05 16.32/0.4330 14.98/0.4477 17.41/0.5913 17.52/0.5319 21.58/0.7326 19.31/0.6666 23.56/0.7947 23.55/0.8211
0.1 18.71/0.5545 17.65/0.6308 22.21/0.7999 20.11/0.6581 22.78/0.7511 24.56/0.8479 27.94/0.8968 27.46/0.9015
0.2 21.44/0.6622 22.80/0.8348 26.84/0.8994 22.74/0.7596 24.85/0.8200 30.17/0.9355 30.96/0.9386 31.24/0.9504
0.3 23.50/0.7233 26.46/0.9052 29.99/0.9375 24.80/0.8227 29.54/0.9286 33.66/0.9604 33.98/0.9680 33.67/0.9698
0.4 25.26/0.7698 29.77/0.9387 34.24/0.9648 25.81/0.8401 30.34/0.9362 36.50/0.9741 35.60/0.9791 35.60/0.9791
Head 0.01 17.60/0.3134 9.44/0.0964 24.62/0.4831 21.33/0.5025 25.30/0.6133 21.08/0.5199 28.19/0.6603 28.19/0.6603
0.03 24.79/0.5102 24.42/0.4616 27.37/0.6304 25.09/0.6025 28.21/0.6939 25.54/0.6128 30.73/0.7368 30.54/0.7351
0.05 25.97/0.5512 26.65/0.5626 28.62/0.6692 26.97/0.6408 29.71/0.7469 19.31/0.6831 31.96/0.7813 31.74/0.7792
0.1 29.86/0.7232 28.50/0.6380 30.31/0.7282 28.90/0.7115 30.88/0.8091 30.82/0.7561 33.89/0.8426 33.60/0.8400
0.2 31.74/0.7876 31.17/0.7610 32.37/0.7926 30.41/0.7694 32.46/0.8574 33.34/0.8352 35.99/0.8996 35.81/0.8977
0.3 33.10/0.8320 32.67/0.8036 33.50/0.8238 32.08/0.8054 35.51/0.8975 35.07/0.8819 37.71/0.9311 37.43/0.9285
0.4 34.27/0.8650 33.92/0.8380 34.55/0.8513 32.53/0.8276 36.02/0.9100 36.38/0.9117 38.85/0.9485 38.85/0.9485
Woman 0.01 15.29/0.3608 5.10/0.0074 17.89/0.4613 17.06/0.4475 20.30/0.5762 16.41/0.4490 21.30/0.6457 21.30/0.6457
0.03 19.01/0.5153 18.01/0.4867 20.92/0.6684 20.32/0.5861 23.47/0.7258 20.08/0.5684 25.44/0.7995 24.71/0.7958
0.05 20.63/0.5902 19.19/0.5684 22.72/0.7325 22.06/0.6674 25.55/0.8125 23.97/0.7711 27.50/0.8450 26.48/0.8529
0.1 24.13/0.7354 22.41/0.7235 27.95/0.8640 24.45/0.7702 27.67/0.8497 27.62/0.8753 32.12/0.9247 30.22/0.9226
0.2 27.36/0.8242 26.88/0.8747 31.88/0.9456 27.36/0.8497 30.06/0.8994 32.46/0.9421 36.30/0.9684 34.37/0.9645
0.3 29.55/0.8724 32.50/0.9465 39.19/0.9751 29.23/0.8898 33.77/0.9557 35.24/0.9661 38.90/0.9835 37.38/0.9811
0.4 31.64/0.9053 35.46/0.9722 42.74/0.9865 30.59/0.9068 34.90/0.9615 37.32/0.9773 39.74/0.9888 39.74/0.9888
TABLE III: Detailed comparison of PSNR in dB and SSIM in the range of [0, 1] on the images of Set5.
Images SR BCS Damp BCS-Damp ReconNet I-Recon Ista BCS-Net BCS-Net(WA)
House 0.01 17.70/0.5555 5.01/0.0704 21.76/0.6336 19.10/0.5401 22.16/0.6125 19.30/0.5487 23.65/0.6767 23.65/0.6767
0.03 22.04/0.6457 20.59/0.6651 25.74/0.7618 22.47/0.6046 25.51/0.7173 22.48/0.6034 29.76/0.8128 28.28/0.7914
0.05 23.70/0.6766 24.38/0.7092 28.53/0.8049 24.00/0.6678 27.70/0.7814 26.68/0.7607 31.21/0.8413 30.29/0.8281
0.1 26.36/0.7372 27.25/0.7951 32.45/0.8525 26.55/0.7398 29.06/0.8016 30.55/0.8366 33.62/0.8795 32.75/0.8734
0.2 29.72/0.8104 32.12/0.8540 35.48/0.8828 28.90/0.7927 30.96/0.8430 35.10/0.8921 37.30/0.9183 36.23/0.9134
0.3 31.49/0.8390 35.05/0.8833 38.12/0.9326 30.40/0.8220 34.42/0.8984 37.17/0.9198 38.62/0.9386 38.16/0.9360
0.4 33.92/0.8921 37.08/0.9177 40.15/0.9532 31.57/0.8379 35.24/0.9057 38.67/0.9400 39.71/0.9557 39.71/0.9557
Cameraman 0.01 16.56/0.4689 6.33/0.1057 17.98/0.4725 17.01/0.4502 19.65/0.5543 16.33/0.4624 20.38/0.6148 20.38/0.6148
0.03 18.70/0.5384 17.64/0.5119 20.52/0.6318 19.12/0.5225 21.65/0.6490 19.44/0.5151 23.56/0.7430 22.64/0.7117
0.05 20.05/0.5863 18.95/0.5920 21.68/0.6760 20.08/0.5774 23.00/0.7143 20.82/0.6598 25.43/0.7935 23.82/0.7666
0.1 21.71/0.6478 20.92/0.6640 24.44/0.7635 21.76/0.6553 23.81/0.7380 23.50/0.7559 28.02/0.8612 26.06/0.8376
0.2 24.30/0.7380 23.67/0.7711 28.54/0.8463 23.27/0.7194 25.43/0.7948 27.46/0.8556 30.74/0.9142 29.05/0.9004
0.3 25.97/0.7880 26.33/0.8266 30.97/0.8805 24.60/0.7618 28.12/0.8739 30.20/0.9025 32.44/0.9392 31.11/0.9285
0.4 27.90/0.8334 27.86/0.8570 34.50/0.9435 25.53/0.7853 29.03/0.8862 32.25/0.9274 32.62/0.9463 32.62/0.9463
Parrot 0.01 17.58/0.5604 5.85/0.0690 20.09/0.6576 17.26/0.5491 20.75/0.6509 16.75/0.5325 22.23/0.7164 22.23/0.7164
0.03 20.46/0.6447 20.16/0.6586 22.14/0.7185 20.53/0.6197 22.79/0.7377 20.41/0.6269 25.35/0.8229 24.23/0.8014
0.05 22.30/0.7160 21.23/0.6935 23.45/0.7705 21.68/0.6868 24.53/0.8031 23.32/0.7585 27.87/0.8638 25.50/0.8405
0.1 24.05/0.7771 22.83/0.7693 28.12/0.8563 23.52/0.7562 26.03/0.8293 26.72/0.8499 31.14/0.9126 28.39/0.8957
0.2 26.21/0.8375 25.60/0.8418 32.90/0.9073 25.84/0.8171 28.00/0.8731 29.54/0.9101 34.36/0.9496 31.67/0.9389
0.3 28.18/0.8754 28.19/0.8878 35.55/0.9351 27.20/0.8453 31.35/0.9286 32.92/0.9398 36.90/0.9668 34.71/0.9602
0.4 30.00/0.9057 30.82/0.9169 38.17/0.9574 28.38/0.8648 31.85/0.9344 35.41/0.9579 37.23/0.9736 37.23/0.9736
Peppers 0.01 14.53/0.4051 5.78/0.0621 17.37/0.5129 16.57/0.4309 19.20/0.5369 15.94/0.4263 20.67/0.6088 20.67/0.6088
0.03 17.96/0.4620 18.58/0.5333 19.93/0.6371 19.36/0.5086 22.28/0.6969 19.47/0.5328 23.69/0.7542 23.90/0.7607
0.05 20.99/0.5872 19.87/0.5936 21.58/0.6839 20.63/0.6023 24.15/0.7776 22.49/0.6901 24.87/0.7996 25.46/0.8254
0.1 23.76/0.6694 22.60/0.7055 25.97/0.7939 22.91/0.7002 25.26/0.7986 27.28/0.8170 28.35/0.8897 28.30/0.8868
0.2 24.86/0.6751 28.10/0.8371 31.10/0.8854 25.02/0.7691 26.91/0.8459 32.52/0.9075 31.99/0.9305 32.08/0.9337
0.3 26.83/0.7374 31.01/0.8925 34.85/0.9262 26.73/0.8167 31.33/0.9141 35.45/0.9383 34.59/0.9506 34.38/0.9502
0.4 29.88/0.8303 34.02/0.9213 37.08/0.9435 28.00/0.8349 31.27/0.9214 37.47/0.9537 36.10/0.9617 36.10/0.9617
Boats 0.01 16.46/0.3843 5.46/0.0593 19.42/0.4605 18.29/0.4089 20.85/0.4837 17.54/0.4058 22.05/0.5402 22.05/0.5402
0.03 20.66/0.4870 17.82/0.4902 21.55/0.5623 20.97/0.5073 23.95/0.6375 20.95/0.5030 25.67/0.7043 25.27/0.6946
0.05 21.31/0.5552 20.78/0.5396 23.21/0.6315 22.48/0.5823 25.80/0.7234 23.93/0.6708 27.54/0.7777 27.27/0.7782
0.1 22.65/0.6355 23.46/0.6514 26.27/0.7502 24.55/0.6790 27.55/0.7891 27.46/0.7966 30.39/0.8679 30.01/0.8672
0.2 24.54/0.7426 27.76/0.8147 31.41/0.8873 26.93/0.7698 29.60/0.8565 31.94/0.9057 33.67/0.9244 33.49/0.9326
0.3 28.59/0.8024 31.11/0.8973 35.87/0.9469 28.61/0.8256 33.04/0.9234 35.27/0.9479 36.14/0.9546 35.69/0.9550
0.4 30.49/0.8506 35.55/0.9471 39.14/0.9702 29.76/0.8509 34.02/0.9335 37.75/0.9668 37.63/0.9686 37.63/0.9686
TABLE IV: Detailed comparison of PSNR in dB and SSIM in the range of [0, 1] on the images of Set11.
Images SR BCS Damp BCS-Damp ReconNet I-Recon Ista BCS-Net BCS-Net(WA)
Monarch 0.01 13.38/0.3399 6.25/0.0462 14.94/0.3961 14.69/0.3612 17.43/0.4803 14.35/0.3708 17.82/0.4981 17.82/0.4981
0.03 16.68/0.4536 15.51/0.4449 17.77/0.5305 17.92/0.5005 21.11/0.6781 17.40/0.4917 23.30/0.7287 22.72/0.7409
0.05 18.43/0.5281 17.05/0.5321 20.01/0.6324 19.06/0.5875 23.47/0.7758 21.13/0.6930 24.65/0.7718 25.54/0.8295
0.1 21.08/0.6369 20.09/0.6519 24.07/0.7821 21.82/0.7020 24.61/0.8016 25.53/0.8366 29.32/0.8937 28.86/0.9065
0.2 24.14/0.7480 24.87/0.8413 28.76/0.9005 24.45/0.7960 26.67/0.8620 30.98/0.9335 33.23/0.9527 32.77/0.9563
0.3 26.79/0.8144 28.49/0.9181 32.79/0.9499 26.24/0.8468 31.03/0.9396 34.87/0.9640 36.11/0.9743 35.30/0.9742
0.4 28.49/0.8527 31.76/0.9519 37.98/0.9750 27.54/0.8685 31.91/0.9481 37.73/0.9766 37.35/0.9826 37.35/0.9826
Foreman 0.01 18.06/0.5834 4.92/0.1082 23.52/0.7136 19.71/0.5991 23.68/0.6997 19.00/0.5895 26.22/0.7565 26.22/0.7565
0.03 24.22/0.7102 23.81/0.7271 28.49/0.8314 23.77/0.6807 27.15/0.8056 23.5/0.6740 30.68/0.8442 30.70/0.8579
0.05 25.95/0.7359 26.06/0.7846 30.96/0.8664 25.73/0.7466 29.33/0.8534 28.49/0.8272 32.35/0.8752 32.27/0.8886
0.1 29.04/0.8149 30.64/0.8518 33.78/0.9066 27.79/0.8039 29.90/0.8549 33.18/0.8995 35.85/0.9325 35.18/0.9293
0.2 32.01/0.8674 34.74/0.9193 36.62/0.9325 30.03/0.8551 31.94/0.8892 37.75/0.9456 39.00/0.9619 38.29/0.9608
0.3 34.48/0.9019 37.28/0.9396 38.80/0.9484 31.81/0.8802 35.96/0.9421 40.32/0.9647 40.99/0.9753 40.23/0.9738
0.4 36.40/0.9253 38.60/0.9497 41.24/0.9649 32.73/0.8887 36.20/0.9425 42.33/0.9757 41.77/0.9811 41.77/0.9811
Barbara 0.01 16.89/0.3458 5.66/0.0402 19.38/0.4291 18.17/0.3727 20.82/0.4607 17.51/0.3665 21.75/0.4959 21.75/0.4959
0.03 20.66/0.4694 19.53/0.4380 20.90/0.4804 20.43/0.4642 22.61/0.5813 20.32/0.4681 23.24/0.5996 23.32/0.6216
0.05 21.31/0.5193 20.86/0.4716 21.71/0.5128 21.13/0.5157 23.40/0.6388 21.80/0.5661 23.64/0.6380 23.75/0.6570
0.1 22.65/0.5961 22.36/0.5683 23.52/0.6199 22.20/0.5859 23.88/0.6915 23.53/0.6842 24.73/0.7283 24.50/0.7270
0.2 24.53/0.7091 25.12/0.7253 29.26/0.8764 23.08/0.6525 24.75/0.7571 26.76/0.8331 29.29/0.8805 26.64/0.8279
0.3 26.06/0.7814 29.76/0.8777 35.40/0.9553 23.87/0.7019 27.99/0.8616 30.76/0.9255 33.03/0.9479 30.78/0.9225
0.4 27.47/0.8356 34.29/0.9527 38.03/0.9735 25.14/0.7598 29.56/0.8986 34.09/0.9606 34.23/0.9637 34.23/0.9637
Lena 0.01 14.79/0.3779 5.83/0.0700 19.35/0.5126 17.81/0.4427 21.02/0.5437 17.22/0.4371 22.00/0.5956 22.00/0.5956
0.03 20.55/0.5481 19.66/0.5570 22.34/0.6565 20.94/0.5322 23.89/0.6788 20.91/0.5453 25.88/0.7504 25.64/0.7461
0.05 22.41/0.6115 20.96/0.6153 23.80/0.7082 22.52/0.6285 25.85/0.7614 24.69/0.7265 26.11/0.7824 27.15/0.8047
0.1 24.20/0.6803 24.19/0.7137 26.32/0.7794 24.45/0.7079 26.86/0.7957 27.69/0.8227 30.62/0.8895 29.36/0.8737
0.2 26.43/0.7736 27.05/0.8172 30.93/0.8895 26.55/0.7825 28.82/0.8581 31.37/0.9067 34.36/0.9481 32.29/0.9316
0.3 28.87/0.8301 29.01/0.8677 34.82/0.9432 27.99/0.8235 32.18/0.9261 33.83/0.9419 36.71/0.9686 35.07/0.9612
0.4 30.44/0.8680 31.31/0.9063 37.49/0.9655 29.24/0.8521 33.13/0.9372 36.25/0.9622 37.30/0.9755 37.30/0.9755
Flintstones 0.01 12.23/0.2009 4.33/0.0060 14.50/0.2814 13.59/0.2421 16.13/0.3335 13.45/0.2500 16.65/0.3800 16.65/0.3800
0.03 14.96/0.3500 14.48/0.3242 16.04/0.4268 15.87/0.3365 18.72/0.4968 15.92/0.3458 19.74/0.5683 19.49/0.5664
0.05 15.07/0.3575 15.56/0.4251 17.21/0.5738 17.32/0.4397 20.51/0.6134 18.99/0.5629 22.61/0.6708 21.38/0.6751
0.1 18.22/0.6299 17.64/0.6448 21.62/0.8325 19.65/0.5663 22.63/0.6929 23.47/0.7489 25.29/0.8044 25.03/0.8065
0.2 20.99/0.7820 22.87/0.8816 28.20/0.9540 22.74/0.7061 25.37/0.7852 28.56/0.8586 29.37/0.8792 29.23/0.8876
0.3 23.26/0.8617 27.51/0.9512 30.47/0.9701 24.59/0.7637 29.00/0.8758 30.86/0.8920 31.63/0.9127 31.26/0.9141
0.4 25.18/0.9040 29.92/0.9690 31.80/0.9761 25.87/0.7915 29.73/0.8862 32.31/0.9142 32.60/0.9300 32.60/0.9300
Fingerprint 0.01 13.96/0.1487 4.93/0.0027 15.48/0.0913 14.74/0.1625 16.08/0.1634 14.64/0.1631 16.29/0.1722 16.29/0.1722
0.03 16.78/0.3512 15.37/0.1338 15.89/0.1378 16.39/0.3438 18.81/0.5152 16.38/0.3629 18.30/0.4676 18.32/0.4686
0.05 17.55/0.4712 15.89/0.1905 16.61/0.2580 17.87/0.4935 22.23/0.7433 18.83/0.5556 21.18/0.6812 22.39/0.7430
0.1 19.68/0.6941 18.05/0.5441 21.64/0.7736 20.93/0.6896 26.04/0.8843 22.50/0.7626 26.36/0.8847 26.49/0.8872
0.2 22.77/0.8718 23.46/0.8654 25.01/0.9111 24.18/0.8308 28.68/0.9383 26.76/0.8903 30.35/0.9513 31.10/0.9590
0.3 25.06/0.9329 27.28/0.9515 28.72/0.9687 26.13/0.8829 31.55/0.9644 29.69/0.9426 34.46/0.9806 34.47/0.9806
0.4 27.08/0.9633 31.03/0.9836 32.39/0.9900 27.77/0.9180 32.88/0.9738 32.10/0.9663 37.19/0.9894 37.19/0.9894
TABLE V: Detailed comparison of PSNR in dB and SSIM in the range of [0, 1] on the images of Set11.

V-B2 Performance evaluation without assigning sampling resources

In this subsection, we evaluate the performance of our BCS-Net scheme without assigning sampling rate (WA), since the sensing resources may not be allocated in certain scenarios. That is, all blocks in an image are assigned the same sampling rate, and accordingly, they are fed into our model from the same channel corresponding to the target rate. In this case, our multi-channel architecture becomes a unified deep network for target sampling rates. In the simulation, is set to 7, and these seven channels are in a range of . The comparison of average PSNR and SSIM of recovered Set5, Set11 and BSD100 can be observed in Table I. The detailed comparison of PSNR and SSIM for Set5 and Set11 is illustrated in Table III and IV, where we omit the results of testing set BSD100 due to limited space.

We should note that, our multi-channel model decreases large amount of storage requirements, since we use a unified deep reconstruction network to serve all sampling rates. For instance, Ista approach for seven different sampling rates has about 0.34 million (M) parameters each, totaling 2.4M, while our reconstruction network has only 1.1M parameters.

Table I shows that, the proposed WA scheme still achieves higher PSNR and SSIM than the best results of the existing BCS, Damp, ReconNet, I-Recon and Ista, thanks to the strategy of block-wise approximation and full-image based denoising. We can see from Table I that, our WA scheme always has lower PSNR than the proposed BCS-Net with adaptive allocation at the sampling rates of for Set5, Set11 and BSD100. We also notice that, for Set5, our WA scheme achieves slightly better SSIM than BCS-Net with adaptive allocation. From Table III, IV, the quality gap between BCS-Net and BCS-Net (WA) increases in some images, but decreases in other images. For example, the gap of recovered “Cameraman” and “Parrot” is about 28.02-26.06=1.96(dB) and 31.14-28.39=2.59(dB), but it drops to 25.29-25.03=0.26(dB) and 26.36-24.49=-0.13(dB) for “Flintstones” and “Fingerprint” with sampling rate of 0.1, respectively. This is because, the foreground objects of “Cameraman” and “Parrot” can be clearly distinguished from background information. In other words, the main meaning information in these images is limited to some local regions. As a consequence, the average assignment of sampling rate is obviously less efficient than adaptive allocation. However, the meaning information in “Flintstones” and “Fingerprint” is almost uniformly distributed in their respective image, and thus the effect of adaptive allocation is not so obvious. From Table I, our WA scheme has the same values of PSNR and SSIM with adaptive allocation at the target rates of 0.01 and 0.4. The reason is, in our experiment 0.01 and 0.4 are set to be minimal and maximal sampling rates respectively. According to (12) illustrated in Section V-A2, the adaptive allocation strategy degenerate to equal allocation, i.e., WA scheme.

Fig. 8: Comparison of visual effect of the recovered “Parrot” with different sampling rates: a) 0.03; b) 0.3.

V-B3 Comparison of visual effects

Using test image “Parrot” as an example, this section gives the visual effect of recovered images with our BCS-Net, BCS-Damp and other existing methods.

As shown in Fig. 8(a), in the case of very low sampling rate of 0.03, the proposed BCS-Net has obviously better visual effect than other approaches. We can observe significant blocking artifacts in recovered “Parrot” with traditional Damp and network-based Ista, ReconNet and I-Recon. This is because all these algorithms reconstruct “Parrot” block by block, without consideration of blocking artifacts due to block partition. We also notice that the block artifacts in BCS and our proposed BCS-Damp are not so obvious as other competing methods. The reason is that, in BCS and BCS-Damp, block-wise approximation is interleaved with full-image-based denoising, and the artifacts can then be gradually ameliorated as iterations progress. Our BCS-Net synthesizes the common merits of BCS algorithm and deep network approach, and thus achieves the best performance.

As we can see from Fig. 8(b), when the sampling rate increases to 0.3, the recovered “Parrot” with seven approaches are all improved. However, we can still observe obvious blocking artifacts for Damp and ReconNet algorithms. With I-Recon and Ista, some weak blocking artifacts are also noted. Compared with those competing approaches, our BCS-Net is capable of reconstructing more details and sharper edges, and has not more blocking artifacts.

Vi Conclusion

In this paper, we further studied the problem of block-based image compressive sensing, and proposed a multi-channel deep neural network architecture, termed ‘BCS-Net’. The proposed architecture originates from the popular block-based CS algorithm, where block-wise iterative approximation together with full-image-based denoising is key for improving the recovered image. We then cast this idea into a carefully designed deep network, so that our proposed BCS-Net is capable of benefitting both from the learning capacities of deep network and from the hand-designed structure of BCS algorithm. Extensive experimental results show that our BCS-Net with adaptive sensing resource allocation achieves far better reconstruction quality and superb visual effect compared with state-of-the-art methods. At the same time, BCS-Net with WA approach also has excellent reconstruction performance with significantly reduced number of network parameters.