ConvCSNet: A Convolutional Compressive Sensing Framework Based on Deep Learning

01/31/2018 ∙ by Xiaotong Lu, et al. ∙ Xidian University 0

Compressive sensing (CS), aiming to reconstruct an image/signal from a small set of random measurements has attracted considerable attentions in recent years. Due to the high dimensionality of images, previous CS methods mainly work on image blocks to avoid the huge requirements of memory and computation, i.e., image blocks are measured with Gaussian random matrices, and the whole images are recovered from the reconstructed image blocks. Though efficient, such methods suffer from serious blocking artifacts. In this paper, we propose a convolutional CS framework that senses the whole image using a set of convolutional filters. Instead of reconstructing individual blocks, the whole image is reconstructed from the linear convolutional measurements. Specifically, the convolutional CS is implemented based on a convolutional neural network (CNN), which performs both the convolutional CS and nonlinear reconstruction. Through end-to-end training, the sensing filters and the reconstruction network can be jointly optimized. To facilitate the design of the CS reconstruction network, a novel two-branch CNN inspired from a sparsity-based CS reconstruction model is developed. Experimental results show that the proposed method substantially outperforms previous state-of-the-art CS methods in term of both PSNR and visual quality.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 4

page 6

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Compressive sensing (CS) is a new image/signal acquisition framework acquiring only a few linear measurements [1, 2] with random measurement matrices. Such framework has the potentials of significantly improving the imaging speed and sensor energy efficiency in real applications. Based on CS, several new imaging systems, including single-pixel camera [3], compressive spectral imaging system [4], high-speed video camera [5], and fast Magnetic Resonance Imaging (MRI) system [6]

have been developed. Perfect reconstruction from the linear measurements is guaranteed if the following two conditions are met, i.e., the sensing matrices have the restricted isometry property (RIP) and the images have sparse representation with respect to a dictionary. These two conditions can be easily satisfied as random Gaussian matrices have the RIP with high probabilities

[1, 2] and natural images have sparse representation under many off-the-shelf or learned dictionaries (e.g., wavelets, learned dictionaries [7]

). However, in practice the promise of CS is often offset by challenges relating to two conditions. First, it is difficult or even impossible to implement the random sensing matrix for a large image. Second, the sparsity-based CS reconstruction algorithms are very slow to converge for obtaining good estimates of the original images.

For a given image , the CS measurement of can be expressed as , where is the sensing matrix and (). In practice, as the dimension of natural images are often very high, the memory requirement for storing the matrix is huge. For example, for a moderate image of size and , the dimension of is . The computational complexity of the CS reconstruction with such large sensing matrix is also prohibitively high. To avoid these difficulties, block-based CS (BCS) methods have been proposed [8, 9], where an image is divided into many non-overlapping blocks and each block is sensed individually. As such, the sensing matrices become much smaller and images can be efficiently measured. However, as the blocks are sensed and reconstructed individually, the BCS methods lead to serious blocking artifacts. To reduce the blocking artifacts, post-processing is often required to improve the visual quality.

According to the CS theory [1, 2]

, the original images can be well-reconstructed by exploiting the sparsity prior of natural images. However, the sparsity-based CS methods recover the original images by solving an optimization algorithm, which is very slow to converge. Thus, the optimization-based CS methods cannot be used for real-time applications. Recently, inspired by the great successes of deep neural network (DNN) for computer vision tasks

[10, 11, 12], DNN-based CS reconstruction methods have also been proposed [13, 14, 15]. With DNN, both the linear sensing and nonlinear reconstruction can be performed by a single neural network. Through end-to-end training, both the sensing matrix and the reconstruction method can be jointly optimized, and the speed is usually hundreds-times faster than optimization-based methods. However, to the best of our knowledge, current DNN based methods are all block-based, i.e., image blocks are sensed and reconstructed individually. Blocking artifacts can also be observed in the reconstructed images by the DNN-based methods [13, 14].

In this paper, we propose a novel convolutional compressive sensing (ConvCS) framework based on deep convolutional neural network (DCNN). In the proposed ConvCS network, the first layer senses an input image by convolving the whole image with a set of random filters, followed by subsampling. The advantage of the proposed ConvCS is that the whole image can be efficiently sensed with a set of small filters that are easy to store, and effectively reconstructed without introducing blocking artifacts. The remaining layers of the proposed ConvCS network perform the nonlinear reconstruction of the whole image from the measurements. To design the reconstruction network, the domain knowledge of the sparsity-based CS reconstruction is incorporated, leading to a novel CNN for image reconstruction. By end-to-end training, both the convolutional sensing filters and the reconstruction CNN are jointly optimized. Experimental results show that the proposed method substantially outperforms current state-of-the-art CS methods in terms of PSNR and visual quality.

Ii Related Works

Ii-a Background of CS

Instead of sampling the entire signal, the CS theory samples a signal by taking only linear measurements, i.e., , , where , is the sensing matrix and is the measurement noise. Since , recovering from is generally an ill-posed inverse problem. However, the CS theory guarantees perfect reconstruction of if is sparse in some sparsifying spaces and holds the RIP. It has been proven that the Gaussian random matrices have the RIP with very high probabilities. Standard CS methods recover by solving a minimization problem,

(1)

where D is the dictionary and are the sparse codes of . After estimating the sparse codes , can be reconstructed as . It has been proven in [1, 2] that can be faithfully recovered from measurements, where denotes the number of nonzero coefficients of . The -minimization problem can be solved by many optimization algorithms [16, 17]. However, as mentioned above these methods are very slow to converge.

Ii-B Block-based image CS

When applying CS to images, the image can be sensed by representing it into a vector and measured as

. However, for images of high dimensionality, the sensing matrix becomes very large and it is impossible to store it and compute with such large matrix . To avoid such difficulties, block-based CS (BCS) methods have been proposed [8, 9]. In these methods, the input image is divided into many non-overlapped blocks and each block is sensed independently using a much smaller matrix . The full image is recovered by placing back the reconstructed blocks, followed by full-image smoothing. To reduce the blocking artifacts, full-image iterative shrinkage algorithm was proposed [9], and improvements can be achieved by using more advanced transforms, such as contourlets and dual-tree discrete wavelet transforms [8]. In addition to the sparsity prior, model-based CS recovery algorithms have also been developed to exploit the high-order dependencies between the wavelet coefficients [18], leading to better performance. The effective nonlocal self-similarity prior has also been integrated into the objective function through nonlocal low-rank regularization [19].

Ii-C Structured CS for images

To overcome the drawback of the BCS, structured CS operators [20, 21, 22] have been proposed. The convolutional CS methods, which performs the sensing by first convoluting a signal with a random filter and then subsampling, have been proposed in [20, 22]. These CS methods are easy to implement and have many potential applications, such as Fourier optics [20], Radar imaging [21] and coded aperture imaging [23]. In addition to the random filter, deterministic filter that is more convenient to implement has also been proposed in [22]. Though the convolution-based CS are more easy to implement, the use of only one random filter makes the reconstruction problem more difficult [20]. Also, all these convolution-based CS use iterative algorithms to reconstruct the original images, which are very slow to converge.

Ii-D Deep learning based CS for images

Recently, inspired by the successes of the deep neural networks, non-iterative CS reconstruction methods have been proposed [13, 14, 15]. In [13], a stacked auto-encoder denoising network was developed for image CS. Similarly, a full-connected neural network has been proposed in [15]. Convolutional neural network has also been proposed for this task, where a BM3D denoising [24] stage is adopted to further improve the reconstruction performance [14]. Through end-to-end training, both the sensing matrices and the reconstruction network can be jointly optimized for better performance. However, all those methods perform CS and reconstruction on image blocks, leading to limited reconstruction performance.

In this paper, we propose a new convolutional compressive sensing (denoted as ConvCS) framework using deep convolutional neural network (DCNN), where the first layer implements the sensing by convolving the input image with a set of random filters followed by subsampling. The remaining layers reconstruct the input image from the linear measurements. Furthermore, inspired from the sparsity-based reconstruction model, a novel CNN containing two branches is proposed for CS reconstruction. By performing sensing and reconstruction on the whole image, the proposed method significantly outperforms the previous CS methods. Different from previous convolution-based CS methods [20, 22], both the sensing filters and the reconstruction algorithm can be jointly optimized. Experimental results show that the proposed method outperforms existing state-of-the-art CS methods by a large margin.

Iii Proposed Convolutional Compressive Sensing using Deep Learning

Iii-a Proposed convolutional compressive sensing

Unlike existing BCS and convolutional CS, we propose to sense an image by convolving it with a set of random filters, followed by spatial subsampling of the convolved images. Specifically, for a given image , we convolve it with a set of random filters of size , , to generate the CS measurements. Mathematically, the sensing matrix can be expressed as

(2)

where is set to be a sparse matrix such that is equivalent to convolving with filter , and is a subsampling matrix. Then, the proposed convolutional CS (ConvCS) for can be formulated as , , where .

As convolution can also be implemented with matrix-vector multiplication, the proposed ConvCS is equivalent to the CS process that first extracts blocks of size with sliding step and then measures the blocks with the sensing matrix, whose -th rows are composed with the vectorized filter coefficients of . Thus, the proposed ConvCS matrix of Eq. (2) still holds the advantages of the random Gaussian matrix for CS (i.e., the RIP). However, the proposed ConvCS is clearly distinct from the previous BCS methods in two aspects. First, the proposed ConvCS for images of large dimensions is much easier to implement. Second, the convolutional nature of the proposed ConvCS makes the joint optimization of the sensing filters and reconstruction of the whole image much more effective, without introducing any blocking artifacts.

The proposed ConvCS can be easily implemented using a convolutional neural network, as shown in Fig. 1 (a). The first layer convolves the input image with a set of random filters of size

with stride

. The measurements can be obtained by representing the obtained feature maps into a -dimensional vector

. Note that nonlinearity is not involved in the CS process. More convolutional layers with nonlinear activity function could be added into the ConvCS process, which may lead to better performance. However, for simplicity, here we only use one linear layer to obtain the measurements. As shown in Sec.

IV, the simple ConvCS can already lead to excellent CS recovery performances.

Essentially, the ConvCS encodes the visual information of images and plays a similar role as autoencoders. In the context of CS, the stacked denoising autoencoders (SDA)

[13] have been proposed to encode the images. However, the SDA method performs CS encoding at image block levels, resulting in serious blocking artifacts.

Fig. 1: The proposed convolutional compressive sensing network (ConvCSNet). (a) Convolutional compressive sensing layer; (b) Two-branch convolutional neural network for CS reconstruction.

Iii-B Sparse model inspired DCNN for CS reconstruction

After obtaining the CS measurements, we aim to recover the original image from the measurements. In recent years, DCNN has shown very promising performances for many low-level image processing tasks, e.g., image super-resolution

[25, 26]. However, the problem of reconstructing images from the ConvCS measurements is different from previous image SR problem, and existing reconstruction network can not be applied directly for this task. To facilitate the design of the CS reconstruction network, we propose to incorporate the domain knowledge of the sparsity-based CS reconstruction. Specifically, we first propose the following analysis sparse representation model for CS reconstruction,

(3)

where is the analysis filter, denotes the 2D convolution, and denotes a regularization term imposed on sparse codes . Classic sparsity enforcing regularizers, e.g., (), as well as the non-negative indicator function

(as inspired by the ReLU function) can be adopted. Solving Eq. (

3) amounts to alternatively solving two subproblems, i.e.,

(4)

where . Both sub-problems can be easily optimized. With a fixed estimate obtained at -th iteration, can be solved in closed-form for several commonly used sparsity regularizer, as

(5)

for -norm sparse regularizer and non-negative regularizer, respectively, where denotes the soft-thresholding with threshold . The -subproblem is a quadratic optimization problem and can be solved in closed-form. However, to avoid large matrix inversion, we prefer to solve it via a gradient descent method. With a fixed estimate of , we iteratively update as

(6)

where is the sparse matrix such that is equivalent to convolving with , and is the predefined constant. By alternatively updating and , the iterative process will converge. However, the convergence speed is very slow.

In this paper, we propose to convert the alternative update of and into a deep network. For simplicity, we rewrite Eq. (6) as

(7)

where denotes the process that first senses with and then reconstructs from the CS measurements by back-projection. For simplicity, we let and . Then, Eq. (7) can be approximated as

(8)

where , , denotes the initial reconstruction of from , and denotes the reconstructed from .

Inspired from the alternative update of and , we propose a novel DCNN for CS reconstruction. As shown in Fig. 1 (b), the proposed reconstruction network contains two branches. The first branch implements a conventional CNN for generating the feature maps . The first branch taking the measurements vector as input back projects into the feature maps, where all entries are zero except the sampled set of pixels (as marked red shown in Fig. 1 (b)). The first layer uses kernels of size and generates feature maps, while the remaining layers use kernels of size and generate feature maps. The ReLU function is applied following convolution. Compared to the directly implementation of Eq. (5), the deep CNN is more powerful in learning the representation of the original image . The first CNN branch can also be regarded as a nonlinear mapping function used to accurately predict the sparse codes . The second branch recursively reconstructs the image based on the feature maps from the first branch and the previously reconstructed images and , which mimics the computations of Eqs. (7) and (8). In each layer of the second branch, the feature map from the CNN branch is fed into a convolutional layer to produce an image, which is further added with previously reconstructed and for an updated estimate . The kernel size used in the reconstruction branch is also .

Iv Experimental results

Iv-a Training details

To achieve better performance, the sensing layers and the remaining reconstruction layers are jointly trained, and thus the sensing filters and the reconstruction network can be jointly optimized. Let ConvCSNet denote the proposed network performing convolutional CS and reconstruction. To verify the effectiveness of the proposed two-branch reconstruction network, we also implement a variant of the ConvCSNet (denoted as ConvCSNet-baseline), which uses only the first branch of the reconstruction network shown in Fig. 2

(b) for CS reconstruction. To train the proposed networks, we collected natural images from the ImageNet dataset

[27] and extracted the central part of each image. The extracted patches are converted into grayscale and augmented via horizontal and vertical flips and rotations. Finally, we obtained a training set containing image patches. We empirically set the parameters of the convolutional sensing layer as: for rate ; for rate , for rate , and for rate , where and denotes the filter size, number of filters and the convolutional stride, respectively. The parameters of the reconstruction layers are the same for different measurement rates, except the first layer of the reconstruction part of the ConvCSNet. In that layer, we use corresponding filter size of used in the sensing layer.

The proposed network is trained using the loss function, as , where denotes all the network parameters (including the sensing filters) and is the total number of training patches. The proposed network is trained with ADAM optimizer [28] by setting , and . The minibatch size is set to . We initialize the learning rate as and halved at every

minibatch updates. We implemented the proposed network with TensorFlow framework and training them using 4 NVIDIA 1080Ti GPUs. It takes one day to train ConvCSNet. Currently, we trained one model for each sensing rates. In future, a general reconstruction network may be trained for different measurement rates.

Iv-B Comparison with state-of-the-art methods

Fig. 2: The test images. From left to right: Butterfly, Flowers, Starfish, Parthenon, raccoon, Girl, and Leaves.
Images Noiseless Noisy
Ratio 0.05 0.1 0.2 0.3 0.05 0.1 0.2 0.3
butterfly
TV[29]
FCN-CS[15]
D-AMP[30]
ConvCSNet-baseline
ConvCSNet
19.01
22.93
22.70
24.17
25.63
22.73
25.78
26.55
27.50
28.48
28.03
29.07
31.38
31.44
32.91
31.99
30.87
34.32
34.72
35.36
18.56
22.82
20.52
23.99
25.35
21.88
24.83
23.40
27.11
28.46
25.11
27.01
27.75
29.85
30.70
26.28
29.40
28.93
32.32
33.49
parthenon
TV[29]
FCN-CS[15]
D-AMP[30]
ConvCSNet-baseline
ConvCSNet
22.75
25.28
24.59
25.34
25.79
24.90
27.18
27.43
26.80
27.39
27.28
29.93
30.72
30.01
30.94
29.18
31.96
33.11
33.35
33.89
22.16
25.21
23.78
24.99
25.65
23.76
26.59
25.50
26.46
27.31
25.21
28.58
27.62
28.70
29.53
25.95
30.49
28.90
31.23
32.44
starfish
TV[29]
FCN-CS[15]
D-AMP[30]
ConvCSNet-baseline
ConvCSNet
22.07
24.99
23.87
25.04
25.79
24.31
27.35
27.44
27.29
28.79
27.58
30.91
31.73
32.61
34.10
30.24
33.25
34.46
36.17
37.51
21.65
24.94
22.71
24.62
25.67
23.50
27.16
25.20
26.94
28.72
25.37
29.76
27.82
30.27
31.77
26.57
32.11
29.25
33.19
35.00
flower
TV[29]
FCN-CS[15]
D-AMP[30]
ConvCSNet-baseline
ConvCSNet
23.48
26.00
24.96
26.10
26.80
25.81
28.08
28.34
27.88
29.14
28.85
30.91
32.49
32.14
33.38
31.33
32.83
36.18
35.67
36.11
22.9
25.92
24.07
25.72
26.60
24.39
27.81
26.13
27.45
29.07
26.03
29.90
28.20
30.40
31.45
26.88
31.83
29.64
32.82
34.49
girl
TV[29]
FCN-CS[15]
D-AMP[30]
ConvCSNet-baseline
ConvCSNet
28.78
31.29
30.35
31.55
31.97
30.41
32.87
31.87
33.10
33.43
32.20
34.58
33.43
34.69
35.22
33.47
35.76
34.42
34.5
36.55
27.00
31.24
28.18
31.05
31.57
27.78
32.68
29.69
32.22
33.17
28.31
33.95
30.87
32.72
33.34
28.46
35.02
31.59
33.64
34.14
leaves
TV[29]
FCN-CS[15]
D-AMP[30]
ConvCSNet-baseline
ConvCSNet
17.64
22.13
18.69
22.09
23.51
20.10
24.67
26.80
24.76
27.03
24.50
28.82
33.00
30.93
33.32
28.08
31.3
36.77
34.73
36.98
17.27
22.06
20.20
21.82
23.38
19.81
24.31
24.42
24.44
27.05
23.02
27.27
27.77
29.18
30.96
24.94
30.12
29.2
33.15
35.20
raccoon
TV[29]
FCN-CS[15]
D-AMP[30]
ConvCSNet-baseline
ConvCSNet
24.08
26.17
25.16
25.68
26.27
25.66
27.63
27.12
27.12
27.95
27.53
29.82
29.44
30.73
31.20
29.12
31.66
31.77
33.38
34.17
23.60
26.16
24.34
25.44
26.14
24.60
27.53
25.87
26.73
27.87
25.83
29.43
27.48
29.45
29.71
26.47
31.25
28.47
31.29
32.58
Average
TV[29]
FCN-CS[15]
D-AMP[30]
ConvCSNet-baseline
ConvCSNet
22.54
25.54
24.33
25.71
26.54
24.85
27.65
27.94
27.78
28.89
28.00
30.58
31.74
31.79
33.01
30.49
32.52
34.43
34.65
35.80
21.88
25.48
23.40
25.38
26.34
23.68
27.27
25.96
27.34
28.81
25.55
29.41
28.22
30.08
31.07
26.51
31.46
29.43
32.52
33.91
TABLE I: PSNR results on the set of images of Fig. 2 by the test methods at different measurement rates.

We compare the proposed ConvCSNet method with two iterative CS methods, including the total variation method (denoted as TV [29]), the denoising-based approximate message passing (D-AMP) method [30], and one recently developed fully-connected network based CS method (denoted as FCN-CS) [15]. Note that both TV [29] and D-AMP methods [30] conduct compressive sensing on the whole image, using very large measurement matrices. For an input image of size and measurement rate , the measurement matrix is of size , requiring about memory for storing it in Matlab platform. The computational complexity of the iterative CS reconstruction using such large sensing matrices also become very slow. However, the advantage of using such large measurement matrices is that the reconstruction quality can be much improved. Also note that the D-AMP method [30] uses the well-known BM3D denoising method [24] in its iterative reconstruction process. As the BM3D denoising method is very effective in suppressing noise and artifacts, the D-AMP method achieves the state-of-the-art CS performance. The deep learning based FCN-CS method [15] performing the sensing and reconstruction based on blocks. As the authors of FCN-CS method only provided the test code in their website, we re-implemented the training algorithm of FCN-CS method and trained the network with our training dataset. As we use larger training dataset, the performance of FCN-CS method is much improved, compared with those obtained using the model provided by the authors of [15].

We have also tried to compare our method with the ReconNet of [14]. However, the results obtained using the code downloaded from their website are worse than those reported in their paper. We think that this may be caused by different parameters settings, and tuning the parameters of the method for better results is out of the scope of this paper. Hence, we didn’t include the ReconNet [14] into our comparison study. All the codes of the competing methods are downloaded from authors’ websites. We generate the measurements at four measurement rates , and

. To verify the robustness of the reconstruction algorithms to the measurement noise, we also conducted CS reconstruction using noisy measurements. To this end, Gaussian noise of standard deviation of

is added to the measurements. A set of natural images of size is used as test images, as shown in Fig. 2. The well-known Berkeley segmentation dataset containing 100 natural images (denoted as BSD100) is also used to verify the performances of the test methods. For the BSD100 dataset, we extract the central part of each image as test images. Note that all the test images are excluded from the training dataset.

Noiseless Noisy
Ratio 0.05 0.1 0.2 0.3 0.05 0.1 0.2 0.3
TV[29] 24.09 26.00 28.46 30.45 21.96 22.95 24.17 25.16
FCN-CS[15] 25.92 27.54 29.85 31.57 25.89 27.29 29.03 30.74
D-AMP[30] 25.48 27.66 30.65 33.12 22.63 23.72 24.95 25.81
ConvCSNet-baseline 26.25 28.12 30.36 33.08 25.80 27.66 28.96 31.15
ConvCSNet 26.47 28.19 31.03 33.49 26.22 28.09 29.63 32.25
TABLE II: Average PSNR results of different methods on test set of BSD100.
(a) Original
(b) TV[29]
(c) FCN-CS[15]
(d) D-AMP[30]
(e) Proposed ConvCSNet
Fig. 3: Reconstructed results for Leaves and Butterfly images from noiseless CS measurements at sensing rate . The PSNR results: (b) TV [29] (17.64 dB, 19.01 dB); (c)FCN-CS[15] (22.13 dB, 22.93 dB); (d)D-AMP[30] (18.69 dB, 22.70 dB); (e) Proposed ConvCSNet (23.51 dB, 25.63 dB)
Fig. 4: Reconstructed results for Starfish images from noisy CS measurements at sensing rate . The PSNR results: (b) TV [29] (23.50 dB); (c)FCN-CS[15] (27.16 dB); (d)D-AMP[30] (25.20 dB); (e) Proposed ConvCSNet (28.72 dB)
Methods TV[29] FCN-CS[15] D-AMP[30] ConvCSNet
Time 82.47 0.41 60.79 0.08
TABLE III: Running time (second) of different methods on a test image of size with measurement rate 0.2.

Table I reports the PSNR results of the test methods on the set of test images shown in Fig. 2. It can be seen that the proposed ConvCSNet outperforms the ConvCSNet-baseline method by a large margin, demonstrating the effectiveness of the proposed CS reconstruction network. When compared with other competing methods, the proposed ConvCSNet method also performs better than the other methods on all measurement rates. It significantly outperforms the block-based FCN-CS method on all measurement rates. The D-AMP method performs very well for high measurement rates (e.g., 0.2 and 0.3). The proposed ConvCSNet also performs much better than D-AMP method on all measurement rates. For noisy cases, the proposed ConvCSNet also significantly outperforms the other methods by large margins. Table II shows the average PSNR results of the test methods on the BSD100 dataset. From Table II, we can see that the proposed ConvCSNet also outperforms all the other competing methods, for both noiseless and noisy cases.

Figs. 3 and 4 show parts of the reconstructed images by the test methods. Clearly, the visual quality of the images reconstructed by the proposed ConvCSNet method is significantly better than other competing methods. The proposed method can generate visually pleasant images at measurement rate , while other methods produce images with severe visual artifacts. For more visual comparisons, please refer to the supplementary material.

Regarding the computational complexity of the test methods, we report the running time of the test methods on a test image of size for the noiseless case, as shown in Table III. For TV [29] and D-AMP [30] methods, the computer with Intel i7-6700 3.4GHz and G memory is used to run these algorithms provided by the authors. For the FCN-CS [15] and the proposed methods, a Nvidia GTX 1080Ti GPU is used to compute the CS reconstructions. From Table III, it can be seen that the time taken by the proposed ConvCSNet for reconstructing a image is about times faster than the TV method [29] and times faster than the D-AMP method [30]. Note that the high computational complexity of TV and D-AMP is not only due to slow convergence, but also the use of huge measurement matrices. When compared to the block-based FCN-CS method, our method is also about times faster.

V Conclusions

Compressive sensing is a new image/signal acquisition paradigm, which has potentials in high-speed and energy efficient imaging applications. However, a practical issue in the use of CS theory is the huge memory and computation requirements for sensing a whole image. While performing CS on image block level leads to efficient measurement, it will significantly decrease the reconstruction performance. In this paper, we propose a novel convolutional compressive sensing (ConvCS) method based on deep learning. In the proposed ConvCS network, the first layer conducts sensing of the whole image using a set of convolutional filters and the remaining layers performs the reconstruction of the whole image. For better CS reconstruction, a novel two-branch convolutional neural network is proposed. Through end-to-end training, both the sensing filters and the reconstruction network can be jointly optimized. Experimental results show that the proposed method significantly outperforms existing iterative and deep learning based CS methods.

References

  • [1] E. J. Candès, J. K. Romberg, and T. Tao, “Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information,” IEEE Transactions on Information Theory, vol. 52, no. 2, pp. 489–509, 2006.
  • [2] D. L. Donoho, “Compressed sensing,” IEEE Transactions on Information Theory, vol. 52, no. 4, pp. 1289–1306, 2006.
  • [3] M. Duarte, M. Davenport, D. Takbar, J. Laska, T. Sun, K. Kelly, and R. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Processing Magazine, vol. 25, no. 2, pp. 21–30, 2008.
  • [4] M. E. Gehm, R. John, D. Brady, R. Willett, and T. J. Schulz, “Single-shot compressive spectral imaging with a dual-disperser architecture,” Optics Express, vol. 15, no. 21, pp. 14 013–14 027, 2007.
  • [5] Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Video from a single coded exposure photograph using a learned over-complete dictionary,” in Proc. of the IEEE ICCV, 2011, pp. 287–294.
  • [6] M. Lustig, D. L. Donoho, J. M. Santos, and J. M. Pauly, “Compressed sensing mri,” IEEE Signal Processing Magazine, vol. 25, no. 2, pp. 72–82, 2008.
  • [7] M. Elad and M. Aharon, “Image denoising via sparse and redundant representation over learned dictionaries,” IEEE Transactions on Image Processing, vol. 15, no. 12, pp. 3736–3745, 2006.
  • [8] J. E. Fowler, S. Mun, and E. W. Tramel, “Block-based compressed sensing of images and video,” Foundations and Trends in Signal Processing, vol. 4, no. 4, pp. 297–416, 2012.
  • [9] S. Mun and J. E. Fowler, “Block compressed sensing of images using directional transforms,” in Proc. of the ICIP, 2009, pp. 3021–3024.
  • [10] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Proc. of the NIPS, 2012, pp. 1097–1105.
  • [11] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proc. of the CVPR, 2014, pp. 580–587.
  • [12] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proc. of the CVPR, 2015, pp. 3431–3440.
  • [13] A. Mousavi, A. B. Patel, and R. G. Baraniuk, “A deep learning approach to structured signal recovery,” in Proc. of Annual Allerton Conf. on Comm., Control, and Computing, 2015, pp. 1336–1343.
  • [14] K. Kulkarni, S. Lohit, P. Turaga, R. Kerviche, and A. Ashok, “Reconnet: non-iterative reconstruction of images from compressively sensed random measurements,” in Proc. of CVPR, 2016, pp. 449–458.
  • [15] A. Adler, D. Boublil, M. Elad, and M. Zibulevsky, “A deep learning approach to block-based compressed sensing of images,” in Proc. of ICASSP, 2017.
  • [16] I. Daubechies, M. Defriese, and C. DeMol, “An iterative thresholding algorithm for linear inverse problems with a sparsity constraint,” Commun. Pure Appl. Math., vol. 57, no. 11, pp. 1413–1457, 2004.
  • [17] M. Zibulevsky and M. Elad, “l1-l2 optimization in signal and image processing,” IEEE Signal Processing Magazine, vol. 27, no. 3, pp. 76–88, 2010.
  • [18] R. Baraniuk, V. Cevher, M. Duarte, and C. Hegde, “Model-based compressive sensing,” IEEE Trans. on Information Theory, vol. 56, no. 4, pp. 1982–2001, 2010.
  • [19] W. Dong, G. Shi, X. Li, Y. Ma, and F. Huang, “Compressive sensing via nonlocal low-rank regularization,” IEEE Trans. on Image Processing, vol. 23, no. 8, pp. 3618–3632, 2014.
  • [20] J. Romberg, “Compressive sensing by random convolution,” SIAM J. Imaging Sci., vol. 2, no. 4, pp. 1098–1128, 2009.
  • [21] J. Y. W. Yin, S. Morgan and Y. Zhang, “Practical compressive sensing with toeplitz and circulant matrices,” in Proc. of VCIP, 2010.
  • [22] K. Li, L. Gan, and C. Ling, “Convolutional compressed sensing using deterministic sequences,” IEEE Trans. on Signal Processing, vol. 61, no. 3, pp. 740–752, 2013.
  • [23] R. Marcia, Z. Harmany, and R. Willett, “Compressive coded aperture imaging,” in Proc. of SPIE Electron. Imag., 2009.
  • [24] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-d transform-domain collaborative filtering,” IEEE Trans. on Image Processing, vol. 16, no. 8, pp. 2080–2095, 2007.
  • [25] C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional network for image super-resolution,” in Proc. of ECCV, 2014.
  • [26] J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution using very deep convolutional networks,” in Proc. of CVPR, 2016, pp. 1646–1654.
  • [27] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” Int. J. Computer Vision, vol. 56, no. 3, pp. 507–530, 2013.
  • [28] D. Kingma and J. Ba, “Adam: a method for stochastic optimization,” in Proc. of ICLR, 2014.
  • [29] C. Li, W. Yin, H. Jiang, and Y. Zhang, “An efficient augmented lagrangian method with applications to total variation minimization,” Computational Optimization and Applications, vol. 56, no. 3, pp. 507–530, 2013.
  • [30] C. A. Metzler, A. Maleki, and R. G. Baraniuk, “From denoising to compressed sensing,” IEEE Trans. on Information Thoery, vol. 62, no. 9, pp. 5117–5144, 2016.