1 Motivation
Inspired by compressive sensing (CS) [2, 3], diverse compressive cameras [4, 5, 6, 7, 8, 9, 10, 11, 12, 13] have been built. The singlepixel camera [4], which implemented the compressive imaging in space, is an elegant architecture to prove the concept of CS. However, the hardware elements used in the singlepixel camera are more expensive than conventional CCD or CMOS cameras, thus limit the applications of this new imaging regime. The lensless compressive camera (), proposed in [1], enjoys lowcost property and has demonstrated excellent results using advanced reconstruction algorithms [14, 19, 30].
Though enjoying various advantages, the existing [1] suffers from low capture rates. Specifically, the currently used sensor can only capture the scene at around Hz. For a image, if we desire a high resolution image, we need on the order of measurements (relative to the pixel numbers). This requires about 1 minute with current aperture switching technique. This is far from our real time requirement as cameras are built to get instant images/videos. In order to obtain images in a shorter time, we have to sacrifice the spatial resolution, i.e. providing a lowresolution image. An alternative solution is to increase the refresh rate of the aperture assembly (Fig. 1). However, even we can achieve a higher refresh rate using an expensive hardware, the sensor needs a relative long time to integrate the light or a more expensive sensor is required.
There are strong advantages with the , directly implementing compressive sensing. These include minimizing problems of lenses sensor. But with current approaches, speed is an issue. In this paper, we propose the blockwise to mitigate this problem.
2 Blockwise Lensless Compressive Camera
Figure 1 depicts the geometry of the proposed blockwise . It consists of three components as shown in Fig. 2: a) the sensor board which contains multiple sensors and each one corresponding to one block, b) the isolation chamber which prevents the light from other blocks, and c) the aperture assembly which can be the same as that used in the [1].
Note that the block sizes can be different such that each block can reconstruct different resolution image fractions, thus leading to multiscale compressive camera [15]. The pattern used for each block can also be different and can be adapted to the content of the image part, thus leading to adaptive compressive sensing [16]. In this work, we consider each block uses the same pattern as this will not only save the memory to store these patterns but also enables fast reconstruction for each block.
This new blockwise enjoys the following advantages compared to the existing .

Since each block can be very small, e.g. pixels, we only need to capture a small number of measurements to achieve high resolution reconstruction. Therefore the capture time can be short.

The coding patterns used in each block can be the same, therefore the sensing matrix is only of the block size. This saves the memory requirement of sensing matrix as well as speeds up the reconstruction.

Patch based image reconstruction is fast and since real time stitching algorithms exist, we can perform real time reconstruction.

These blocks can be integrated to any desirable number, leading to extra high resolution images [15] while retaining the capture rate and fast reconstruction;

The sensor layer can be very close to the aperture assembly, which leads to small size camera. In particular, the thickness of the camera will be extremely small [17].
2.1 Overlapping Regions and Stitching
From the simple geometry shown in Fig. 1, we can see that if the scene is far from the sensor, we will have significant overlapping regions. This will be discussed in next section. On the other hand, the image stitching algorithms are usually based on the features within the overlapping areas to perform registration. Since the angular resolution is limited by the distance between sensors and the aperture assembly, we can adapt this distance in different applications.
2.2 ConcentrationSensor Regime
In order to mitigate the problem mentioned in Section 2.1, i.e. the low angular resolution for far scenes, we propose the following ConcentrationSensor Regime, where the sensors are put together in a “cellular” shape. The aperture assembly can be a plane or a spherical surface as shown in Fig. 3. The planer aperture assembly in Fig. 3(ab) can be replaced by a spherical aperture as shown in Fig. 3(c). The sensor layout of the spherical regime is detailed in Fig. 4. In this configuration, the isolation chamber will be a “trumpet” shape starting from the sensor to the aperture assembly, as demonstrated in Fig. 5. Note the configuration in Fig. 3 (c) and Fig. 4 is a wide angle camera.
3 Fast Reconstruction
We consider that the patterns used for each block are the same. The measurements can be expressed as
(1) 
where with
denoting the dimension of the vectorized block (with
pixels) and is the number of blocks used in the camera. with denotes the measurements and each column corresponds to one block (measurements captured by one sensor), and signifies the measurement noise. is the sensing matrix, which can be random [18] or the Hadamard matrix with random permutation [19].3.1 Dictionary Based Inversion
Introducing a basis or (blockbased) dictionary , equation (1) can be reformulated as
(2) 
where can be an orthonormal basis with or an overcomplete dictionary [20] with . This dictionary can be prelearned for fast inversion or learned in situ [21, 22]. is desired to be sparse so that various based algorithms can be used to solve the following problem [23, 24]
subject to  (3) 
given and or variant problems [25].
Diverse algorithms have been proposed to solve the above problem and we will use the advanced Gaussian mixture model described below since it does not need any iteration as closedform analytic solution exists.
3.2 Closedform Inversion via Gaussian Mixture Models
The Gaussian mixture model (GMM) has recently been rerecognized as an efficient dictionary learning algorithm [26, 27, 28, 29, 30]. Recall the image blocks extracted from the image. For th patch , it is modeled by a GMM with Gaussian components [26]:
(4) 
where represent the mean and covariance matrix of th Gaussian, and denote the weights of these Gaussian components, and .
Dropping the block index , in a linear model , , if in (4), then has the following analytical form [26]
(5) 
where
(6)  
(7)  
(8) 
While (5) provides a posterior distribution for
, we obtain the point estimate of
via the posterior mean:(9) 
which is a closedform solution.
Note that are pretrained on other datasets and given , only needs to be computed once and saved. Same techniques can be used for . The only computation left for each block is to calculate , which can be obtained very efficiently.
Most importantly, no iteration is required using the above GMM method, leading to realtime
reconstruction for each block. Furthermore, each block can be reconstructed in parallel with GPU. Since real time stitching algorithm exists, after blocks are obtained via the GMM, we can get the entire image instantly. Alternatively, we can pretrain a neural network for the inversion and obtain the reconstruction in real time
[31].4 Results
In this section, we first conduct the simulation to verify the proposed approach and then describe the hardware prototypes we have built to demonstrate real data results.
4.1 Simulation
We consider each block with size pixels and reconstruct the images with different numbers of measurements. The image is of size . We assume that there is no overlapping among blocks and the image part is ideally captured by different sensors. The PSNR (PeakSignaltoNoiseRatio) of the reconstructed image is used as a metric and plotted in Fig. 6 with exemplar reconstructed images shown in Fig. 7. The GMM is used for reconstruction. We can observe that even a single measurement in each block provides excellent reconstruction. The running time for reconstruction is seconds using an i7 CPU.
4.2 Proof of Concept Prototypes and Results
We first used 4 sensors to build the blockwise to prove the concept, with photos shown in Fig. 8. We used pixels pattern for each sensor and these patterns are shared across these 4 sensors. The scene is a photo printed on a paper and pasted on a wall. Without employing any stitching algorithm, one result is shown in Fig. 9, where the CSr is defined as the number of measurements relative to pixels in the reconstructed image. It can be observed that good results can be obtained by only using of the pixel data (CSr = 0.1), which is 102 measurements captured by each sensor.
Next we show the reconstruction results with 16 sensors (a array) in Fig. 10, where we restricted the light to test the performance of the camera in dark environment. It can be observed that even when CSr (as each block is of pixels, CSr denotes each sensor only captured 12 measurements), we can still get reasonable results.
5 Conclusions
We have proposed the blockwise lensless compressive camera to mitigate the speed issue existing in the current . Multiple geometries of this blockwise have been developed. Prototypes have been built to demonstrate the feasibility of the proposed imaging architecture. Excellent results of real data have been shown to verify the fast capture and real time reconstruction of the proposed camera.
References
 [1] G. Huang, H. Jiang, K. Matthews, and P. Wilford, “Lensless imaging by compressive sensing,” IEEE International Conference on Image Processing, 2013.
 [2] D. L. Donoho, “Compressed sensing,” IEEE Transactions on Information Theory, vol. 52, no. 4, pp. 1289–1306, April 2006.
 [3] E. J. Candès, J. Romberg, and T. Tao, “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” IEEE Transactions on Information Theory, vol. 52, no. 2, pp. 489–509, February 2006.
 [4] M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, “Singlepixel imaging via compressive sampling,” IEEE Signal Processing Magazine, vol. 25, no. 2, pp. 83–91, 2008.
 [5] A. Wagadarikar, R. John, R. Willett, and D. J. Brady, “Single disperser design for coded aperture snapshot spectral imaging,” Applied Optics, vol. 47, no. 10, pp. B44–B51, 2008.
 [6] P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, “Coded aperture compressive temporal imaging,” Optics Express, pp. 698–706, 2013.

[7]
D. Reddy, A. Veeraraghavan, and R. Chellappa,
“P2C2: Programmable pixel compressive camera for high speed
imaging,”
IEEE Computer Vision and Pattern Recognition (CVPR)
, 2011.  [8] Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Video from a single coded exposure photograph using a learned overcomplete dictionary,” in IEEE International Conference on Computer Vision (ICCV), 2011.
 [9] X. Yuan, P. Llull, X. Liao, J. Yang, G. Sapiro, D. J. Brady, and L. Carin, “Lowcost compressive sensing for color video and depth,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
 [10] T.H. Tsai, X. Yuan, and D. J. Brady, “Spatial light modulator based color polarization imaging,” Optics Express, vol. 23, no. 9, pp. 11912–11926, May 2015.
 [11] T.H. Tsai, P. Llull, X. Yuan, D. J. Brady, and L. Carin, “Spectraltemporal compressive imaging,” Optics Letters, vol. 40, no. 17, pp. 4054–4057, Sep 2015.
 [12] Y. Sun, X. Yuan, and S. Pang, “Highspeed compressive range imaging based on active illumination,” Opt. Express, vol. 24, no. 20, pp. 22836–22846, Oct 2016.
 [13] X. Cao, T. Yue, X. Lin, S. Lin, X. Yuan, Q. Dai, L. Carin, and D. J. Brady, “Computational snapshot multispectral cameras: Toward dynamic capture of the spectral world,” IEEE Signal Processing Magazine, vol. 33, no. 5, pp. 95–108, Sept 2016.
 [14] X. Yuan, H. Jiang, G. Huang, and P. Wilford, “SLOPE: Shrinkage of local overlapping patches estimator for lensless compressive imaging,” IEEE Sensors Journal, vol. 16, no. 22, pp. 8091–8102, November 2016.
 [15] D. J. Brady, M. E. Gehm, R. A. Stack, D. L. Marks, D. S. Kittle, D. R. Golish, E. M. Vera, and S. D. Feller, “Multiscale gigapixel photography,” Nature, , no. 486, pp. 386–389, 2012.
 [16] X. Yuan, J. Yang, P. Llull, X. Liao, G. Sapiro, D. J. Brady, and L. Carin, “Adaptive temporal compressive sensing for video,” in 2013 IEEE International Conference on Image Processing (ICIP), Sept 2013, pp. 14–18.
 [17] M. S. Asif, A.Ayremlou, A. Sankaranarayanan, A. Veeraraghavan, and R. Baraniuk, “Flatcam: Thin, baresensor cameras using coded aperture and computation,” arXiv:1509.00116, 2015.
 [18] E. J. Candès and T. Tao, “Nearoptimal signal recovery from random projections: universal encoding strategies?,” IEEE Transactions on Information Theory, 2006.
 [19] X. Yuan, H. Jiang, G. Huang, and P. Wilford, “Lensless compressive imaging,” arXiv:1508.03498, 2015.
 [20] M. Aharon, M. Elad, and A. Bruckstein, “KSVD: An algorithm for designing overcomplete dictionaries for sparse representation,” IEEE Transactions on Signal Processing, vol. 54, no. 11, pp. 4311–4322, 2006.
 [21] X. Yuan, T.H. Tsai, R. Zhu, P. Llull, D. J. Brady, and L. Carin, “Compressive hyperspectral imaging with side information,” IEEE Journal of Selected Topics in Signal Processing, vol. 9, no. 6, pp. 964–976, September 2015.
 [22] Xin Yuan, “Compressive dynamic range imaging via bayesian shrinkage dictionary learning,” Optical Engineering, vol. 55, no. 12, pp. 123110, 2016.
 [23] X. Liao, H. Li, and L. Carin, “Generalized alternating projection for weighted minimization with applications to modelbased compressive sensing,” SIAM Journal on Imaging Sciences, vol. 7, no. 2, pp. 797––823, 2014.
 [24] X. Yuan, “Generalized alternating projection based total variation minimization for compressive sensing,” in 2016 IEEE International Conference on Image Processing (ICIP), Sept 2016, pp. 2539–2543.
 [25] A. Beck and M. Teboulle, “A fast iterative shrinkagethresholding algorithm for linear inverse problems,” SIAM J. Img. Sci., vol. 2, no. 1, pp. 183–202, Mar. 2009.
 [26] M. Chen, J. Silva, J. Paisley, C. Wang, D. Dunson, and L. Carin, “Compressive sensing on manifolds using a nonparametric mixture of factor analyzers: Algorithm and performance bounds,” IEEE Transactions on Signal Processing, vol. 58, no. 12, pp. 6140–6155, December 2010.
 [27] G. Yu, G. Sapiro, and Stéphane Mallat, “Solving inverse problems with piecewise linear estimators: From Gaussian mixture models to structured sparsity,” IEEE Transactions on Image Processing, 2012.
 [28] J. Yang, X. Yuan, X. Liao, P. Llull, G. Sapiro, D. J. Brady, and L. Carin, “Video compressive sensing using Gaussian mixture models,” IEEE Transaction on Image Processing, vol. 23, no. 11, pp. 4863–4878, November 2014.
 [29] J. Yang, X. Liao, X. Yuan, P. Llull, D. J. Brady, G. Sapiro, and L. Carin, “Compressive sensing by learning a Gaussian mixture model from measurements,” IEEE Transaction on Image Processing, vol. 24, no. 1, pp. 106–119, January 2015.
 [30] X. Yuan, H. Jiang, G. Huang, and P. Wilford, “Compressive sensing via lowrank Gaussian mixture models,” arXiv:1508.06901, 2015.
 [31] K. Kulkarni, S. Lohit, P. Turaga, R. Kerviche, and A. Ashok, “Reconnet: Noniterative reconstruction of images from compressively sensed random measurements,” in CVPR, 2016.