In recent years, vision sensor networks have been gaining an ever increasing popularity enforced by the availability of cheap semiconductor components. These networks typically produce highly redundant information so that an efficient estimation of the correlation between images becomes primordial for effective coding, transmission and storage applications. The distributed coding paradigm becomes particularly attractive in such settings; it permits to efficiently exploit the correlation between images with low encoding complexity and minimal inter-sensor communication, which translates into power savings in sensor networks. One of the most important challenging tasks however resides in the proper modeling and estimation of the correlation between images.
In this paper, we consider the problem of finding an efficient distributed representation for correlated images, where the common objects are displaced due to the viewpoint changes or motion in dynamic scenes. In particular, we are interested in a scenario where the images are given under the form of few quantized linear measurements computed by very simple sensors. Even with such a simple acquisition stage, the images can be reconstructed under the condition that they have a sparse representation in particular basis (e.g., DCT, wavelet) that is sufficiently different from the sensing matrices [3, 4]. Rather than independent image reconstruction, we are however interested in the joint reconstruction of the images and in particular the estimation of their correlation from the compressed measurements. In contrary to most distributed compressive schemes in the literature, we want to estimate the correlation prior to image reconstruction for improved robustness at low coding rates.
We propose to model the correlation between images as geometric transformations of visual features, which provides a more efficient representation than block-based translational models that are commonly used in state-of-the-art coding solutions. We first compute the most prominent visual features in a reference image through a sparse approximation with geometric functions drawn from a parametric dictionary. Then, we formulate a regularized optimization problem whose objective is to identify in the compressed images the features that correspond to the prominent components in the reference images. Correspondences then define relative transformations between images that form the geometric correlation model. A regularization constraint ensures that the estimated correlation is consistent and corresponds to the actual motion of visual objects. We then use the estimated correlation in a new joint decoding algorithm that approximates the multiple images. The joint decoding is cast as an optimization problem that warps the reference image according to the transformation described in the correlation information, while enforcing the decoded images to be consistent with the quantized measurements. We finally propose an extension of our algorithm to the joint decoding of multi-view images.
While our novel framework could find applications in several problems such as distributed video coding or multi-view imaging, we focus on the latter for illustrating the joint decoding performance. We show by experiments that the proposed algorithm computes a good estimation of the correlation between multi-view images. In particular, the results confirm that dictionaries based on geometric basis functions permit to capture the correlation more efficiently than a dictionary built on patches or blocks from the reference image . In addition, we show that the estimated correlation model can be used to decode the compressed image by disparity compensation. Such a decoding strategy permits to outperform independent coding solutions based on JPEG 2000 and state-of-the-art distributed coding schemes based on disparity learning [6, 7] in terms of rate-distortion (RD) performance due to accurate correlation estimation. Finally, the experiments outline that enforcing consistency in image prediction is very effective in increasing the decoding quality when the images are given by the quantized linear measurements.
The rest of this paper is organized as follows. Section II briefly overviews the related work with a emphasis on reconstruction from random projections and distributed coding algorithms. The geometric correlation model used in our framework is presented in Section III. Section IV describes the proposed regularized energy model for an image pair and the optimization algorithm. The consistent image prediction algorithm is described in Section V. Section VI describes the extension of our scheme to multi-view images. Finally, experimental results are presented in Section VII and Section VIII concludes this paper.
Ii Related work
We present in this section a brief overview of the related works in distributed image coding, where we mostly focus on simple sensing solutions based on linear measurements. In recent years, signal acquisition based on random projections has actually received a significant attention in many applications like medical imaging, compressive imaging or sensor networks. Donoho  and Candes et al.  have shown that a small number of linear measurements contain enough information to reconstruct a signal, as long as it has sparse representation in a basis that is incoherent with the sensing matrix . Rauhut et al.  extend the concept of signal reconstruction from linear measurements using redundant dictionaries. Signal reconstruction from linear measurements has been applied to different applications such as image acquisition [10, 11, 12] and video representation [13, 14, 15].
At the same time, the key in effective distributed representation certainly lies in the definition of good correlation models. Duarte et al. [16, 17] have proposed different correlation models for the distributed compression of correlated signals from linear measurements. In particular, they introduce three joint sparsity models (JSM) in order to exploit the inter-signal correlation in the joint reconstruction. These three sparse models are respectively described by (i) JSM-1, where the signals share a common sparse support plus a sparse innovation part specific to each signal; (ii) JSM-2, where the signals share a common sparse support with different coefficients, and (iii) JSM-3 with a non-sparse common signal with individual sparse innovation in each signal. These correlation models permit a joint reconstruction with a reduced sampling rate or equivalently a smaller number of measurements compared to independent reconstruction for the same decoding quality. The sparsity models developed in  have then been applied to distributed video coding [18, 19] with random projections. The scheme in  used a modified gradient projection sparse algorithm  for the joint signal reconstruction. The authors in  have proposed a distributed compressive video coding scheme based on the sparse recovery with decoder side information. In particular, the prediction error between the original and side information frames is assumed to be sparse in a particular orthonormal basis (e.g., wavelet basis). Another distributed video coding scheme has been proposed in , which relies on an inter-frame sparsity model. A block of pixels in a frame is assumed to be sparsely represented by linear combination of the neighboring blocks from the decoded key frames. In particular, an adaptive block-based dictionary is constructed from the previously decoded key frames and eventually used for signal reconstruction. Finally, iterative projection methods are used in [21, 22] in order to ensure a joint reconstruction of correlated images that are sparse in a dual tree wavelet transform basis and at the same time consistent with the linear measurements in multi-view settings. In general, state-of-the-art distributed compressive schemes [18, 19, 21, 22] estimates the correlation model from two reconstructed reference images, where the reference frames are reconstructed from the respective linear measurements by solving an -TV or - optimization problem. Unfortunately, reconstructing the reference images based on solving an - or -TV optimization problem is computationally expensive [3, 4]. Also, the correlation model estimated from highly compressed reference images usually fails to capture the actual geometrical relationship between images. Motivated by these issues, we estimate in this paper a robust correlation model directly from the highly compressed linear measurements using a reference image, without explicitly reconstructing the compressed images.
In multi-view imaging or distributed video coding, the correlation is explained by the motion of objects or the change of viewpoint. Block-based translation models that are commonly used for correlation estimation fail to efficiently capture the geometry of objects. This results in poor correlation model especially with low resolution images. Furthermore, most of the above mentioned schemes (except 
) assume that the signal is sparse in a particular orthonormal basis (e.g., DCT or Wavelet). This is also the case of the JSM models described above which cannot be used to relate the scene objects by means of a local transform, and unfortunately fail to provide an efficient joint representation of correlated images at low bit rates. It is more generic to assume the signals to be sparse in a redundant dictionary which allows greater flexibility in the design of the representation vectors. The most prominent geometric components in the images can be captured efficiently by dictionary functions. The correlation can be then estimated by comparing the most prominent features in different images. Few works have been reported in the literature for the estimation of a correlation model using redundant structured dictionaries in multi-view or video applications . However, these frameworks do not construct the correlation model from the linear measurements. In general, most of the schemes in classical disparity and motion estimation focus on estimating correlation from original images [25, 26], and not from compressed images. We rather focus here on estimating the correlation from compressed images where the image is given with random linear measurements. The correlation model is built using the geometric transformations captured by a structured dictionary which leads to an effective estimation of the geometric correlation between images.
Finally, the distributed schemes in the literature that are based on compressed measurements usually fail to estimate the actual number of bits for the image sequence representation (except ), and hence cannot be applied directly in practical coding applications. Quantization and entropy coding of the measurements is actually an open research problem due to the two following reasons: (i) the reconstructed signal from quantized measurements does not necessarily satisfy the consistent reconstruction property ; (ii) the entropy of the measurements is usually large which leads to unsatisfactory coding performance in imaging applications . Hence, it is essential to adapt the quantization techniques and reconstruction algorithms in order to reduce the distortion in the reconstructed signal such as [29, 30]. The authors in [31, 32] have also studied the asymptotic reconstruction performance of the signal under uniform and non-uniform quantization schemes. They have shown that a non-uniform quantization scheme usually gives smaller distortion in the reconstruction signal comparing to a uniform quantization scheme. Recently, optimal quantization strategy for the random measurements has been designed based on distributed functional scalar quantizers . In this paper, we use a simple quantization strategy for realistic compression along with consistent prediction constraints in the joint decoding of correlated images in order to illustrate the potential of low complexity sensing solutions in practical multi-view imaging applications.
We consider a pair of images and (with resolution ) that represent a scene taken from different viewpoints; these images are correlated through motion of visual objects. The captured images are encoded independently and are transmitted to a joint decoder. The joint decoder estimates the relative transformations between the received signals and jointly decodes the images. While the description is given here for pairs of images, we later extend the framework to multiple images in Section VI.
We focus on the particular problem where one of the images serves as a reference for the correlation estimation and the decoding of second image as illustrated in Fig. 1. While the reference image could be encoded with any compression algorithm (e.g., JPEG, compressed sensing framework ), we choose here to encode the reference image with JPEG 2000 coding solutions. Next, we concentrate on the independent coding and joint decoding of the second image, where the first image serves as side information. The second image
is projected on a random matrixto generate the measurements . The measurements are quantized with a uniform quantization algorithm and the quantized linear measurements are finally compressed with an entropy coder.
At the decoder, we first estimate the prominent visual features that carry the geometry information of the objects in the scene. In particular, the decoder computes a sparse approximation of the image using a parametric dictionary of geometric functions. Such an approximation captures the most prominent geometrical features in the image . We then estimate the corresponding features in the second image directly from the quantized linear measurements without explicit image reconstruction. In particular, the corresponding features between images are related using a geometry-based correlation model, where the correspondences describe local geometric transformations between images. The correlation information is further used to decode the compressed image from the reference image . We finally ensure a consistent prediction of by explicitly considering the quantized measurements during the warping process. Before getting into the details of the correlation estimation algorithm, we describe the sparse approximation algorithm and the geometry-based correlation model built on a parametric dictionary.
We describe now the geometric correlation model that is based on matching the sparse geometric features in different images. We first compute a sparse approximation of the reference image using geometric basis functions in a structured dictionary where is called an atom. The dictionary is typically constructed by applying geometric transformations (given by the unitary operator ) to a generating function to form the atom . A geometric transformation indexed by consists of a combination of operators for anisotropic scale by , rotation by , and translation by . For example, when is a Gaussian function , the transformation is expressed as
In Fig. 2 we illustrate Gaussian atoms for different translation, rotation and anisotropic scaling parameters. Now, we can write the linear approximation of the reference image with functions in as
where represents the coefficient vector. The number of atoms used in the approximation of is usually much smaller than the dimension of the image . We use here a suboptimal solution based on matching pursuit [34, 35] in order to estimate the set of atoms.
The correlation between images can now be described by the geometric deformation of atoms in different images [23, 24]. Once the reference image is approximated as given in Eq. (2), the second image could be approximated with transformed versions of the atoms used in the approximation of . We can thus approximate as
where represents a local geometrical transformation of the atom . Due to the parametric form of the dictionary it is interesting to note that the transformation on boils down to a transformation of the atom parameters, i.e.,
For clarity, we show in Fig. 3 a sample synthetic correlated image pair and their sparse approximations using atoms in the dictionary. We see that the sparse approximations of images can be described with the transform of atom parameters.
The true transformations however are unknown in practical distributed coding applications. Therefore, the main challenge in our framework consists in estimating the local geometrical transformations when the second image is only available in the form of quantized linear measurements .
Iv Correlation estimation from compressed linear measurements
Iv-a Regularized optimization problem
We describe now our optimization framework for estimating the correlation between images. Given the set of atoms that approximate the first image , the correlation estimation problem consists in finding the corresponding visual patterns in the second image that is given only by compressed random measurements . This is equivalent to finding the correlation between images and with the joint sparsity model based on local geometrical transformations, as described in Section III.
In more details, we are looking for a set of atoms in that correspond to the visual features selected in the first image. We denote their parameters by where = for some , . We propose to select this set of atoms in a regularized energy minimization framework as a trade-off between efficient approximation of and smoothness or consistency of the local transformations between images. The energy model proposed in our scheme is expressed as
where and represent the data and smoothness terms, respectively. The regularization parameter balances the importance of the data and smoothness terms. The solution to our correlation estimation is given by the set of atom parameters that minimizes the energy , i.e.,
The parameter represents the search space given by
The multidimensional search window is defined as where determine the window size for each of the atom parameters (i.e., translations , rotation and scales ). Even if our formulation is able to handle complex transformations, they generally take the form of motion vectors or disparity information in video coding or stereo imaging applications. The label sets and the search space are drastically reduced in this case. The terms used in OPT-1 are described in the next paragraphs.
Iv-B Data cost function
The data cost function computes (in the compressed domain) the accuracy of the sparse approximation of the second image with geometric atoms linked to the reference image. The decoder receives the measurements that are computed by the quantized projections of onto a sensing matrix . For each set of atom parameters the data term reports the error between measurements and orthogonal projection of onto that is formed by the compressed versions of the atoms, i.e., . It turns out that the orthogonal projection of is given as , where represents the pseudo-inverse operator. More formally, the data cost is computed using the following relation:
The data cost function given in Eq. (7) therefore first calculates the coefficients and then measures the distance between the observations and . In other words, the data cost function accounts for the intensity variations between images by estimating the coefficients of the warped atoms.
When the measurements are quantized, the coefficient vector fails to properly account for the error introduced by the quantization. The quantized measurements only provide the index of the quantization interval containing the actual measurement value and the actual measurement value could be any point in the quantization interval. Let be the coordinate of the original measurement and be the corresponding quantized value. It can be noted that the joint decoder has only access to the quantized value and not the original value . Henceforth, the joint decoder knows that the quantized measurement lies within the quantization interval, i.e., where and define the lower and upper bounds of the quantizer bin . We therefore propose to refine the data term in the presence of quantization by computing a coefficient vector as the most consistent coefficient vector when considering all the possible measurement vectors that can result in the quantized measurements vector . In more details, the quantized measurements can be produced by all the observation vectors , where is the Cartesian product of all the quantized regions , i.e., . The data cost term given in Eq. (7) can thus be modified as
Therefore, the robust data term first jointly estimates the coefficients and the measurements , and then computes the distance between the and . It can be shown that the Hessian of the objective function in Eq. (8) is positive semidefinite, i.e., , and hence the objective function is convex. Also, the region forms a closed convex set as each region , forms a convex set. Henceforth, the optimization problem given in Eq. (8) is convex, which leads to effective solutions.
Iv-C Smoothness cost function
The goal of the smoothness term in OPT-1 is to regularize the atom transformations such that the transformations are coherent for neighbor atoms. In other words, the atoms in a spatial neighborhood are likely to undergo similar transformations, when the correlation between images is due to object or camera motion. Instead of penalizing directly the transformation to be coherent for neighbor atoms, we propose to generate a dense disparity (or motion) field from the atom transformations and to penalize the disparity (or motion) field such that it is coherent for adjacent pixels. This regularization is easier to handle than a regular set of transformations and directly corresponds to the physical constraints that explain the formation of correlated images.
In more details, for a given transformation value at pixel we compute the horizontal component and vertical component of the motion field as
where represent the Euclidean coordinates. The matrices , and represent the grid transformations due to scale, rotation and translation changes respectively. They are defined as
Finally, the smoothness cost in OPT-1 is given as
where are the adjacent pixel locations and is the usual 4-pixel neighborhood. The term in Eq. (10) captures the distance between local transformations in neighboring pixels. It is defined as
Iv-D Optimization algorithm
We describe now the optimization methodology that is used solve OPT-1. Recall that our objective is to assign a transformation to each atom in the reference image in order to build a set of smooth local transformations that is consistent with the quantized measurements . The candidate transformations are chosen from a finite set of labels where , , , and refer to the label sets corresponding to translation along and directions, rotations and anisotropic scales respectively (see Eq. (6)). One could use an exhaustive search on the entire label to solve OPT-1. However, the cost for such a solution is high as the size of the label set grows exponentially with the size of the search windows . Rather than doing an exhaustive search, we use graph-based minimization techniques that converge to strong local minima or global minima in a polynomial time with tractable computational complexity [36, 37].
Usually in Graph Cut algorithms a graph is constructed using set of vertices and edges . The set of vertices are given as , where define of nodes corresponding to the pixels in the images (p-nodes) and define the label nodes (l-nodes), as shown in Fig. 4. The p-nodes that are in the neighborhood are connected by an edge called n-link. The cost of n-link usually corresponds to the penalty of assigning different labels to the adjacent pixels as given by . Also, each p-vertex is connected to the l-node by an edge called t-link. The cost of a t-link connecting a pixel and a label corresponds to the penalty of assigning the corresponding label to that pixel; this cost is normally derived from the data term. The final solution is given by a multi-way cut that leaves each p-vertex connected with exactly one t-link. For more details we refer the reader to .
In order to solve our OPT-1 problem, we first need to map our cost functions on the graph in order to assign weights to the n-links and t-links. For a given pair of transformation labels at pixels and , it is straightforward to calculate the weights of the n-links using Eq. (11). It should be noted that the motion field for a given label is computed using Eq. (9). We now describe how to calculate the cost of the t-links based on the data cost . Let be the set of pixels in the support of the atom that is given as
where is a constant. Using this definition, we calculate the t-link penalty cost of connecting a label node to all the pixel nodes in the support of the atom as given in Eq. (7), where . That is, the t-link cost computed between the label and pixels is . However, due to atom overlapping the pixels in the overlapping region could be assigned more than one label. In such cases, we compute the cost corresponding to the index of the atom that has the maximum atom response. The index is computed as
where is the response of the atom at the location , i.e., . After mapping the cost functions on the graph we calculate the correlation solution using a max-flow/min-cut algorithm . Finally, the data term in OPT-1 can be replaced with the robust data term given in Eq. (8) in order to provide robustness to quantization errors. The resulting optimization problem can be efficiently solved using Graph Cut algorithms as described above.
Iv-E Complexity considerations
We discuss now briefly the computational complexity of our correlation estimation algorithm which can basically be divided into two stages. The first stage finds the most prominent features in the reference image using sparse approximations in a structured dictionary. The second stage estimates the transformation for all the features in the reference image by solving a regularized optimization problem OPT-1.
Overall, our framework offers a very simple encoding stage with image acquisition based on random linear projections. The computational burden is shifted to the joint decoder which can still trade-off complexity and performance. Even if the decoder is able to handle computationally complex tasks in our framework, the complexity of our system stays reasonable due to the efficiency of Graph Cuts algorithms whose complexity is bounded by a low order polynomial [36, 37]. The complexity can be further reduced in both stages compared to the generic implementation proposed above. For example, the complexity of the sparse approximation of the reference image can be reduced significantly using a tree-structured dictionary, without significant loss in the approximation performance . In addition, a block-based dictionary can be used in order to reduce the complexity of the transformation estimation problem with block-based computations. Experiments show however that this comes at a price of a performance penalty in the reconstruction quality. Overall, it is clear that the decoding scheme proposed above offers high flexibility with an interesting trade-off between the complexity and the performance. For example, one might decide to use the simple data cost even when the measurements are quantized; it leads to a simpler scheme but to a reduced reconstruction quality.
V Consistent image prediction by warping
After correlation estimation, one can simply reconstruct an approximate version of the second image by warping the reference image using a set of local transformations that forms the warping operator (see Fig. 1). The resulting approximation is however not necessarily consistent with the quantized measurements ; the measurements corresponding to the projection of the image on the sensing matrix are not necessarily equal to . The consistency error might be significant, because the atoms used to compute the correlation and the warping operator do not optimally handle the texture information.
We therefore propose to add a consistency term in the energy model of OPT-1 and to form a new optimization problem for improved image prediction. The consistency term forces the image reconstruction through the warping operator to be consistent with the quantized measurements. We define this additional term as the norm error between the quantized measurements generated from the reconstructed image and the measurements . The consistency term is written as
where is the quantization operator. In the absence of quantization the consistency term simply reads as
We then merge the three cost functions , and with regularization constants and in order to form a new energy model for consistent reconstruction. It is given as
We now highlight the differences between the terms and used in OPT-2. The data cost adapts the coefficient vector to consider the intensity variations between images but it fails to properly handle the texture information. On the other hand, the consistency term warps the atoms by considering the texture information in the reconstructed image but it fails to carefully deal with the intensity variations between images. These two terms therefore impose different constraints on the atom selection that effectively reduce the search space. We have observed experimentally that the quality of the predicted image is maximized when all three terms are activated in the OPT-2 optimization problem.
We propose to use the optimization method based on Graph Cuts described in Section IV-D in order to solve OPT-2. In particular, we map the consistency cost into the graph (see Fig. 4) in addition to the data cost and smoothness cost . For a given , we propose to compute the t-link cost of connecting the label to the pixels as a cumulative sum of . In the overlapping regions, as described earlier we take the value corresponding to the atom index that has maximum response as given in Eq. (13). Then, the n-link weights for the adjacent pixels and are computed based on Eq. (11). After mapping the cost functions on the graph the correlation solution is finally estimated using max-flow/min-cut algorithms . Finally, the data cost in OPT-2 can be again replaced by the robust data term given in Eq. (8). We show later that the performance of our scheme improves by using the robust data term in the presence of quantization. At last, the complexity of estimating the correlation model with OPT-2 problem is tractable, thanks to the efficiency of Graph Cuts algorithms [36, 37].
Vi Correlation estimation of multiple image sets
So far, we have focused on the distributed representation of image pairs. In this section, we describe the extension of our framework to the datasets with correlated images denoted as . Similar to the stereo setup, we consider as the reference image. This image is given in a compressed form and its prominent features are extracted at decoder with a sparse approximation over the dictionary (see Section IV-A). The images are sensed independently using the measurement matrix and their respective measurements are quantized and entropy coded. Our framework can be applied to image sequences or multi-view imaging. For the sake of clarity, we focus on a multi-view imaging framework where the multiple images are captured from different viewpoints.
We are interested in estimating a depth map that captures the correlation among images by assuming that the camera parameters are given a priori. The depth map is constructed using the set of features in the reference image and the quantized measurements . We assume that the depth values are discretized such that the inverse depth is uniformly sampled in the range where and are the minimal and maximal depth in the scene, respectively . The problem is equivalent to finding a set of labels that effectively captures the depth information for each atom or pixel in the reference image, where is a discrete set of labels corresponding to different depths. We propose to estimate the depth information with an energy minimization problem OPT-3 which includes three cost functions as follows:
where represent the data, smoothness and consistency terms respectively. These three terms are balanced with regularization constants and .
The data term assigns a set of labels respectively to the atoms while respecting consistency with the quantized measurements. It reads as
where . The operator represents the projection of the atom to the view when the local transformation is given by the depth label (see Fig. 5). It can be noted that the data term in Eq. (16) is similar to the data term described earlier for image pairs (see Eq. (7)) except that the sum is computed for all the views. Depending on the relative position of the camera with respect to the reference camera, the projection can involve changes in the translation, rotation or scaling parameter, or combinations of them. Therefore, the projection of the atom to the view approximately corresponds to another atom in the dictionary . It is interesting to note that the data cost is minimal if the projection of the atom onto another view corresponds to its actual position in this view111we assume here that we have no occlusions.. This happens when the depth label corresponds to the true distance to the visual object represented by the atom . For example, the projection of the feature in Fig. 5 corresponds to the actual position of the features in views and . Therefore, the data cost for this feature at label =2 is minimal. On the other hand, the projection of the feature is far from the actual position of the corresponding feature in the view . The corresponding data cost is high in this case which indicates a suboptimal estimation of the depth label .
The smoothness cost enforces consistency in the depth label for the adjacent pixels and . It is given as
where is a constant and represents the usual 4-pixel neighborhood. Finally, the consistency term favors depth labels that lead to image predictions that are consistent with the quantized measurements. We compute the consistency for all the views as the cumulative sum of terms given in Eq. (14). More formally, the consistency term in the multi-view scenario is computed as
where predicts the view using the set of labels and the set of atoms . Finally, the OPT-3 optimization problem can be solved in polynomial time using the graph-based optimization methodologies described in Section IV-D. In this case, the weights to the t-links connecting between the label and the pixels are assigned as . The n-link cost for the neighboring pixels is assigned as .
Vii Experimental results
In this section, we report the performance of the correlation estimation algorithms in stereo and multi-view imaging frameworks. In order to compute a sparse approximation of the reference image at decoder, we use a dictionary that is constructed using two generating functions, as explained in . The first one consists of 2D Gaussian functions in order to capture the low frequency components (see Fig. 2). The second function represents a Gaussian function in one direction and the second derivative of a Gaussian in the orthogonal direction in order to capture the edges. The discrete parameters of the functions in the dictionary are chosen as follows. The translation parameters and take any positive value and cover the full height and width of the image. Ten rotation parameters are used between 0 and with increments of . Five scaling parameters are equi-distributed in the logarithmic scale from to vertically, and to horizontally. The image is captured by random linear projections using a scrambled block Hadamard transform with a block size of 8 . The measurements are quantized using a uniform quantizer and the bit rate is computed by encoding the quantized measurements using an arithmetic coder. Unless stated differently, the parameters and in the optimization problems are selected based on trial and error experiments such that the estimated transformation field maximizes the quality of the predicted image .
Vii-B Generic transformation
We first study the performance of our scheme with a pair of synthetic images that contains three objects. The original images and are given in Fig. 6(a) and Fig. 6(b) respectively. It is clear that the common objects in the images have different positions and scales. The absolute error between the original images is given in Fig. 6(c), where the PSNR between and is found to be 15.6 dB.
We encode the reference image to a quality of dB and the number of features used for the approximation of is set to . The transformation field is estimated with pixels, samples and . We first estimate the transformation field with the OPT-1 problem by setting , i.e., the smoothness term is not activated. The resulting motion field is shown in Fig. 6(d). From Fig. 6(d) we observe that the proposed scheme gives a good estimation of the transformation field even with a measurement rate that are quantized with bits. We further see that the image predicted with help of the estimated correlation information is closer to the original image than to (see Fig. 6(e)). We then include the consistency term in addition to the data cost and we solve the problem OPT-2 without activating the smoothness term, i.e., . The estimated transformation field and the prediction error are shown in Fig. 6(f) and Fig. 6(g), respectively. We observe that the consistency term improves the quality of the motion field and the prediction quality. Finally, we highlight the benefit of enforcing smoothness constraint in our OPT-2 problem. The estimated transformation field with the OPT-2 problem including the smoothness term is shown in Fig. 6(h). By comparing the motion fields in Fig. 6(d) and Fig. 6(f) we see that the motion field in Fig. 6(h) is smoother and more coherent; this confirms the benefit of the smoothness term. Quantitatively, the smoothness energy of the motion field shown in Fig. 6(h) is 1479, which is clearly smaller comparing to the solutions given Fig. 6(d) and Fig. 6(f) (resp. 4309 and 4851). Also, the smoothness term effectively improves the quality of the predicted image and the predicted image gets closer to the original image as shown in Fig. 6(i).
Vii-C Stereo image coding
We now study the performance of our distributed image representation algorithms in stereo imaging frameworks. We use two datasets, namely Plastic and Sawtooth222These image sets are available at http://vision.middlebury.edu/stereo/data/ . The images are downsampled to a resolution , (original resolution of the datasets are and respectively). We carry out experiments using the views 1 and 3 for the Plastic dataset and views 1 and 5 for the Sawtooth dataset. These datasets have been captured by a camera array where the different viewpoints are uniformly arranged on a line. As this corresponds to translating the camera along one of the image coordinate axis, the disparity estimation problem becomes a one-dimensional search problem and the smoothness term in Eq. (10) is simplified accordingly. The viewpoint 1 is selected as the reference image and it is encoded such that the quality of is approximately dB. Matching pursuit is then performed on with and atoms for the Plastic and Sawtooth datasets respectively. The measurements on the second image are quantized using a 2-bit quantizer. At the decoder, the search for the geometric transformations is carried out along the translational component with window size pixels and no search is consider along the vertical direction, i.e., . Unless stated explicitly, we use the data cost given in Eq. (7) in the OPT-1 and OPT-2 problems.
We first study the accuracy of the estimated disparity information. In Fig. 7 we show the estimated disparity field from quantized measurements (i.e., a measurement rate of %) for the Plastic dataset. The groundtruth is given in Fig. 7(a). The transformation is estimated by solving OPT-1 and the resulting dense disparity field is illustrated in Fig. 7(b). In this particular experiment, the parameter is selected such that the error in the disparity map is minimized. The disparity error DE is computed between the estimated disparity field and the groundtruth as where represents the pixel resolution of the image . From Fig. 7(b) we observe that OPT-1 gives a good estimation of the disparity map; in particular the disparity value is correctly estimated in the regions with texture or depth discontinuities. We could also observe that the estimation of the disparity field is however less precise in smooth regions as expected from feature-based methods. Fortunately, the wrong estimation of the disparity value corresponding to the smooth region in the images does not significantly affect the warped or predicted image quality . Fig. 7(c) confirms such a distribution of the disparity estimation error where the white pixels denote an estimation error larger than one. We can see that the error in the disparity field is highly concentrated along the edges, since crisp discontinuities cannot be accurately captured due to the scale and smoothness of the atoms in the chosen dictionary. The disparity information estimated by OPT-2 is presented in Fig. 7(d) and the corresponding error is shown in Fig. 7(e). In this case, the regularization constants and in the OPT-2 problem are selected such that the DE is minimized. We see that the addition of the consistency term in the correlation estimation algorithm improves the performance.
We then study the rate-distortion (RD) performance of the proposed algorithms in the prediction of the image in Fig. 8. We show the performance of the reconstruction by warping the reference image according to the correlation computed by OPT-1 and OPT-2. We then highlight the benefit of using the robust data term in OPT-2 problem (denoted as OPT-2 (Robust)). We use the optimization toolbox based on CVX  in order to solve the optimization problem given in Eq. (8
). We then compare the RD performance to a distributed coding solution (DSC) based on the LDPC encoding of DCT coefficients, where the disparity field is estimated at the decoder using Expected Maximization (EM) principles (denoted as Disparity learning). Then, in order to demonstrate the benefit of geometric dictionaries we propose a scheme denoted as block-based that adaptively constructs the dictionary using blocks or patches in the reference image . We construct a dictionary in the joint decoder from the reference image segmented into blocks. The search window size is pixels along the horizontal direction. We then use the optimization scheme described in OPT-2 to select the best block from the adaptive dictionary. In order to have a fair comparison, we encode the reference image similarly for both schemes (Disparity learning and block-based) with a quality of dB (see Section III). Finally, we also provide the performance of a standard JPEG 2000 independent encoding of the image . From Fig. 8, we first see that the measurement consistency term significantly improves the decoding quality, as OPT-2 gives better performance than OPT-1. We further see that the OPT-2 problem with robust data cost improves the quality of the reconstructed image by - dB at low bit rates. Then, the results confirm that the proposed algorithms unsurprisingly outperform independent coding based on JPEG 2000, which outlines the benefits of the use of correlation in the decoding of compressed correlated images. At high rate, the performance of the proposed algorithms however tends to saturate as our model mostly handles the geometry and the correlation between images; but it is not able to efficiently handle the fine details or texture in the scene due to the image decoding based on warping. From Fig. 8, it is then clear that the reconstruction of image based on OPT-1 and OPT-2 outperforms the DSC coding scheme based on EM principles due to the accurate correlation estimation. It is worth mentioning that state-of-the-art DSC scheme based on disparity learning compensate also for the prediction error in addition to correlation estimation. Though this is the case, our scheme outperforms DSC scheme due to an accurate disparity field estimation. Finally, the experimental results also show that our schemes outperform the scheme based on block-based dictionary mainly because of the richer representation of the geometry and local transformations with the structured dictionaries.
We then study the influence of the quality of reference image on the reconstruction performance. We use OPT-2 to reconstruct (viewpoint 5) by warping when the reference image has been encoded at different qualities (i.e., different bit rates). Fig. 9 shows that the reconstruction quality improves with the quality of the reference image as expected. While we have observed that the error in the disparity estimation is not dramatically reduced by improved reference quality, the warping stage permits to provide more details in the representation of when the reference is of better quality. Finally, we study the overall RD performance for the Sawtooth dataset between views 1 and 5 that also includes the bit rate and quality of the reference image, in addition to the rate and quality of image . Fig. 10 shows the overall RD performance at reference image bit rates and bpp. In our experiments, for a given reference image quality we estimate the correlation model using OPT-2 (with 2-bit quantized measurements), and we compute the overall RD performance at that specific reference image bit rate. As shown before, the RD performance improves with increasing reference image quality. When we take the convex hull of the RD performances (which corresponds to implementing a proper rate allocation strategy), we outperform independent coding solutions based on JPEG 2000.
We now study the influence of the quantization bit rate on the RD performance of with the OPT-2 optimization scheme. We compress the measurements using -, - and -bits uniform quantizers. As expected, the quality of the correlation estimation degrades when the number of bits reduces as shown in Fig. 11(a). However, it is largely compensated by the reduction in bit rate in the RD performance as confirmed by Fig. 11(b). This means that the proposed correlation estimation is relatively robust to quantization so that it is possible to attain good RD performance by drastic quantization of the measurements. Finally, we study the improvement offered by the robust data term (see Eq. (8)) in OPT-2, when the measurements have been compressed with a 2-bit uniform quantizer. From Fig. 11(a) it is clear that the proposed robust data term improves the performance due to the efficient handling of noise in the quantized measurements.
Vii-D Multi-view image representation
We finally evaluate the performance of our multi-view correlation estimation algorithms using five images from the Tsukuba dataset (center, left, right, bottom and top views), and five frames (frames -) from the Flower Garden sequence . These datasets are down-sampled by a factor 2 and the resolution used in our experiments are of and pixels respectively. In both datasets, the reference image (center view and frame resp.) is encoded with a quality of approximately dB. The measurements computed from the remaining four images are quantized using a -bit quantizer. We first compare our results to a stereo setup where the disparity information is estimated with OPT-2 between the center and left images in Tsukuba dataset. Fig. 12 compares the inverse depth error (sum of the labels with error larger than one with respect to groundtruth) between the multi-view and stereo scenarios. In this particular experiment, the parameters and are selected such that they minimize the error in the depth image with respect to the groundtruth. It is clear from the plot that the depth error is small for a given measurement rate when all the views are available. It should be noted that the -axis in Fig. 12 represents the measurement rate per view. Hence, the total number of measurements used in the multi-view scenario is higher than for the stereo case. However, these experiments show that the proposed multi-view scheme gives a better depth image when more images are available. Similar experimental findings have been observed for the Flower Garden sequence.
We then study the RD performance of the proposed multi-view scheme in the decoding of four images (top, left, right, bottom images in the Tsukuba and frames in the Flower Garden). The images are decoded by warping the reference image using the estimated depth image. Fig. 13 compares the overall RD performance (for 4 images) of our multi-view scheme with respect to independent coding performance based on JPEG 2000. As expected, the proposed multi-view scheme outperforms independent coding solutions based on JPEG 2000 as it benefits from the correlation between images. Furthermore, as observed in distributed stereo coding the proposed multi-view coding scheme saturates at high rates, as the warping operator captures only the geometry and correlation between images and not the texture information.
Finally, we compare our results with a joint encoding approach where the depth image is estimated from the original images and transmitted to the joint decoder. At the decoder, the views are predicted from the reconstructed reference image and the compressed depth image with the help of view prediction. The results are presented in Fig. 13 (denoted as Joint Encoding), where the bit rate is computed only on the depth image encoded using a JPEG 2000 coding solution. The main difference between the proposed and joint encoding frameworks is that the quantized linear measurements are transmitted for a depth estimation in the former scheme, while the depth information is directly transmitted in the latter scheme. Therefore, by comparing these two approaches we can judge the accuracy of the estimated correlation model or equivalently the quality of the predicted view at a given bit rate. From Fig. 13 we see that at low bit rate , the proposed scheme estimates a better structural information compared to the joint encoding scheme, thanks to the geometry-based correlation representation. However at rates above , we see that our scheme becomes comparable with joint coding solutions. This leads to the conclusion that the proposed scheme effectively estimates the depth information from the highly compressed quantized measurements. It should be noted that in joint encoding framework the depth images are estimated at a central encoder. In contrary to this, we estimate the depth images at the central decoder from the independently compressed visual information; this advantageously reduces the complexity at the encoder which makes it attractive for distributed processing applications.
In this paper, we have presented a novel framework for the distributed representation of correlated images with quantized linear measurements, along with joint decoding algorithms that exploit the geometrical correlation among multiple images. We have proposed a regularized optimization problem in order to identify the geometrical transformations between compressed images, which result in smooth disparity or depth fields between a reference and one or more predicted image(s). We have proposed a low complexity algorithm for the correlation estimation problem which offers an effective trade-off between the complexity and accuracy of the solution. In addition, we have proposed a new consistency criteria such that transformations are consistent with the compressed measurements in the predicted image. Experimental results demonstrate that the proposed methodology provides a good estimation of dense disparity/depth fields in different multi-view image datasets. We also show that our geometry-based correlation model is more efficient than block-based correlation models. Finally, the consistent constraints prove to offer effective decoding quality such that the proposed algorithm outperforms JPEG 2000 and DSC schemes in terms of rate-distortion performance, even if the images are reconstructed by warping. This clearly positions our scheme as an effective solution for distributed image processing with low encoding complexity.
-  V. Thirumalai and P. Frossard, “Motion estimation from compressed linear measurements,” Proc. IEEE International Conference on Acoustics, Speech and Signal Processing, 2010.
-  ——, “Joint reconstruction of correlated images from compressed linear measurements,” Proc. European Signal Processing Conference, 2010.
-  D. Donoho, “Compressed sensing,” IEEE Trans. Information Theory, vol. 52, pp. 1289–1306, 2006.
-  E. J. Candes, J. Romberg, and T. Tao, “Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. Information Theory, vol. 52, pp. 489–509, 2006.
-  J. P. Nebot, Y. Ma, and T. Huang, “Distributed video coding using compressive sampling,” in Proc. Picture Coding Symposium, 2009.
D. Varodayan, Y. C. Lin, A. Mavlankar, M. Flierl, and B. Girod, “Wyner-ziv coding of stereo images with unsupervised learning of disparity,” inProc. Picture Coding Symposium, 2007.
-  D. Varodayan, D. Chen, M. Flierl, and B. Girod, “Wyner-ziv coding of video with unsupervised motion vector learning,” EURASIP Signal Processing: Image Communication, vol. 23, pp. 369–378, 2008.
-  E. J. Candes and J. Romberg, “Practical signal recovery from random projections,” in Proc. SPIE Computational Imaging, 2005.
-  H. Rauhut, K. Schnass, and P. Vandergheynst, “Compressed sensing and redundant dictionaries,” IEEE Trans. Information Theory, vol. 54, pp. 2210–2219, 2006.
-  M. Duarte, M. Davenport, D. Takhar, J. Laska, T. Sun, K. Kelly, and R. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Processing Magazine, vol. 25, no. 2, pp. 83–91, 2008.
-  S. Mun and J. Fowler, “Block compressed sensing of images using directional transforms,” Proc. IEEE International Conference on Image Processing, 2009.
-  L. Gan, T. T. Do, and T. D. Tran, “Fast compressive imaging using scrambled hadamard ensemble,” in Proc. European Signal and Image Processing Conference, 2008.
-  V. Stankovic, L. Stankovic, and S. Cheng, “Compressive video sampling,” in Proc. European Signal and Image Processing Conference, 2008.
-  J. Y. Park and M. B. Wakin, “A multiscale framework for compressive sensing of video,” in Proc. Picture Coding Symposium, 2009.
N. Vaswani, “Kalman filtered compressed sensing,” inProc. IEEE International Conference on Image Processing, 2008.
-  M. F. Duarte, S. Sarvotham, D. Baron, M. B. Wakin, and R. G. Baraniuk, “Distributed compressed sensing of jointly sparse signals,” in Proc. Asilomar Conference on Signal System and Computing, 2005.
-  ——, “Universal distributed sensing via random projections,” in Proc. Information Processing in Sensor Networks, 2006.
-  L. W. Kang and C. S. Lu, “Distributed compressive video sensing,” in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing, 2009.
-  T. T. Do, Y. Chen, D. T. Nguyen, N. Nguyen, L. Gan, and T. D. Tran, “Distributed compressed video sensing,” in Proc. IEEE International Conference on Image Processing, 2009.
-  M. A. T. Figueiredo, R. D. Nowak, and S. J. Wright, “Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems,” IEEE Journal of Selected Topics in Signal Processing, vol. 1, pp. 586–597, 2007.
-  M. Trocan, T. Maugey, J. E. Fowler, and B. Pesquet-Popescu, “Disparity-compensated compressed-sensing reconstruction for multiview images,” Proc. IEEE International Conference on Multimedia and Expo, 2010.
-  M. Trocan, T. Maugey, E. W. Tramel, J. E. Fowler, and B. Pesquet-Popescu, “Multistage compressed-sensing reconstruction of multiview images,” Proc. IEEE International workshop on Multimedia Signal Processing, 2010.
I. Tosic and P. Frossard, “Geometry based distributed scene representation with omnidirectional vision sensors,”IEEE Trans. Image Processing, vol. 17, pp. 1033–1046, 2008.
-  O. D. Escoda, G. Monaci, R. M. Figueras, P. Vandergheynst, and M. Bierlaire, “Geometric video approximation using weighted matching pursuit,” IEEE Trans. Image Processing, vol. 18, pp. 1703–1716, 2009.
D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense stereo,”
International Journal on Computer Vision, vol. 47, pp. 7–42, 2002.
-  S. Baker, S. Roth, D. Scharstein, M. Black, J. Lewis, and R. Szeliski, “A database and evaluation methodology for optical flow,” International Journal of Computer Vision, vol. 1, no. 92, pp. 1–31, 2011.
-  P. Boufounos and R. G. Baraniuk, “1-bit compressive sensing,” in Proc. International Conference on Information Science and Systems, 2008.
-  A. Schulz, L. Velho, and E. A. B. da Silva, “On the empirical rate-distortion performance of compressive sensing,” in Proc. IEEE International Conference on Image Processing, 2009.
-  L. Jacques, D. K. Hammond, and M. J. Fadili, “Dequantizing compressed sensing: When oversampling and non-gaussian constraints combine,” IEEE Trans. on Information Theory, vol. 57, pp. 559–571, Jan 2011.
-  A. Zymnis, S. Boyd, and E. Candes, “Compressed sensing with quantized measurements,” IEEE Signal Processing Letters, vol. 17, no. 2, pp. 149–152, 2010.
-  A. Fletcher, S. Rangan, and V. Goyal, “On the rate-distortion performance of compressed sensing,” in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing, 2007.
-  W. Dai, H. V. Pham, and O. Milenkovic. (2009) Distortion-rate functions for quantized compressive sensing. [Online]. Available: http://arxiv.org/abs/0901.0749
-  J. Sun and V. Goyal, “Optimal quantization of random measurements in compressed sensing,” in Proc. IEEE International Symposium on Information Theory, 2009.
-  G. Mallat and Z. Zhang, “Matching pursuits with time-frequency dictionaries,” IEEE Trans. Signal Processing, vol. 41, pp. 3397–3415, 1993.
-  R. M. Figueras, P. Vandergheynst, and P. Frossard, “Low-rate and flexible image coding with redundant representations,” IEEE Trans. Image Processing, vol. 15, pp. 726–739, 2006.
-  Y. Boykov, O. Veksler, and R. Zabih, “Fast approximate energy minimization via graph cuts,” IEEE Trans. on Pattern Analysis and machine intelligence, vol. 23, no. 11, pp. 1222–1239, Jan 2002.
-  Y. Boykov and V. Kolmogorov, “An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision,” IEEE Trans. on Pattern Analysis and machine intelligence, vol. 26, no. 9, pp. 1124–1137, 2004.
-  P. Jost, P. Vandergheynst, and P. Frossard, “Tree-based pursuit: Algorithm and properties,” IEEE Transactions on Signal Processing, vol. 54, no. 12, pp. 4685–4697, 2006.
-  V. Kolmogorov and R. Zabih, “Multi-camera scene reconstruction via graph cuts,” Computer Vision—ECCV 2002, pp. 8–40, 2002.
-  M. Grant and S. Boyd, “CVX: Matlab software for disciplined convex programming, version 1.21,” http://cvxr.com/cvx, Apr. 2011.