Consider a set of
jointly Gaussian memoryless sources with joint probability density function (pdf) known only to belong to a given family of pdfs. A fixed subset ofsources are sampled at each time instant and compressed jointly by a (block) source code, with the objective of reconstructing all the sources within a specified level of distortion under a mean-squared error criterion. “Universality” requires that the sampling and lossy compression code be designed without precise knowledge of the underlying pdf. In this paper we study the tradeoffs – under optimal processing – among sampling, compression rate and distortion level. This study builds on our prior works [3, 4] on sampling rate distortion for multiple discrete sources with known joint pmf and universal sampling rate distortion for multiple discrete sources with joint pmf known only to lie in a finite class of pmfs, respectively. Here, we do not assume the class of pdfs to be finite.
Problems of combined sampling and compression have been studied extensively in diverse contexts for discrete and Gaussian sources. Relevant works include lossless compression of analog sources in an information theoretic setting ; compressed sensing with an allowed detection error rate or quantization distortion ; sub-Nyquist temporal sampling of Gaussian sources followed by lossy reconstruction ; and rate distortion function for multiple sources with time-shared sampling . See also [13, 33].
Closer to our approach that entails spatial sampling, in a setting of distributed acoustic sensing and reconstruction, centralized as well as distributed coding schemes and sampling lattices are studied in . The rate distortion function has been characterized when multiple Gaussian signals from a random field are sampled and quantized (centralized or distributed) in , [24, 25]. In , a Gaussian field on the interval and i.i.d. in time, is reconstructed from compressed versions of -sampled sequences under a mean-squared error criterion. The rate distortion function is studied for schemes that reconstruct only the sampled sources first and then reconstruct the unsampled sources by forming minimum mean-squared error (MMSE) estimates based on the reconstructions for the sampled sources. All the sampling problems above assume a knowledge of the underlying distribution.
In the realm of rate distortion theory where a complete knowledge of the signal statistics is unknown, there is a rich literature that considers various formulations of universal coding; only a sampling is listed here. Directions include classical Bayesian and nonBayesian methods [41, 23, 26, 30]; “individual sequences” studies [42, 35, 36]; redundancy in quantization rate or distortion [20, 18, 19]; and lossy compression of noisy or remote signals [21, 34, 6]. These works propose a variety of distortion measures to investigate universal reconstruction performance.
Our work differs materially from the approaches above. Sampling is spatial rather than temporal. Our notion of universality involves a lack of specific knowledge of the underlying pdf in a given compact family of pdfs. Accordingly, in Bayesian and nonBayesian settings, we consider average and peak distortion criteria, respectively, with an emphasis on the former.
Our technical contributions are as follows. In Bayesian and nonBayesian settings, we extend the notion of a universal sampling rate distortion function (USRDf)  to Gaussian memoryless sources, with the objective of characterizing the tradeoffs among sampling, compression rate and distortion level. To this end, we consider first the setting with known underlying pdf, and characterize its sampling rate distortion function (SRDf). This uses as an ingredient the rate distortion function for a discrete “remote” source-receiver model with known distribution [8, 1, 2, 39]. When the underlying pdf is known, we show that the overall reconstruction can be performed – optimally – in two steps: the sampled sources are reconstructed first under a modified weighted mean-squared error criterion and then MMSE estimates are formed for the unsampled sources based on the reconstructions for the sampled sources. This is akin to the structure observed in  for reconstructing discrete sources from subsets of sources under the probability of error criterion and in  for reconstructing remote Gaussian sources. The USRDf for Gaussian memoryless sources with known pdf will serve as a key ingredient in characterizing the USRDf for the Gaussian field, with known distribution, previously studied in  in a restricted setting. Building on the ideas developed, for the SRDf (with known pdf), we characterize next the USRDf for Gaussian memoryless sources in the Bayesian and nonBayesian settings and show that it remains optimal to reconstruct first the sampled sources and then form estimates for the unsampled sources based on the reconstructions of the sampled sources.
Our model is described in Section II and our main results and illustrative examples are presented in Section III. In Section IV, we present achievability proofs first when the pdf is known and then, building on it, the achievability proof for the universal setting, with an emphasis on the Bayesian setting. A unified converse proof is presented thereafter.
Denote and let
-valued zero-mean (jointly) Gaussian random vector with a positive-definite covariance matrix. For a nonempty setwith , we denote by the random (column) vector , with values in . Denote repetitions of , with values in , by . Each takes values in . Let and let be the reproduction alphabet for . All logarithms and exponentiations are with respect to the base 2 and all norms are -norms.
Let ††footnotetext: is a collection of covariance matrices indexed by . By an abuse of notation, we shall use to refer to the covariance matrix itself. be a set of -positive-definite matrices, and assume to be convex and compact in the Euclidean topology on . For instance, for ,
with and Hereafter, all covariance matrices under consideration will be taken as being positive-definite without explicit mention. We assume to be a -valued rv with a pdf that is absolutely continuous with respect to the Lebesgue measure on . We assume
and that is continuous on . We consider a jointly Gaussian memoryless multiple source (GMMS) consisting of i.i.d. repetitions of the rv with pdf known only to the extent of belonging to the family of pdfs ††footnotetext: Throughout this paper, is used to denote the pdf of a Gaussian random vector with mean and covariance matrix . ,. Two settings are studied: in a Bayesian formulation, the pdf is taken to be known, while in a nonBayesian formulation is an unknown constant in .
For a fixed with , a k-fixed-set sampler (-FS), , collects at each , from . The output of the -FS is .
For , an -length block code with -FS for a GMMS with reproduction alphabet is the pair where the encoder maps the -FS output into some finite set and the decoder maps into . We shall use the compact notation suppressing . The rate of the code with -FS is .
Our objective is to reconstruct all the components of a GMMS from the compressed representations of the sampled GMMS components under a suitable distortion criterion with (single-letter) mean-squared error (MSE) distortion measure
For threshold an -length block code with -FS will be required to satisfy one of the following distortion criterion depending on the setting.
(i) Bayesian: The expected distortion criterion is
(ii) NonBayesian: The peak distortion criterion is
where denotes .
A number is an achievable universal -sample coding rate at distortion level if for every and sufficiently large , there exist -length block codes with -FS of rate less than and satisfying the fidelity criterion in (4) or (5) above; will be termed an achievable universal -sample rate distortion pair under the expected or peak distortion criterion. The infimum of such achievable rates foe each fixed is denoted by . We shall refer to as the universal sampling rate distortion function (USRDf), suppressing the dependence on . For the USRDf is termed simply the sampling rate distortion function (SRDf), denoted by
(ii) When , the pdf of the GMMS is, in effect, known.
Below, we recall (Chapter 1, 
) the definition of mutual information between two random variables.
For real-valued rvs and with a joint probability distribution , the mutual information between the rvs and is given by
where denotes that is absolutely continuous with respect to and is the Radon-Nikodym derivative of with respect to .
We begin with a setting where the pdf of is known and provide a (single-letter) characterization for the SRDf. Next, in a brief detour, we introduce an extension of GMMS, namely a Gaussian memoryless field (GMF) and show how the ideas developed for a GMMS can be used to characterize the SRDf for a GMF. Finally, building on the SRDf for a GMMS, a (single-letter) characterization of the USRDf is provided for a GMMS in the Bayesian and nonBayesian settings.
Throughout this paper, a recurring structural property of our achievability proofs is this: it is optimal to reconstruct the sampled GMMS components first under a (modified) weighted MSE criterion with reduced threshold and then form deterministic (MMSE) estimates of the unsampled components based on the reconstruction of the former.
Before we present our first result, we recall that for a GMMS with pdf reconstructed under the MSE distortion criterion, the standard rate distortion function (RDf) is
s are the eigenvalues of, and is chosen to satisfy
Iii-a : Known pdf
Starting with , for a GMMS with (known) pdf , our first result shows that the fixed-set SRDf for a GMMS is, in effect, the RDf of a GMMS with a weighted MSE distortion measure and a reduced threshold; here is given by
For a GMMS with pdf and fixed , the SRDf is
and s are the eigenvalues of , and is chosen to satisfy .
Comparing (11) with (7), it can be seen that (11) is, in effect, the RDf for a GMMS with weighted MSE distortion measure. In contrast to the RDf (7), in (11) the minimization involves only (and not ) under a weighted MSE criterion with reduced threshold level. For , i.e., , however this reduces to the RDf (7). Also, for every feasible distortion level the SRDf for any is no smaller than that with .
In Section IV, the achievability proof of the theorem above involves reconstructing the sampled components of the GMMS first, and then forming MMSE estimates for the unsampled components based on the former. Accordingly, in (11), the MSE in the reconstruction of the entire GMMS is captured jointly by the weighted MSE (with weight-matrix ) in the reconstructions of the sampled components and the minimum distortion .
Observing that (11) is equivalent to the RDf of a GMMS with a weighted MSE distortion measure enables us to provide an analytic expression for the SRDf using the standard reverse water-filling solution (12) . An instance of this is shown in the example below.
For a GMMS with a -FS with , this example illustrates the effect of the choice of the sampling set on SRDf. Consider a GMMS with covariance matrix given by
where For , we have
and hence from (12), the SRDf is
for where . Observe that every SRDf is a monotonically increasing function of and that the SRDfs are translations of each other and hence decrease at the same rate. Thus, the SRDf with the smallest is uniformly best among all fixed-set SRDfs. For however, there may not be any whose fixed-set SRDf is uniformly best for all distortion levels. ∎
Before turning to the USRDf for a GMMS, the ideas involved in Theorem 1 are used to study sampling and lossy compression of a Gaussian field which affords greater flexibility in the choice of sampling set. While Gaussian fields have been studied extensively under different formulations, we consider a Gaussian memoryless field (GMF) as in , which is described next. In lieu of and Gaussian rv in Section II, consider and let be a -valued zero-mean Gaussian process††footnotetext: A Gaussian process on an interval means that any finite collection of rvs are jointly Gaussian. with a bounded covariance function , such that, for any finite
is a positive-definite matrix and
A GMF††footnotetext: Extensive studies of memoryless repetitions of a Gaussian process exist, cf. , , under various terminologies. consists of i.i.d. repetitions of . We consider a GMF sampled finitely by a -FS at , with , and with a reconstruction alphabet .
For a GMF with fixed-set sampler and MSE distortion measure
Analogous to a GMMS, for a GMF sampled at our next result shows that the SRDf is, in effect, the RDf of a GMMS with a weighted MSE distortion measure with weight-matrix given by
with connoting element-wise integration. Note that for every , (19) and the boundedness of imply that the integral
exists and hence (22) is well-defined.
For a GMF with , the SRDf is
and s are the eigenvalues of , and satisfies .
The SRDf for a GMF (24) and its equivalent form (25) can be seen as counterparts of (11) and (12), with (25) being the reverse water-filling solution for (24). As before, the expression (24) is the RDf of a GMMS with a weighted MSE distortion measure. In Section IV, an achievability proof for the proposition above is provided by adapting the ideas developed for Theorem 1; a converse proof for the proposition is provided involving a set of techniques different from the converse proof provided for Theorem 1.
In contrast to a GMMS with a discrete set , for a GMF, being an interval affords greater flexibility in the choice of the sampling set allowing for a better understanding of the structural properties of the “best” sampling set. In contrast to Example 1 in the example below, considering a GMF with a stationary Gauss-Markov process, we show the structure of the optimal set for minimum distortion for as well. In general, the optimal sampling set is a function of the threshold .
Consider a GMF with a zero-mean, stationary Gauss-Markov process over with covariance function
and . Note that the correlation between any two points in the interval depends only on the distance between them. For the Gauss-Markov process , for any , it holds that
For a -FS with and ,
and . In (25), the eigenvalue is itself and hence, the SRDf is
for , where
Note that the SRDf is a monotonically increasing function of , which in turn is a monotonically increasing function of . Thus, is uniformly best among all SRDfs , , for all distortion levels. Now, for a -FS with and , with the minimum distortion admits a simple form
where is according to
The minimum reconstruction error is the “sum” of the minimum error in reconstructing each segment of the GMF. Now, the Markov property (28) implies that the minimum error in reproducing each component is determined by its nearest sampled points and hence the minimum error in reconstructing each segment of the GMF is independent of the location of sampling points other than , and is given by
The stationarity of the field means that this minimum error depends on the length alone. Observing that is a concave function of over , above is seen to be minimized when , i.e., when the sampling points are spaced uniformly. However, such a placement is not optimal for all distortion levels. ∎
Iii-B Universal setting
Turning to the universal setting with a GMMS, consider a set with indexing the members of , i.e., ††footnotetext: The collection of covariance matrices are indexed by and by an abuse will also be used to refer to itself. . An encoder associated with a -FS observes alone and cannot distinguish among jointly Gaussian pdfs in that have the same marginal pdf . Accordingly (and akin to ), consider a partition of comprising “ambiguity” atoms, with each atom of the partition comprising s with identical , i.e., identical and for each , is the collection of s in the ambiguity atom indexed by , i.e.,
Let be a -valued rv induced by . It is easy to see that and are convex, compact subsets of and the rv admits a pdf induced by .
In the Bayesian setting,
In the nonBayesian setting, in order to retain the same notation, we choose to be the right-side above.
Our characterization of the USRDf builds on the structure of the SRDf for a GMMS. Accordingly, in the Bayesian setting, consider the set of (constrained) probability measures
and (constraint) minimized mutual information
Correspondingly, in the nonBayesian setting, consider
The minimized conditional mutual informations above will be a key ingredient in the characterization of USRDf. First, we show in the proposition below that (40) and (43) admit simpler forms involving rvs corresponding to the sampled components of the GMMS and their reconstruction alone. In the Bayesian setting, for each , the mentioned simpler form involves a weighted MSE distortion measure with weight-matrix , defined as in (10) with replaced by and
In the Bayesian setting, the modified distortion measure plays a role similar to that of .
Remark: Clearly, is a nonincreasing function of Convexity of can be shown as in , and convexity implies the continuity of . Now, to show the convexity, pick any and For let be such that
For by the standard convexity arguments, it can be seen that and
Since (46) holds for any , in the limit, we have
For each , in the Bayesian setting
for , where
For each , in the nonBayesian setting
where the infimum in (50) is over such that
Remark: From (48), notice that is, in effect, the rate distortion function for a GMMS with pdf and weighted MSE distortion measure. Hence, the minimum in (48) and ergo that in (40) exist and the standard properties of a rate distortion function are applicable to as well, i.e., is a convex, nonincreasing, continuous function of .
For a GMMS with fixed , the Bayesian USRDf is
for , where
The nonBayesian USRDf is
for , where
Remark: In Appendix -C a simple proof (using contradiction arguments) is provided to show the existence of