Consider the following continuous-time white Gaussian channel
where denotes the standard Brownian motion, and the channel input is an independent stationary Gaussian process with power spectral density . This paper is to give an elementary proof of the following classical information-theoretic formula (see, e.g., Theorem of )
This renowned formula was first established by Shannon in his seminal work 
through a heuristic yet rather convincing spectrum-splitting argument and then treated more rigorously by numerous authors, predominantly using alternative channel formulations obtained via some orthogonal expansion representations in the relevant Hilbert space. Representative work in this direction include[9, 10, 8, 3, 11], and the heart of all the approaches therein lies in a continuous-time version of the famed Szego’s theorem (see, e.g., the theorem on page of ). In a different direction, there have been efforts devoted to analyze continuous-time Gaussian channels using tools and techniques from stochastic calculus [2, 12, 6, 7], where the channel mutual information has been found to linked to an optimal linear filter. These links, together with well-known results from filtering theory [17, 18], will conceivably recover (2).
It appear to us that all existing treatments either rely on heavy mathematical machineries or have to resort to some “external” results. By comparison, our proof, which hinges on a recently proven sampling theorem (Theorem in ), is elementary and self-contained, only using some well-known facts from basic calculus and matrix theory: it turns out that the aforementioned sampling theorem enables us to sidestep numerous complications that are otherwise present in the continuous-time regime and allows us to employ a spectral analysis of finite-dimensional matrices, rather than infinite-dimensional operators that some previous approaches would have to deal with. Moreover, as elaborated in Section 4, our approach gives rise to a “scalable” version of Szego’s theorem and naturally connects a continuous-time Gaussian channel to its sampled discrete-time versions, and thereby promising further applications in more general settings.
2 A Heuristic Proof
We first explain the aforementioned sampling theorem. For any given and , choose evenly spaced sampling times , , such that and let
Sampling the channel (1) over the time interval with respect to , we obtain its sampled discrete-time version as follows:
Loosely speaking, Theorem in  says that as the above sampling gets increasingly finer, the mutual information of the discrete-time channel (3) will converge to that of the original continuous-time channel (1).
Note that the mutual information of the channel (3) can be computed as
where is the identity matrix and is an matrix whose -th entry is defined as
It then follows from the stationarity of that
where, setting for , we have defined
Noting that is a Hermitian (and Toeplitz) matrix and letting
denote all its eigenvalues, we have
Now consider an matrix defined by
where and , for . Obviously, is an circulant matrix whose eigenvalues can be readily computed as
Now, for large , approximating by , we have, for ,
and for ,
where we have used the definition
Now, collecting all the results above, we conclude that, for appropriately chosen large and large ,
The above proof is by no means rigourous, but, as elaborated in the next section, a refinement with some elementary - arguments and Fourier analysis arguments will certainly make it so to reach (2), which yields a rigorous proof of the classical formula.
3 A Rigorous Proof
First of all, we rigorously state our theorem.
Assume that both and are Lesbegue integrable over . Then,
It is well known that and
are a Fourier transform pair, and the integrability of one implies that the other one is uniformly bounded and uniformly continuous over. Moreover, it is easy to verify that is non-negative, and both and are symmetric.
We next state the sampling theorem that will be used in our proof, which is a weakened version of Theorem in  that holds true in a more general setting where sampling times may not be evenly spaced, and moreover, feedback and memory are possibly involved.
For any fixed and any sequence satisfying for any feasible , we have
We are now ready to give the proof of our main result.
Proof of Theorem 3.1.
Our proof consists of the following several steps.
Step 1. In this step, we show that both and are bounded from above uniformly over all and , namely, there exists such that for all and , . Here denotes the operator norm induced by the
-norm for vectors.
It is straightforward to verify (cf. the proof of Lemma in ) that
where denotes the -norm and
So, to establish the uniform boundedness of , it suffices to prove that is bounded from above uniformly all , and . Towards this end, we note that for any feasible ,
which immediately implies that
which, together with (6), further implies that for any ,
Note that a similar argument as above yields that for all , and ,
which implies the uniform boundedness of , and moreover, together with (6), that of .
Step 2. In this step, we show that both and are bounded from above uniformly over all and . Here denotes the Frobenious norm.
To prove the uniform boundedness of , note that
where, for the last inequality, we have used (12).
A similar argument can be used to establish the uniform boundedness of .
Step 3. In this step, we show that one can first fix a large enough and then choose a large enough such that is arbitrarily small; more precisely, for any , there exists such that for any , there exists such that for all , .
Towards this goal, we first note that
In light of the integrability of , for any given , there exists such that
Now, it can be easily verified that for any given , we can first fix a large enough and then choose large enough such that , and furthermore,
where we have used (3) in deriving (a). It then follows that for and as above,
establishing Step .
Step 4. In this step, fixing a polynomial , we show that for any , there exists such that for any , there exists such that for all ,
To achieve this goal, it suffices to prove that, given any fixed , for any , one can first fix a large enough and then choose a large enough such that
which is equivalent to
First of all, we note that
And for the first term, using the well-known fact that for any two compatible matrices ,
we deduce that
It then follows from Steps , and that for any , one can first fix a large enough and then choose a large enough such that
A completely parallel argument can be used to establish the same statement for other terms, which in turn implies our goal in this step.
Step 5. In this step, we finish the proof of the theorem. First of all, let be a monotone decreasing sequence of positive real numbers convergent to . For any , we first arbitrarily choose a monotone increasing sequence of positive real numbers divergent to infinity, and then, applying Theorem 3.3, choose for each such that
Then, applying the Weierstrass approximation theorem to the continuous function , we choose two polynomials such that for all ,
which obviously leads to
Re-choosing a larger first and then a larger if necessary, we have, by Step ,
Now, as elaborated in Appendix A, one can show that (again re-choosing for each if necessary),
And moreover, from (15), we deduce that
This, together with the integrability of , implies that
The theorem then immediately follows from a typical subsequence argument, as desired.
4 Concluding Remarks
Some remarks about the approach employed in this work are in order.
First, echoing , we emphasize that time sampling, which is the key ingredient in our approach, ensures the inheritance of causality in converting a continuous-time Gaussian channel to its discrete-time versions, which stands in contrast to the orthogonal expansion representation in some previous approaches that destroys the temporal causality in the conversion process.
More technically, we note that our proof of Theorem 3.1 has actually established that one can appropriately “scale” and with shrinking to as tends to infinity (i.e., the sampling gets finer) such that
Such a result can be regarded as a “scalable” version of Szego’s theorem, which seems to serve as a bridge connecting discrete-time and continuous-time Szego’s theorems.
As argued above, we believe that, other than recovering a classical information-theoretic formula with an elementary proof, our approach promises further applications in more general settings, which, for instance, include possible extensions of the formula (2) to continuous-time Gaussian channels with feedback and memory , or with multi-users , or multi-inputs and multiple-outputs .
Acknowledgement. We would like to thank Professor Shunsuke Ihara for insightful discussions and for pointing out relevant references.
Appendix A Proof of (19)
To prove (19), it suffices to prove that for any ,
where for (a), we have used the easily verifiable fact that if is equal to or , then is equal to , and otherwise. Noting that
with a routine continuity argument using the definition of integral, we arrive at
where we have used the uniform boundedness and uniform continuity of for the first equality, and the last equality follows from the fact that and are a Fourier transform pair. Moreover, using the absolute summability of and the fact that (this follows from the Riemann-Lesbegue lemma), we have