I Introduction
Most of the coding strategies developed in information theory are based on random code ensembles which are constructed using independent and identically distributed (IID) sequences of random variables
[1, 2, 3, 4]. The codes associated with different terminals in the network are mutually independent. Moreover, the blocklengths associated with these codes are asymptotically large. This allows the application of the laws of large numbers and concentration of measure theorems when analyzing the performance of coding strategies; and leads to characterizations of their achievable regions in terms of information quantities that are the functionals of the underlying distribution used to construct the codes. These characterizations are often called singleletter characterizations. Although the original problem is to optimize the performance of codes with asymptotically large blocklengths, the solution is characterized by a functional (such as mutual information) of just one realization of the source or the channel under consideration. It is wellknown that unstructured random codes with asymptotically large blocklength can be used to achieve optimality in terms of achievable rates in pointtopoint communications. In fact, it can be shown that large blocklength codes are necessary to approach optimal performance. At a high level, this is due to the fact that the efficiency of fundamental tasks of communication such as covering and packing increases as the input dimension is increased
[5].In network communication, one needs to (a) remove redundancy among correlated information sources [2, 4] in a distributed manner in the source coding problems, and (b) induce redundancy among distributed terminals to facilitate [1, 3] cooperation among them. For example, in the network source coding problems such as distributed source coding and multiple description coding, the objective is to exploit the statistical correlation of the distributed information sources. Similarly, in the network channel coding problems, such as the interference channels and broadcast channels, correlation of information among different terminals is induced for better cooperation among them. At a high level, in addition to the basic objectives of efficient packing and covering at every terminal, the network coding strategies need to exploit statistical correlation among distributed information sources or induce statistical correlation among information accessed by terminals in the network.
Witsenhausen [6] and GacsKorner [7] made the observation that distributed processing of pairs of sequences of random variables leads to outputs which are less correlated than the original input sequences. In the network communications context, this implies that the outputs of encoding functions at different terminals in a network are less correlated with each other than the original input sequences. In [8, 9], we built upon these observations and showed that the correlation between the outputs of pairs of encoding functions operating over correlated sequences is inversely proportional to the effective length of the encoding functions. Based on these results, it can be concluded that while random unstructured coding strategies with asymptotically large blocklengths are efficient in performing the tasks of covering and packing, they are inefficient in facilitating coordination between different terminals. Using these results, we showed that finite effective codes are necessary to achieve optimality in various setups involving the transmission of correlated sources. Particularly, we showed that the effective length of optimality achieving codes is bounded from above in the distributed source coding problem as well as the problem of transmission of correlated sources over the multiple access channel (MAC) and the interference channel (IC) [8, 10].
So far, all of the results showing the necessity of finite effective length codes pertain to situations involving the distributed transmission of sources over channels and distributed compression of sources. However, the question of whether such codes are necessary in multiterminal channel coding has remained open. The reason is that the application of the results in [8, 9]
requires the presence of correlated inputs in different terminals of the network. In the case of distributed processing of sources, such correlation is readily available in the form of the distributed source. Whereas, in distributed transmission of independent message it is unclear how such a correlation can be created and exploited. In this work, we argue that in channel coding with feedback, correlation is induced because of the feedback link. More precisely, the feedback sequence at one terminal is correlated with the message set in the other terminal. In order to exploit this correlation efficiently, finite effective length codes are necessary. The contributions of this paper can be summarized as follows. We provide two examples of interference channels with feedback where finite effective length codes are necessary to approach optimality. For each of these examples, we provide an outer bound on the achievable region as a function of the effectivelength of the encoding functions used at the transmitters. Furthermore, we use finite effective length codes to prove the achievability of certain rate vectors which lie outside of the outer bound when the effective length is large. The combination of these two results shows that in these examples any coding strategy which uses encoding functions with asymptotically large effective lengths is suboptimal.
The rest of the paper is organized as follows: In Section II we introduce the problem formulation. Section III provides the prior results which are used in this paper. Section IV explains our main results. Finally, section V concludes the paper.
Ii Definitions and Model
Iia Notations
Random variables are denoted using capital letters such as . The random vector is represented by . Similarly, we use underline letters to denote vectors of numbers and functions. For shorthand, vectors are sometimes represented using underline letters without any superscript such as , and . Calligraphic letters such as and are used to represent sets.
IiB Model
The problem of Interference Channel with Feedback (ICFB) is studied in [11] and [12]. A threeuser interference channel with generalized feedback (ICFB) is characterized by three input alphabets , three output alphabets three feedback alphabets
, and transition probability distributions
. We assume that all the alphabets are finite and that the channel is memoryless. Let be the channel inputs, outputs and the channel feedback after uses of the channel, respectively. The memoryless property implies that:In the three user ICFB, there are three transmitters and three receivers. The th transmitter intends to transmit the message index to the th receiver. It is also assumed that the feedback is causally available at transmitter with one unit of delay. An example of such setup is depicted in Figure 1. In this figure, is trivial, and (i.e. the second transmitter does not receive any feedback).
For positive integers and are arbitrary positive integers.
Definition 1.
A feedbackblockcode for the three user ICFB consists of

Three sets of messages

Three sequences of encoding functions

Three decoding functions:
The message for transmitter is denoted by a random variable . It is assumed that the messages
are mutually independent and uniformly distributed on
The output of the th transmitter at the th use of the channel is denoted by . The ratetriple of an code is defined as . Let be the decoded message at the receiver . Then, the probability of the error is defined asFor this problem, at time , each transmitter can chose an encoding function randomly using a probability measure defined over the set of all encoding functions described in Definition 1. The following defines a randomized coding strategy.
Definition 2.
A randomized coding strategy is characterized by a probability measure on the set of all functions described in Definition 1.
Next, we define the achievable region for the three user ICFB.
Definition 3.
For , a ratetriple is said to be achievable by a feedbackblockcode with parameters , if the following conditions are satisfied.
Definition 4.
For , a ratetriple is said to be achievable by a randomized coding strategy with probability measure , if, with probability one with respect to , there exists a feedbackblockcode for which is achievable.
Definition 5.
For , a ratetriple is said to be achievable, if there exist a feedbackblockcode (randomized coding strategy) for which is achievable.
Definition 6.
A ratetriple is said to be achievable, if it is achievable for any . Given an ICFB, the set of all achievable ratetriples is called the feedbackcapacity.
Iii Background and Prior Results
In this section, we summarize the results in [8] on the correlation between the outputs of Boolean functions of pairs of sequences of random variables. These results are used in the next section to prove the necessity of finite effective length codes.
Definition 7.
Definition 8.
A BinaryBlockEncoder (BBE) is characterized by the triple , where is a mapping , is a finite set, and is an integer.
Definition 9.
For a BBE and DMS X, let . For each Boolean function , the realvalued function corresponding to is defined as follows:
(1) 
Definition 10.
For a BBE , define the decomposition , where . Then, is the component of which is only a function of . The collection , is called the set of kletter components of .
Definition 11.
For a function , with real decomposition vector , the dependency spectrum is defined as the vector
of the variances, where
. The effective length is defined as the expected value , where is the Hamming weight.Lemma 1.
Let , where the supremum is taken over all singleletter functions , and such that and have unit variance and zero mean. the following bound holds:
where 1) , 2) is the variance of , 3) is the real function corresponding to , 4) is the variance of , and 5) .
Remark 1.
The value is decreasing in . So, , is maximized when most of the variance is distributed on which have lower (i.e. operate on smaller blocks). This implies that encoding functions with smaller effectivelengths can have higher correlation between their outputs.
Iv Main Results
In this section, we introduce two examples of three user ICFBs where finite effective length codes are necessary to approach optimality.
Example 1.
Consider the setup shown in Figure 2. Here, are the outputs of the th Encoder , respectively. The channel outputs , , and are received at decoders 1,2 and 3, respectively. The channel corresponding to the transition probability is described by the following relation:
where and are independent Bernoulli random variables with and . Also, the random variables , and in Figure 2 are Bernoulli random variables with and , respectively. The variables and are mutually independent. In this setup feedback is only available at encoder and . The feedback at the first transmitter is with probability one. The feedback at the third transmitter is with probability one. In other words the two transmitters receive noiseless feedback.
The following provides an outer bound on the achievable rates after channel uses. The bound is provided as a function of the average probability of agreement between the encoder outputs and .
Theorem 1.
Any randomized coding strategy for the channel in Example 1 achieves a rate vector satisfying the following inequalities:
Proof.
The proof is given in Appendix A. ∎
Corollary 1.
Define the set as the union of all ratetriples such that
Then, the feedbackcapacity of the channel in Example 1 is contained in .
Corollary 2.
Suppose and are such that , and . Then the feedbackcapacity of Example 1 is characterized by the following
Proof.
The converse follows by Theorem 1. For the achievability, we use standard Shannon random codes at encoder and . Then, the rates , and are achievable. Because of the feedbacks , the noise is available at encoder and . Transmitters one and three send to receiver . We require to be decoded at receiver losslessly. Consider a good sourcechannel code for transmission of over a Binary Symmetric Channel with noise bias . We use this codebook both at encoder and . Since the source and the codebook are available at encoder and , then with probability one. As a result, the channel becomes a binary symmetric channel with bias . Therefore, as then is reconstructed at receiver 2 without any noise. By subtracting from the channel from to becomes a noisless channel. Thus, is achievable. ∎
Lemma 2.
Let denote the feedbackcapacity region of the ICFB in Example 1. For any , there exists a continuous function such that for sufficiently small the ratetriple , where , as .
Proof.
The proof is given in Appendix B. ∎
Theorem 2.
There exist and , such that for any coding strategy achieving the ratetriple the effective length of the encoding functions producing and are bounded from above by a constant. Furthermore, the effective length is greater than 1 (i.e. uncoded transmission is not optimal).
proof outline.
From Theorem 1 the following upperbound holds for .
Therefore, it is required that
This implies that , . However, by Lemma 1, this requires that the effective length be bounded from above. If the effective length is equal to 1, then for all , this implies that . Thus, . However,
As a result it is not possible to reconstruct at the decoder losslessly. More precisely, This contradicts with .
∎
Iva The Second Example
In this subsection, we provide another example to illustrate more the necessity of coding strategy with effective finite length for communications over ICFB.
Example 2.
Consider the IC shown in 3. The outputs of encoder 1 are denoted by , the output of encoder 2 is , and the outputs of encoder 3 are . In this setup, and represent the feedback available at encoder 1 and encoder 3, respectively. All the inputs alphabets in this channel are binary. All the output alphabets are binary; except which a ternary. In this setup , and are mutually independent Bernoulli random variables with parameter , and , respectively. Finally, it is assumed that .
We first study the case in which The following lemma provides an achievable rates for this example.
Lemma 3.
For in the setup given in Example 2, the ratetriple is achievable, where
proof outline.
The bounds on follow from the standard arguments as in pointtopoint channel coding (Fano’s inequality). Next, we show the bound on . Upon receiving , the first encoder recovers . The third encoder receives and recovers . Encoder 1 and employ a sourcechannel coding scheme to encode the sources such that decoder 2 be able to reconstruct within a Hamming distortion . This is a similar problem to the CommonBit OneHelpOne Problem introduced in [13]. Using the results from [13] (Theorem 3), we can show that decoder 2 is able to reconstruct within a Hamming distortion , if the bounds
hold for some . From standard channel coding arguments the transmitted codewords from encoder 1 and 3 are decoded at receiver 2 with small probability of error, if
where is a sufficiently small number. Finally the proof follows by setting , and as in the statement of the lemma.
∎
For the case when , there is no common information between encoder and 3. From the discontinuity argument as in [13], we can show that the minimum distortion level is discontinuous as a function of . This implies that the achievable rates using single letter coding scheme strictly decreases comparing to the case when . Hence, there exists a such that any ratetriple with is not achievable using singleletter coding strategies, where is as in Lemma 3. More precisely, the following Lemma holds.
Lemma 4.
There exist and , such that for any coding strategy achieving the ratetriple the effective length of the encoding functions producing and are bounded from above by a constant. Furthermore, the effective length is greater than 1 (i.e. uncoded transmission is not optimal).
Proof.
The proof for this lemma follows from a similar argument as in Theorem 2. ∎
V Conclusion
We provided two examples of channel coding with feedback over interference networks where finite effective length coding is necessary to achieve optimal performance. We showed that in these examples, optimality achieving coding strategies utilize the feedback available in different terminals to coordinate their outputs. We showed that coding strategies with asymptotically large effective lengths are inefficient in preserving the correlation among their outputs and are hence unable to coordinate their inputs to the channel effectively.
Appendix A Proof of Theorem 1
Proof.
The bounds and follows from standard arguments as in pointtopoint channel coding problem. Note that the feedback does not increase the rate of and since these upperbounds correspond to the pointtopoint capacity and feedback does not increase pointtopoint capacity. To bound we use Fano’s inequality. Therefore, ;
(2) 
where (a), (c) and (e) follow from the fact that are independent of and that is a function of since the second transmitter does not receive feedback. (b) follows from Fano’s inequality and (d) follows from the fact that is binary.
Define the random vector as the indicator function of the event that . Then,
For the innermost term in the above inequality we have:
(3) 
where (a) follows from the definition of , (b) follows from the fact that the binary entropy is upper bounded by one and finally, (c) follows form the fact that is independent of . Combining equations (2) and (3), we get:
∎
Appendix B Proof of Lemma 2
Proof.
We can show that is the set of all ratetriples for which such that
where the joint distribution of the variables is in some set . The proof for this statement follows by a converse and an achievability argument. For the converse, we use Fanoe’s inequality as in Theorem 1. The achievability is straightforward and follows by employing a multiletter random coding scheme. For the case when , and any achievable ratetriple there exist and such that
(4)  
(5)  
(6) 
where the joint distribution of the random variables involved is denoted by . Since the binary entropy function is continuous, then there exists such that . Next, we show continuity for the righthand side of (6). Fix , and consider the third inequality in the characterization of . Note that the only probability distribution depending on is . Since this conditional probability is continuous with then so is . Thus, for any fixed , the bound on in is continuous as a function function of . As a result, there exists a function such that the righthand side of the inequality (6) is upper bounded by for some joint distribution . As a result, the following bounds hold for .
This implies that there exists sufficiently small such that . Thus, we establish the continuity of at . ∎
References
 [1] Te Han and K. Kobayashi. A new achievable rate region for the interference channel. IEEE Transactions on Information Theory, 27(1):49–60, Jan 1981.
 [2] S. Y. Tung. Multiterminal Source Coding. PhD thesis, Cornell University, Ithaca, NY, 1978.
 [3] K. Marton. A coding theorem for the discrete memoryless broadcast channel. IEEE Transactions on Information Theory, 25(3):306–311, May 1979.
 [4] Zhen Zhang and T. Berger. New results in binary multiple descriptions. IEEE Transactions on Information Theory, 33(4):502–521, Jul 1987.
 [5] I. Csiszár and J. Korner. Information Theory: Coding Theorems for Discrete Memoryless Systems. Academic Press Inc. Ltd., 1981.
 [6] H. S. Witsenhausen. On sequences of pair of dependent random variables. SIAM Journal of Applied Mathematics, 28(1):100–113, 1975.
 [7] P. Gacs and J. Körner. Common information is far less than mutual information. Problems of Control and Information Theory, 2(2):119–162, 1972.
 [8] Farhad Shirani and S Sandeep Pradhan. On the suboptimality of singleletter coding in multiterminal communications. arXiv preprint arXiv:1702.01376, 2017.
 [9] Farhad Shirani Chaharsooghi and S Sandeep Pradhan. On the correlation between boolean functions of sequences of random variables. In Information Theory (ISIT), 2017 IEEE International Symposium on, pages 1301–1305. IEEE, 2017.
 [10] F. Shirani and S. S. Pradhan. Finite blocklength gains in distributed source coding. In 2014 IEEE International Symposium on Information Theory, pages 1702–1706, June 2014.
 [11] S. Yang and D. Tuninetti, ”Interference Channel With Generalized Feedback (a.k.a. With Source Cooperation): Part I: Achievable Region," in IEEE Transactions on Information Theory, vol. 57, no. 5, pp. 26862710, May 2011.
 [12] G. Kramer, “Feedback strategies for white Gaussian interference networks," IIEEE Trans. Inf. Theory, vol. 48, no. 6, pp. 1423–1438, Jun. 2002.
 [13] A. B. Wagner, B. G. Kelly and Y. Altug, ”Distributed RateDistortion With Common Components," in IEEE Transactions on Information Theory, vol. 57, no. 7, pp. 40354057, July 2011.
Comments
There are no comments yet.