Bounds on the Effective-length of Optimal Codes for Interference Channel with Feedback

by   Mohsen Heidari, et al.
University of Michigan

In this paper, we investigate the necessity of finite blocklength codes in distributed transmission of independent message sets over channels with feedback. Previously, it was shown that finite effective length codes are necessary in distributed transmission and compression of sources. We provide two examples of three user interference channels with feedback where codes with asymptotically large effective lengths are sub-optimal. As a result, we conclude that coded transmission using finite effective length codes is necessary to achieve optimality. We argue that the sub-optimal performance of large effective length codes is due to their inefficiency in preserving the correlation between the inputs to the distributed terminals in the communication system. This correlation is made available by the presence of feedback at the terminals and is used as a means for coordination between the terminals when using finite effective length coding strategies.



There are no comments yet.


page 1

page 2

page 3

page 4


Gaussian Channels with Feedback: A Dynamic Programming Approach

In this paper, we consider a communication system where a sender sends m...

On Optimal Finite-length Binary Codes of Four Codewords for Binary Symmetric Channels

Finite-length binary codes of four codewords are studied for memoryless ...

On exact asymptotics of the error probability in channel coding: symmetric channels

The exact order of the optimal sub-exponentially decaying factor in the ...

Achievable Information-Energy Region in the Finite Block-Length Regime with Finite Constellations

This paper characterizes an achievable information-energy region of simu...

Non-adaptive and two-stage coding over the Z-channel

In this paper, we developed new coding strategies for the Z-channel. In ...

Achievable Error Exponents of One-Way and Two-Way AWGN Channels

Achievable error exponents for the one-way with noisy feedback and two-w...

Variable-Length Codes Independent or Closed with respect to Edit Relations

We investigate inference of variable-length codes in other domains of co...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Most of the coding strategies developed in information theory are based on random code ensembles which are constructed using independent and identically distributed (IID) sequences of random variables

[1, 2, 3, 4]

. The codes associated with different terminals in the network are mutually independent. Moreover, the blocklengths associated with these codes are asymptotically large. This allows the application of the laws of large numbers and concentration of measure theorems when analyzing the performance of coding strategies; and leads to characterizations of their achievable regions in terms of information quantities that are the functionals of the underlying distribution used to construct the codes. These characterizations are often called single-letter characterizations. Although the original problem is to optimize the performance of codes with asymptotically large blocklengths, the solution is characterized by a functional (such as mutual information) of just one realization of the source or the channel under consideration. It is well-known that unstructured random codes with asymptotically large blocklength can be used to achieve optimality in terms of achievable rates in point-to-point communications. In fact, it can be shown that large blocklength codes are necessary to approach optimal performance. At a high level, this is due to the fact that the efficiency of fundamental tasks of communication such as covering and packing increases as the input dimension is increased


In network communication, one needs to (a) remove redundancy among correlated information sources [2, 4] in a distributed manner in the source coding problems, and (b) induce redundancy among distributed terminals to facilitate [1, 3] cooperation among them. For example, in the network source coding problems such as distributed source coding and multiple description coding, the objective is to exploit the statistical correlation of the distributed information sources. Similarly, in the network channel coding problems, such as the interference channels and broadcast channels, correlation of information among different terminals is induced for better cooperation among them. At a high level, in addition to the basic objectives of efficient packing and covering at every terminal, the network coding strategies need to exploit statistical correlation among distributed information sources or induce statistical correlation among information accessed by terminals in the network.

Witsenhausen [6] and Gacs-Korner [7] made the observation that distributed processing of pairs of sequences of random variables leads to outputs which are less correlated than the original input sequences. In the network communications context, this implies that the outputs of encoding functions at different terminals in a network are less correlated with each other than the original input sequences. In [8, 9], we built upon these observations and showed that the correlation between the outputs of pairs of encoding functions operating over correlated sequences is inversely proportional to the effective length of the encoding functions. Based on these results, it can be concluded that while random unstructured coding strategies with asymptotically large blocklengths are efficient in performing the tasks of covering and packing, they are inefficient in facilitating coordination between different terminals. Using these results, we showed that finite effective codes are necessary to achieve optimality in various setups involving the transmission of correlated sources. Particularly, we showed that the effective length of optimality achieving codes is bounded from above in the distributed source coding problem as well as the problem of transmission of correlated sources over the multiple access channel (MAC) and the interference channel (IC) [8, 10].

So far, all of the results showing the necessity of finite effective length codes pertain to situations involving the distributed transmission of sources over channels and distributed compression of sources. However, the question of whether such codes are necessary in multi-terminal channel coding has remained open. The reason is that the application of the results in [8, 9]

requires the presence of correlated inputs in different terminals of the network. In the case of distributed processing of sources, such correlation is readily available in the form of the distributed source. Whereas, in distributed transmission of independent message it is unclear how such a correlation can be created and exploited. In this work, we argue that in channel coding with feedback, correlation is induced because of the feedback link. More precisely, the feedback sequence at one terminal is correlated with the message set in the other terminal. In order to exploit this correlation efficiently, finite effective length codes are necessary. The contributions of this paper can be summarized as follows. We provide two examples of interference channels with feedback where finite effective length codes are necessary to approach optimality. For each of these examples, we provide an outer bound on the achievable region as a function of the effective-length of the encoding functions used at the transmitters. Furthermore, we use finite effective length codes to prove the achievability of certain rate vectors which lie outside of the outer bound when the effective length is large. The combination of these two results shows that in these examples any coding strategy which uses encoding functions with asymptotically large effective lengths is sub-optimal.

The rest of the paper is organized as follows: In Section II we introduce the problem formulation. Section III provides the prior results which are used in this paper. Section IV explains our main results. Finally, section V concludes the paper.

Ii Definitions and Model

Ii-a Notations

Random variables are denoted using capital letters such as . The random vector is represented by . Similarly, we use underline letters to denote vectors of numbers and functions. For shorthand, vectors are sometimes represented using underline letters without any superscript such as , and . Calligraphic letters such as and are used to represent sets.

Ii-B Model

The problem of Interference Channel with Feedback (IC-FB) is studied in [11] and [12]. A three-user interference channel with generalized feedback (IC-FB) is characterized by three input alphabets , three output alphabets three feedback alphabets

, and transition probability distributions

. We assume that all the alphabets are finite and that the channel is memoryless. Let be the channel inputs, outputs and the channel feedback after uses of the channel, respectively. The memoryless property implies that:

In the three user IC-FB, there are three transmitters and three receivers. The th transmitter intends to transmit the message index to the th receiver. It is also assumed that the feedback is causally available at transmitter with one unit of delay. An example of such setup is depicted in Figure 1. In this figure, is trivial, and (i.e. the second transmitter does not receive any feedback).

Fig. 1: An instance of the three-user IC with generalized feedback. Here transmitters 1 and 3 receive noisy feedback, whereas transmitter 2 does not receive feedback.

For positive integers and are arbitrary positive integers.

Definition 1.

A feedback-block-code for the three user IC-FB consists of

  • Three sets of messages

  • Three sequences of encoding functions

  • Three decoding functions:

The message for transmitter is denoted by a random variable . It is assumed that the messages

are mutually independent and uniformly distributed on

The output of the th transmitter at the th use of the channel is denoted by . The rate-triple of an code is defined as . Let be the decoded message at the receiver . Then, the probability of the error is defined as

For this problem, at time , each transmitter can chose an encoding function randomly using a probability measure defined over the set of all encoding functions described in Definition 1. The following defines a randomized coding strategy.

Definition 2.

A -randomized coding strategy is characterized by a probability measure on the set of all functions described in Definition 1.

Next, we define the achievable region for the three user IC-FB.

Definition 3.

For , a rate-triple is said to be -achievable by a feedback-block-code with parameters , if the following conditions are satisfied.

Definition 4.

For , a rate-triple is said to be -achievable by a -randomized coding strategy with probability measure , if, with probability one with respect to , there exists a feedback-block-code for which is -achievable.

Definition 5.

For , a rate-triple is said to be -achievable, if there exist a feedback-block-code (randomized coding strategy) for which is -achievable.

Definition 6.

A rate-triple is said to be achievable, if it is -achievable for any . Given an IC-FB, the set of all achievable rate-triples is called the feedback-capacity.

Iii Background and Prior Results

In this section, we summarize the results in [8] on the correlation between the outputs of Boolean functions of pairs of sequences of random variables. These results are used in the next section to prove the necessity of finite effective length codes.

Definition 7.

is called a pair of DMS’s if we have , where

, for some joint distribution


Definition 8.

A Binary-Block-Encoder (BBE) is characterized by the triple , where is a mapping , is a finite set, and is an integer.

Definition 9.

For a BBE and DMS X, let . For each Boolean function , the real-valued function corresponding to is defined as follows:

Definition 10.

For a BBE , define the decomposition , where . Then, is the component of which is only a function of . The collection , is called the set of k-letter components of .

Definition 11.

For a function , with real decomposition vector , the dependency spectrum is defined as the vector

of the variances, where

. The effective length is defined as the expected value , where is the Hamming weight.

Lemma 1.

Let , where the supremum is taken over all single-letter functions , and such that and have unit variance and zero mean. the following bound holds:

where 1) , 2) is the variance of , 3) is the real function corresponding to , 4) is the variance of , and 5) .

Remark 1.

The value is decreasing in . So, , is maximized when most of the variance is distributed on which have lower (i.e. operate on smaller blocks). This implies that encoding functions with smaller effective-lengths can have higher correlation between their outputs.

Iv Main Results

In this section, we introduce two examples of three user IC-FBs where finite effective length codes are necessary to approach optimality.

Example 1.

Consider the setup shown in Figure 2. Here, are the outputs of the th Encoder , respectively. The channel outputs , , and are received at decoders 1,2 and 3, respectively. The channel corresponding to the transition probability is described by the following relation:

where and are independent Bernoulli random variables with and . Also, the random variables , and in Figure 2 are Bernoulli random variables with and , respectively. The variables and are mutually independent. In this setup feedback is only available at encoder and . The feedback at the first transmitter is with probability one. The feedback at the third transmitter is with probability one. In other words the two transmitters receive noiseless feedback.

Fig. 2: The diagram of the IC-FB given in Example 1. In this setup, is the feedback at Transmitter 1, and is the feedback at transmitter 3.

The following provides an outer bound on the achievable rates after channel uses. The bound is provided as a function of the average probability of agreement between the encoder outputs and .

Theorem 1.

Any -randomized coding strategy for the channel in Example 1 achieves a rate vector satisfying the following inequalities:


The proof is given in Appendix A. ∎

Corollary 1.

Define the set as the union of all rate-triples such that

Then, the feedback-capacity of the channel in Example 1 is contained in .

Corollary 2.

Suppose and are such that , and . Then the feedback-capacity of Example 1 is characterized by the following


The converse follows by Theorem 1. For the achievability, we use standard Shannon random codes at encoder and . Then, the rates , and are achievable. Because of the feedbacks , the noise is available at encoder and . Transmitters one and three send to receiver . We require to be decoded at receiver losslessly. Consider a good source-channel code for transmission of over a Binary Symmetric Channel with noise bias . We use this codebook both at encoder and . Since the source and the codebook are available at encoder and , then with probability one. As a result, the channel becomes a binary symmetric channel with bias . Therefore, as then is reconstructed at receiver 2 without any noise. By subtracting from the channel from to becomes a noisless channel. Thus, is achievable. ∎

Lemma 2.

Let denote the feedback-capacity region of the IC-FB in Example 1. For any , there exists a continuous function such that for sufficiently small the rate-triple , where , as .


The proof is given in Appendix B. ∎

Theorem 2.

There exist and , such that for any coding strategy achieving the rate-triple the effective length of the encoding functions producing and are bounded from above by a constant. Furthermore, the effective length is greater than 1 (i.e. uncoded transmission is not optimal).

proof outline.

From Theorem 1 the following upper-bound holds for .

Therefore, it is required that

This implies that , . However, by Lemma 1, this requires that the effective length be bounded from above. If the effective length is equal to 1, then for all , this implies that . Thus, . However,

As a result it is not possible to reconstruct at the decoder losslessly. More precisely, This contradicts with .

Iv-a The Second Example

In this subsection, we provide another example to illustrate more the necessity of coding strategy with effective finite length for communications over IC-FB.

Example 2.

Consider the IC shown in 3. The outputs of encoder 1 are denoted by , the output of encoder 2 is , and the outputs of encoder 3 are . In this setup, and represent the feedback available at encoder 1 and encoder 3, respectively. All the inputs alphabets in this channel are binary. All the output alphabets are binary; except which a ternary. In this setup , and are mutually independent Bernoulli random variables with parameter , and , respectively. Finally, it is assumed that .

Fig. 3: The diagram of the IC-FB in Example 2. In this setup, , the feedback at Transmitter 1, is a noisy version of .

We first study the case in which The following lemma provides an achievable rates for this example.

Lemma 3.

For in the setup given in Example 2, the rate-triple is achievable, where

proof outline.

The bounds on follow from the standard arguments as in point-to-point channel coding (Fano’s inequality). Next, we show the bound on . Upon receiving , the first encoder recovers . The third encoder receives and recovers . Encoder 1 and employ a source-channel coding scheme to encode the sources such that decoder 2 be able to reconstruct within a Hamming distortion . This is a similar problem to the Common-Bit One-Help-One Problem introduced in [13]. Using the results from [13] (Theorem 3), we can show that decoder 2 is able to reconstruct within a Hamming distortion , if the bounds

hold for some . From standard channel coding arguments the transmitted codewords from encoder 1 and 3 are decoded at receiver 2 with small probability of error, if

where is a sufficiently small number. Finally the proof follows by setting , and as in the statement of the lemma.

For the case when , there is no common information between encoder and 3. From the discontinuity argument as in [13], we can show that the minimum distortion level is discontinuous as a function of . This implies that the achievable rates using single letter coding scheme strictly decreases comparing to the case when . Hence, there exists a such that any rate-triple with is not achievable using single-letter coding strategies, where is as in Lemma 3. More precisely, the following Lemma holds.

Lemma 4.

There exist and , such that for any coding strategy achieving the rate-triple the effective length of the encoding functions producing and are bounded from above by a constant. Furthermore, the effective length is greater than 1 (i.e. uncoded transmission is not optimal).


The proof for this lemma follows from a similar argument as in Theorem 2. ∎

V Conclusion

We provided two examples of channel coding with feedback over interference networks where finite effective length coding is necessary to achieve optimal performance. We showed that in these examples, optimality achieving coding strategies utilize the feedback available in different terminals to coordinate their outputs. We showed that coding strategies with asymptotically large effective lengths are inefficient in preserving the correlation among their outputs and are hence unable to coordinate their inputs to the channel effectively.

Appendix A Proof of Theorem 1


The bounds and follows from standard arguments as in point-to-point channel coding problem. Note that the feedback does not increase the rate of and since these upper-bounds correspond to the point-to-point capacity and feedback does not increase point-to-point capacity. To bound we use Fano’s inequality. Therefore, ;


where (a), (c) and (e) follow from the fact that are independent of and that is a function of since the second transmitter does not receive feedback. (b) follows from Fano’s inequality and (d) follows from the fact that is binary.

Define the random vector as the indicator function of the event that . Then,

For the innermost term in the above inequality we have:


where (a) follows from the definition of , (b) follows from the fact that the binary entropy is upper bounded by one and finally, (c) follows form the fact that is independent of . Combining equations (2) and (3), we get:

Appendix B Proof of Lemma 2


We can show that is the set of all rate-triples for which such that

where the joint distribution of the variables is in some set . The proof for this statement follows by a converse and an achievability argument. For the converse, we use Fanoe’s inequality as in Theorem 1. The achievability is straight-forward and follows by employing a multi-letter random coding scheme. For the case when , and any achievable rate-triple there exist and such that


where the joint distribution of the random variables involved is denoted by . Since the binary entropy function is continuous, then there exists such that . Next, we show continuity for the right-hand side of (6). Fix , and consider the third inequality in the characterization of . Note that the only probability distribution depending on is . Since this conditional probability is continuous with then so is . Thus, for any fixed , the bound on in is continuous as a function function of . As a result, there exists a function such that the right-hand side of the inequality (6) is upper bounded by for some joint distribution . As a result, the following bounds hold for .

This implies that there exists sufficiently small such that . Thus, we establish the continuity of at . ∎


  • [1] Te Han and K. Kobayashi. A new achievable rate region for the interference channel. IEEE Transactions on Information Theory, 27(1):49–60, Jan 1981.
  • [2] S. Y. Tung. Multiterminal Source Coding. PhD thesis, Cornell University, Ithaca, NY, 1978.
  • [3] K. Marton. A coding theorem for the discrete memoryless broadcast channel. IEEE Transactions on Information Theory, 25(3):306–311, May 1979.
  • [4] Zhen Zhang and T. Berger. New results in binary multiple descriptions. IEEE Transactions on Information Theory, 33(4):502–521, Jul 1987.
  • [5] I. Csiszár and J. Korner. Information Theory: Coding Theorems for Discrete Memoryless Systems. Academic Press Inc. Ltd., 1981.
  • [6] H. S. Witsenhausen. On sequences of pair of dependent random variables. SIAM Journal of Applied Mathematics, 28(1):100–113, 1975.
  • [7] P. Gacs and J. Körner. Common information is far less than mutual information. Problems of Control and Information Theory, 2(2):119–162, 1972.
  • [8] Farhad Shirani and S Sandeep Pradhan. On the sub-optimality of single-letter coding in multi-terminal communications. arXiv preprint arXiv:1702.01376, 2017.
  • [9] Farhad Shirani Chaharsooghi and S Sandeep Pradhan. On the correlation between boolean functions of sequences of random variables. In Information Theory (ISIT), 2017 IEEE International Symposium on, pages 1301–1305. IEEE, 2017.
  • [10] F. Shirani and S. S. Pradhan. Finite block-length gains in distributed source coding. In 2014 IEEE International Symposium on Information Theory, pages 1702–1706, June 2014.
  • [11] S. Yang and D. Tuninetti, ”Interference Channel With Generalized Feedback (a.k.a. With Source Cooperation): Part I: Achievable Region," in IEEE Transactions on Information Theory, vol. 57, no. 5, pp. 2686-2710, May 2011.
  • [12] G. Kramer, “Feedback strategies for white Gaussian interference networks," IIEEE Trans. Inf. Theory, vol. 48, no. 6, pp. 1423–1438, Jun. 2002.
  • [13] A. B. Wagner, B. G. Kelly and Y. Altug, ”Distributed Rate-Distortion With Common Components," in IEEE Transactions on Information Theory, vol. 57, no. 7, pp. 4035-4057, July 2011.