Error-Free Communication Over State-Dependent Channels with Variable-Length Feedback

I Introduction

This work considers the zero-error capacity of state-dependent channels. In contrast to the standard notion of capacity, which allows an asymptotically vanishing probability of error at the decoder, the output of a zero-error decoder must always be correct. We focus on variable-length coding schemes with access to noiseless feedback at the encoder, and derive the zero-error capacity under different models of state information availability.

At first glance, the requirement of zero-error decoding is quite stringent. Indeed, the zero-error capacity of many channels, including the binary symmetric channel (BSC), is $0$ , and this holds even for the relatively benign binary erasure channel (BEC). However, it became evident after Burnashev’s work on the error exponent of discrete memoryless channels (DMCs) [3] that, with variable-length encoding and noiseless feedback, error-free communication is not only possible over a large class of DMCs, but the zero-error capacity of such channels is equal to their (vanishing-error) Shannon capacity. The fact that the channel capacity can in many cases be achieved with error probability being fixed to zero is quite interesting, and this is what motivated us to study the corresponding problem in the more general context of channels with states. Furthermore, analyzing fundamental limits of communication under variable-length coding is important in systems with feedback, and even more so in systems where side information about the channel is available as well. In fact, most communication schemes in such systems, e.g., the ARQ mechanism, are adaptive in nature and the length of transmission depends on the side information and the information obtained through the feedback link.

Communicating with zero error has been considered previously in many settings, starting with Shannon’s work [12] on the zero-error capacity of DMCs with and without feedback, and under fixed-length encoding. We refer the reader to [8] for a review of this area. The work most closely related to ours is that of Bracher and Lapidoth [2] where the zero-error feedback capacity of state-dependent channels was determined, under the assumptions that fixed-length encoding is being used and that state information is available only at the encoder. Our work extends these results to the variable-length case, while also analyzing other models of state information availability. We also mention here the work of Zhao and Permuter [16] where the authors characterized the zero-error feedback capacity under fixed-length encoding of channels with state information at both the encoder and the decoder, but in which the state process is not necessarily memoryless and is even allowed to depend on the channel inputs.

The main contributions of this work are threefold. First, we show how to relate the zero-error capacity with variable-length encoding and feedback (“zero-error VLF capacity”) to the standard vanishing-error capacity with fixed-length encoding and no feedback (Theorem 1 and Theorem 3). Second, we give necessary and sufficient conditions for the zero-error VLF capacity to be positive under different models of state information availability at the encoder and the decoder, including none, strictly causal (only past states are available), causal (past and current states available), and non-causal (all states including future states available). These results are summarized in Theorem 2. These theorems together completely characterize the zero-error VLF capacity of state-dependent channels. And third, we obtain analogous results for the zero-error feedback capacity under bounded-length coding (Theorem 5 and Theorem 6). We show that the conditions for positivity are in this case the same as for fixed-length coding schemes, although the final capacity expressions may be different.

In addition, as part of our proofs, we derive an upper bound on the $ϵ$ -error capacity for state-dependent channels (Lemma 10), which may be of independent interest.

I-a The Channel Model, Definitions, and Notation

Let $X, Y, S$

denote the set of channel input letters, the set of channel output letters, and the set of channel states, respectively, all of which are assumed finite. A state-dependent discrete memoryless channel (SD-DMC) is described by conditional probability distributions

W (y | x, s)

x \in X, y \in Y, s \in S

, where the states are drawn i.i.d. across all channel uses according to a distribution

Q (\cdot)

S

. To avoid discussing trivial cases, we assume throughout the paper that

| X | \geq 2

| Y | \geq 2

| S | \geq 1

; that all states in

S

have positive probability:

\forall s \in S Q (s) > 0;

(1)

and that every channel output is reachable from at least one input in at least one state:

\forall y \in Y \exists x \in X, s \in S W (y | x, s) > 0.

(2)

We use the symbol $M$ to denote the set of messages to be transmitted in a particular communication setting. The symbols $M, X, Y, S$

denote random variables taking values in

M, X, Y, S

, respectively, and the lower case versions

m, x, y, s

denote their realizations, i.e., elements of

M, X, Y, S

. The random variable

M

representing the transmitted message is always assumed to be uniform over

M

X^{\infty}

is a shorthand for a random infinite sequence

(X_{1}, X_{2}, \dots)

X^{n}

for a random finite sequence

(X_{1}, \dots, X_{n})

, and

X_{k}^{n}

for a subsequence

(X_{k}, \dots, X_{n})

(hence

X_{1}^{n} \equiv X^{n}

We say that the encoder (resp. decoder) has causal state information if, before the $n$ ’th channel use, it can see all the past channel states as well as the current— $n$ ’th—state, i.e., it is given the state sequence $S^{n}$ and can use it in the $n$ ’th time slot for the encoding (resp. decoding) operation. State information is said to be strictly causal if only past states ( $S^{n - 1}$ ) are available at time instant $n$ , and it is said to be non-causal if all the states ( $S^{\infty}$ ) are available at any time instant. We consider the following cases of state information availability:

SI\coloneqq{(-,-),(% sc,-),(c,-),(nc,-),(sc,c),(c,c% ),(nc,c),(nc,nc)},

(3)

where the first (resp. second) coordinate of $si \in SI$ denotes state information available at the encoder (resp. decoder) and -/sc/c/nc stand for none/strictly-causal/causal/non-causal. The cases that have been omitted from SI are discussed in Remark 1 to follow.

Definition 1.

Consider an SD-DMC with causal state information at both the encoder and the decoder. An $(ℓ, | M |, ϵ)$ variable-length feedback (VLF) code for the message set $M$ , where $ℓ$ is a positive real and $0 \leq ϵ \leq 1$ , is defined by:

A sequence of encoders $f_{n} : M \times Y^{n - 1} \times S^{n} \to X$ , $n \geq 1$ , defining channel inputs $X_{n} = f_{n} (M, Y^{n - 1}, S^{n})$ ;
A sequence of decoders $g_{n} : Y^{n} \times S^{n} \to M$ , $n \geq 1$
, defining decoder’s estimates of the transmitted message,
$g_{n} (Y^{n}, S^{n})$ ;
A positive integer-valued random variable $τ$ (a stopping time of the receiver filtration ${σ (Y^{n}, S^{n})}_{n = 0}^{\infty}$ ) representing the code length and satisfying $E [τ] \leq ℓ$ .

The decoder’s final decision is computed at time $τ$ , $ˆ M = g_{τ} (Y^{τ}, S^{τ})$ , and it must satisfy $P [ˆ M \neq M] \leq ϵ$ .

When $ϵ = 0$ , such a code is called a zero-error VLF code.

If there exists a constant $b < \infty$ such that $τ \leq b$ , such a code is called a bounded-length feedback code, and if $τ = b = ℓ$ , it is called a fixed-length feedback code.

Definitions for the other cases of state information availability $si \in SI$ are the same except $S^{n}$ in 1)–3) is replaced by $S^{0}$ , $S^{n - 1}$ , or $S^{\infty}$ accordingly. $▲$

The rate of an $(ℓ, | M |, ϵ)$ code is defined as $\frac{1}{ℓ} {log}_{2} | M |$ . The vanishing-error capacity of a given channel is defined in the usual way as the supremum of the code rates that are asymptotically achievable (as $ℓ \to \infty$ ) with arbitrarily small error probability. The zero-error capacity of a given channel is the supremum of the rates of all zero-error codes for that channel [12]. Capacity is always denoted by $C$ , with subscripts and superscripts indicating the channel and the coding schemes with respect to which it is defined as follows:

The first subscript is either “ $0$ ” or “ $↓$ ” and serves to distinguish between the zero-error and the vanishing-error case;
The second subscript is either “f” or “-” depending on whether or not the feedback link is present;
The third subscript is vl, bl, or fl, indicating that the capacity in question is defined with respect to variable-length, bounded-length, or fixed-length codes;
Superscripts from the set SI (see (3)) are used to denote state information availability at the encoder and the decoder.

For example, $Cnc,-↓,f,\textscvl$ is the vanishing-error capacity under variable-length feedback codes, where the encoder is given state information in a non-causal manner and the decoder is given no state information; $Cc,c0,-,\textscbl$ is the zero-error capacity under bounded-length coding without feedback, and with state information revealed both to the encoder and to the decoder in a causal manner; etc.

Remark 1.

To conclude this section we explain briefly why, of the sixteen possible cases in ${-, sc, c, nc}^{2}$ , only the cases in SI in (3) are being considered.

First, leaving out the four cases with strictly causal state information at the decoder, ${-, sc, c, nc} \times {sc}$ , is not a loss in generality. This is because strictly causal state information at the decoder can be “made causal” by simply delaying the decoding process by one time slot. Hence, from the viewpoint of capacity issues, $(*, sc)$ is equivalent to $(*, c)$ for any $* \in {-, sc, c, nc}$ .

Second, of the four possible cases where the decoder has non-causal state information ( ${-, sc, c, nc} \times {nc}$ ) we only consider one— $(nc, nc)$ . This is also not a loss in generality because knowing future states can be helpful to the decoder only if the encoder also knows future states and is using them in the encoding operation. Otherwise, these states are independent of the channel inputs and no information can be extracted from them. Hence, for our purposes, $(*, nc)$ is equivalent to $(*, c)$ for any $* \in {-, sc, c}$ .

Finally, note that $(-, c)$ has not been included in SI either. This case is quite subtle and we choose to discuss it separately in Section V. The main issue here is that it is not clear how to define the code length, i.e., the stopping time $τ$ (see Definition 1). Namely, the decoder making a decision at time instant $τ^{'}$ does not necessarily mean that the transmission is over from the encoder’s perspective. This is because the decoder’s decision is based on the channel outputs and the channel states that it sees, and hence the encoder, not knowing the states, may actually never realize that the decoding is completed and may continue transmitting. As we shall see, this is especially important in the zero-error setting. Despite this difficulty, we shall give in Section V a sufficient condition for the zero-error capacity $C-,c0,f,\textscvl$ to be positive, and a necessary and sufficient condition for $C-,c0,f,\textscbl$ to be positive. $▲$

Ii Vanishing-Error Capacity

As already mentioned in Section I, one of the results of this paper is the statement that the zero-error VLF capacity of an SD-DMC, whenever positive, equals the vanishing-error capacity of the same channel. For this reason, we first study the vanishing-error capacity and show that this fundamental limit remains unchanged even if the transmitter and the receiver are allowed to use variable-length feedback-dependent coding schemes. For SD-DMCs with state information only at the transmitter, the fact that feedback does not increase the capacity under fixed-length coding was shown in [10].

Theorem 1.

For every $si \in SI$ , $Csi↓,f,\textscvl=Csi↓,f,\textscbl=Csi↓,f,\textscfl=Csi↓,-,\textscfl$ .

Proof:

Deferred to the Appendix. ∎

Consequently, we shall denote the vanishing-error capacity (with or without feedback) simply by $C_{↓}^{si}$ in the rest of the paper.

By Theorem 1 and the known expressions for the capacity $Csi↓,-,\textscfl$ [5, Ch. 7] we get:

$C-,-↓,f,\textscvl=Csc,-↓,f,\textscvl$	$= max P_{X} I (X; Y)$	(4)
$Cc,-↓,f,\textscvl$	$= max P_{U}, f : U \times S \to X I (U; Y)$	(5)
$Cnc,-↓,f,\textscvl$		(6)
$Csc,c↓,f,\textscvl$	$= max P_{X} I (X; Y \| S)$	(7)
$Cc,c↓,f,\textscvl=Cnc,c↓,f,\textscvl=Cnc,nc↓,f,\textscvl$	$= max P_{X \| S} I (X; Y \| S),$	(8)

where $U$ denotes an auxiliary random variable with alphabet $U$ of cardinality $| U | \leq | X | | S |$ .

Iii Zero-Error Capacity: Variable-Length Codes

For a DMC $W (\cdot | \cdot)$ , a necessary and sufficient condition for $C0,f,\textscvl>0$ is

\exists x \in X, y \in Y W (y | x) = 0.

(9)

This can be concluded from Burnashev’s characterization of the error exponent of DMCs under VLF coding schemes [3]—the corresponding error exponent is infinite if and only if (9) holds. When $W (y | x) = 0$ , $y$ is said to be a disprover¹¹1This terminology is from [9] and reflects the fact that such an output $y$ “disproves” the possibility that $x$ was the channel input. for $x$ . Whenever such a disprover exists, one bit can be transmitted error free in a finite expected number of channel uses as follows. In the first two channel uses the transmitter sends $x, x^{'}$ for $0$ and $x^{'}, x$ for 1, where $x^{'}$ is any other input letter with $W (y | x^{'}) > 0$ (such an input letter necessarily exists by our assumption (2)). Due to the fact that $W (y | x) = 0$ , if the letters obtained at the output are $\neg y, y$ , where $\neg y$ denotes an arbitrary letter from $Y ∖ {y}$ , the receiver concludes that $0$ must have been transmitted; similarly, if the letters obtained at the output are $y, \neg y$ , then $1$ must have been transmitted; finally, if the letters obtained at the output are $\neg y, \neg y$ , the procedure is repeated (the transmitter sees the output letters through feedback and knows whether or not it should repeat the transmission). The expected number of retransmissions needed to complete the protocol is finite because the event $x^{'} \to y$ has positive probability, i.e., $W (y | x^{'}) > 0$ . Therefore, in a finite expected number of channel uses the receiver will recover the bit correctly.

For a study of zero-error VLF communication over a special kind of DMC—the so-called Z-channel—see also [14].

In the following statement we identify generalizations of the notion of disprover for state-dependent channels. The idea is that, in each case, there is a disprover output $y$ which the sender can choose to avoid (using the disproved input $x$ ), so that seeing $y$ gives error-free information that $y$ was intended as a possible output.

Theorem 2.

Necessary and sufficient conditions for positivity of the zero-error VLF capacity of SD-DMCs are as follows:

$C-,-0,f,\textscvl>0$ if and only if there is a pair $(x, y) \in X \times Y$ such that $y$ disproves $x$ in all states:

$\exists x \in X, y \in Y \forall s \in S W (y | x, s) = 0.$ (10)
$Csc,-0,f,\textscvl>0$ if and only if (10) holds.
$Cc,-0,f,\textscvl>0$ if and only if there is an output letter $y$ which is a disprover in each state:

$\exists y \in Y \forall s \in S \exists x \in X W (y | x, s) = 0.$ (11)
$Cnc,-0,f,\textscvl>0$ if and only if (11) holds.
$Csc,c0,f,\textscvl>0$ if and only if there is at least one state with a disprover:

$\exists x, x^{'} \in X, y \in Y, s \in S W (y | x, s) = 0 \land W (y | x^{'}, s) > 0.$ (12)
$Cc,c0,f,\textscvl>0$ if and only if (12) holds.
$Cnc,c0,f,\textscvl>0$ if and only if (12) holds.
$Cnc,nc0,f,\textscvl>0$ if and only if (12) holds.

Proof:

(a) In the case

si = (-, -)

when neither side has any state information, the channel is equivalent to the DMC

˜W(y|x)\coloneqq∑s∈SQ(s)W(y|x,s)

. The condition (10) is the condition for positivity of the zero-error VLF capacity of this DMC; see (9) (recall that

Q (s) > 0

for all

s \in S

by our assumption (1)). (b) Let

si = (sc, -)

. Since

Csc,-0,f,\textscvl≥C-,-0,f,\textscvl

, we only need to show the “only if” part. So suppose that (10) is not satisfied, meaning that for every input-output pair

(x, y) \in X \times Y

there exists a state

s_{x, y}

where

W (y | x, s_{x, y}) > 0

. Then every output sequence

y_{1} \dots y_{n}

can be produced by every input sequence

x_{1} \dots x_{n}

with positive probability (if the state sequence happens to be

s_{x_{1}, y_{1}} \dots s_{x_{n}, y_{n}}

). This means that the decoder cannot decide with certainty at any point in time what the transmitted message was. Therefore, zero-error communication in a finite average number of channel uses is impossible. The main point in this argument, stated informally, is that the encoder cannot avoid the “bad” states because it only sees the past ones, i.e., no matter which letter it chooses to send in the current time slot, the channel state can always be such that the decoder remains confused. (c) Now consider the case

si = (c, -)

. We first prove sufficiency of (11) using a procedure analogous to the one for DMCs outlined in the paragraph following (9). Let

y

be an output letter claimed to exist in (11). For every

s \in S

choose an input letter

x_{s}

such that

W (y | x_{s}, s) = 0

. Also, for every

s \in S

define

x_{s}^{'}

to be some input letter with

W (y | x_{s}^{'}, s) > 0

, if it exists; otherwise, pick

x_{s}^{'}

to be any letter different from

x_{s}

(note that a letter

x^{'}

with

W (y | x^{'}, s) > 0

exists for at least one state

s

; see (2)). If the states realized in the first two time slots are

s_{1}, s_{2}

, the transmitter sends

x_{s_{1}}, x_{s_{2}}^{'}

for

0

and

x_{s_{1}}^{'}, x_{s_{2}}

for

1

(the transmitter knows the channel state before it sends a letter). Since

W (y | x_{s}, s) = 0

, if the letters obtained at the output are

\neg y, y

, the receiver concludes that

0

must have been transmitted; if the letters obtained at the output are

y, \neg y

, then

1

must have been transmitted; if the letters obtained at the output are

\neg y, \neg y

, the procedure is repeated in the next two slots, and so on. In this way the receiver will recover the transmitted bit in a finite expected number of channel uses

k

, implying that

Cc,-0,f,\textscvl≥1k>0

. To prove the converse, suppose that (11) does not hold, i.e., for every output letter

y

there exists a state

s_{y}

such that

W (y | x, s_{y}) > 0

for all input letters

x

. Then for any output sequence

y_{1} \dots y_{n}

the state sequence

s_{y_{1}} \dots s_{y_{n}}

produces

y_{1} \dots y_{n}

with positive probability on any input

x_{1} \dots x_{n}

. This means that the decoder cannot be certain, at any time instant

n

, what was the transmitted message, and hence zero-error communication in a finite average number of channel uses is impossible. (d) The above proof for the

(c, -)

case goes through for the

(nc, -)

case as well. (e) Let

si = (sc, c)

. Suppose that (12) holds and let

x, x^{'}, y, s

be input letters, an output letter, and a state satisfying

W (y | x, s) = 0

W (y | x^{'}, s) > 0

. Let also

* \in X

be an arbitrary input letter. In the first two channel uses the transmitter sends

x, x^{'}, *

for

0

and

x^{'}, x, *

for 1. (The third letter is irrelevant and will be ignored by the receiver. It is sent only because state information is delivered to the transmitter with a one-slot delay, so in order for it to learn the states in the first two slots, a dummy letter is transmitted in the third slot.) Now, if the letters obtained at the output are

\neg y, y

and the channel states in these two slots are

s, s

, the receiver concludes that

0

must have been transmitted (the receiver can see the states); if the letters obtained at the output are

y, \neg y

and the channel states in these two slots are

s, s

, then

1

must have been transmitted; if none of the above two situations occurred, the procedure is repeated. In a finite expected number of channel uses one of the above two situations will happen and the receiver will recover the transmitted bit, implying that

Csc,c0,f,\textscvl>0

. For the converse, notice that if (12) is not satisfied, then in every state an arbitrary output

y

is reachable from either all inputs, or from none of them. Clearly, any such state is useless for zero-error communication. (f)–(h) The proof of (e) goes through in these cases as well (the proof of achievability can in fact be slightly simplified here—there is no need for sending the dummy letter). We should note that, for these models of state information availability, the necessity and sufficiency of (12) also follows from the results in [4], where the error exponent of finite-state ergodic Markov channels with causal state information at both sides has been characterized. Namely, the error exponent of an SD-DMC with causal state information at both sides is infinite at all rates below capacity if and only if (12) holds.

∎

Remark 2 (Shannon strategy).

The usual way of proving achievability results for channels with causal state information at the encoder, $si = (c, -)$ , is via the so-called Shannon strategy [13, 5]. In this approach, one considers the set $U\coloneqqXS$ of all functions from $S$ to $X$ and a related DMC with input alphabet $U$ and output alphabet $X$ defined by $W′(y|u)\coloneqq∑s∈SQ(s)W(y|u(s),s)$ . A code is then defined over the alphabet $U$ and the communication over the original SD-DMC $W (\cdot | \cdot, \cdot)$ proceeds as follows: if the channel state in the current— $n$ ’th—time slot is $s$ , and the $n$ ’th symbol of the codeword is $u$ , then the transmitter sends $x = u (s)$ . It was shown by Shannon that this strategy achieves the capacity of SD-DMCs with causal state information at the encoder [13]. The strategy also achieves the zero-error capacity under fixed-length coding [2]. Thus, in these settings, an SD-DMC $W$ (with $si = (c, -)$ ) is essentially equivalent to a stateless DMC $W^{'}$ .

We wish to point out here that optimality of the Shannon strategy continues to hold in the zero-error VLF setting as well. To see this, recall that a necessary and sufficient condition for positivity of the zero-error VLF capacity of the DMC $W^{'}$ is the existence of an input $u \in U$ and an output $y \in Y$ such that $W^{'} (y | u) = 0$ (see (9)). This condition is equivalent to (11) because:

	$\exists u \in U W^{'} (y \| u) = 0$	$\Leftrightarrow \exists u \in U \forall s \in S W (y \| u (s), s) = 0$		(13)
		$\Leftrightarrow \forall s \in S \exists x \in X W (y \| x, s) = 0,$		(14)

where (13) follows from the definition of $W^{'}$ , and (14) holds because $u$ is a function from $S$ to $X$ . Notice that the condition in (14) is precisely (11). $▲$

Remark 3.

Note that the condition (11) for positivity of the zero-error VLF capacity is the same for causal and non-causal state information at the transmitter. This is not the case in the fixed-length and bounded-length settings; see [2] and Theorem 5 ahead.

Likewise, the condition (12) states that the zero-error VLF capacities for the $(sc, c)$ and $(c, c)$ cases are either both positive or both zero. This is not the case when fixed-length or bounded-length codes are being used; see Theorem 5. $▲$

We next characterize the value of the zero-error VLF capacity of SD-DMCs. The statement is that, whenever this quantity is positive, it equals the vanishing-error capacity of the same channel. The analogous result for DMCs is known and can be inferred from Burnashev’s characterization of the error exponent of such channels under VLF coding (as mentioned above, the error exponent is infinite at all rates below $C_{↓}$ if and only if (9) holds). However, there is a simpler and more direct way of proving this statement which can be extended to channels for which error exponents are not known. One such proof was given by Han and Sato for DMCs and bounded-length codes [7], but virtually no changes are required to extend it to the setting we are interested in.

Theorem 3.

Let $si \in SI$ . If $Csi0,f,\textscvl>0$ , then $Csi0,f,\textscvl=C%si↓$ .

Proof:

Obviously, $Csi0,f,\textscvl≤Csi↓,f,\textscvl≡Csi↓$ . To demonstrate that the reverse inequality also holds whenever $Csi0,f,\textscvl>0$ , we shall use a slight modification of the Han–Sato coding scheme [7]. Assume that $Csi0,f,\textscvl>0$ . Let $M$ be the message set. Let $C$ be a fixed-length feedback code for this message set having the following properties: length $n$ , rate $\frac{1}{n} {log}_{2} | M |$ with $C_{↓}^{si} - δ \leq \frac{1}{n} {log}_{2} | M | \leq C_{↓}^{si}$ , and error probability $\leq ϵ$ (we know that such a code exists for any $ϵ, δ > 0$ and for $n$ large enough). Let $C_{0}$ be a zero-error VLF code for the same message set having the following properties: rate $R_{0}$ , for some $R_{0} > 0$ , and average length $\frac{1}{R_{0}} {log}_{2} | M |$ with $\frac{1}{R_{0}} (C_{↓}^{si} - δ) n \leq \frac{1}{R_{0}} {log}_{2} | M | \leq \frac{1}{R_{0}} C_{↓}^{si} n$ . Based on the codes $C$ and $C_{0}$ we shall devise another variable-length zero-error coding scheme of rate arbitrarily close to $C_{↓}^{si}$ , which will prove the desired claim. The communication protocol is as follows. To send a message $m \in M$ , the transmitter first sends the corresponding codeword from $C$ . Depending on whether or not the receiver has decoded the received sequence correctly, something that the transmitter knows because it can simulate the decoding process after receiving feedback, the transmitter then sends one bit of information through the channel. This bit has the meaning of an $\textscack/\textscnack$ signal that informs the receiver about the correctness of decoding, and can be transmitted error free in a finite expected number of channel uses because the zero-error capacity is positive by assumption. Now, if the sent signal is ack, meaning that the decoding was correct and that both the transmitter and the receiver are aware of that, the protocol stops. If on the other hand the signal was nack, meaning that the decoding was incorrect, the transmitter sends the same message again, but this time it encodes the message using a zero-error code $C_{0}$ , rather than the code $C$ . This ensures that the receiver will decode the received sequence correctly with probability $1$ and the coding scheme just described is therefore zero-error. Moreover, the overall rate of the scheme is approximately equal to the rate of the code $C$ used in the first phase of the protocol, because the second phase of the protocol is active only with probability $\leq ϵ$ (the probability that the transmission in the first phase fails), and this can be made arbitrarily small. ∎

We conclude this section with a theorem that is meant to demonstrate the power of variable-length coding compared to fixed-length coding in channels with feedback—with fixed-length codes, information obtained by the transmitter through the feedback link is not fully utilized.

Theorem 4.

There exists an SD-DMC for which $Cnc,nc0,f,\textscfl=0$ and yet $C-,-0,f,\textscvl>0$ .

Proof:

Consider the following binary-input-binary-output channel with two states: in state $s_{0}$ , we have the so-called Z-channel with $W (0 | 0, s_{0}) = 1$ and $W (1 | 1, s_{0}) = 1 - p$ , $0 < p < 1$ , and in state $s_{1}$ the channel is noiseless, $W (0 | 0, s_{1}) = W (1 | 1, s_{1}) = 1$ .

Zero-error communication with fixed-length feedback codes through this channel is not possible, even if both the transmitter and the receiver have non-causal state information. This is because the state sequence may happen to be $s_{0} \dots s_{0}$ in which case every two input sequences of length $n$ are confusable, meaning that they can produce the same output sequence with positive probability. Hence, $Cnc,nc0,f,\textscfl=0$ .

However, zero-error communication with variable-length feedback codes is possible even if neither the transmitter nor the receiver have any state information, as one can verify from (10) ( $y = 1$ is a disprover for $x = 0$ in both states). In fact, not only is it possible, but the zero-error VLF capacity is by Theorem 3 equal to the vanishing-error capacity of the corresponding channel, $C-,-0,f,\textscvl=C-,-↓$ . ∎

Iv Zero-Error Capacity: Bounded-Length Codes

In the previous section we have demonstrated how variable-length encoding can significantly increase the zero-error feedback capacity of an SD-DMC. We now investigate the same problem in the situation where one wishes to impose a fixed and deterministic upper bound on the codeword lengths, or equivalently on the stopping time of transmission. Variable-length codes in general have no such bound—even though their average length is finite, each message is mapped to possibly infinitely many codewords of different lengths, which means that the decoding delay can in general be arbitrarily large. It is therefore natural, especially from the practical point of view, to consider the case where the duration of transmission is upper bounded and to investigate the corresponding fundamental limits.

Zero-error feedback capacity of DMCs under bounded-length coding was first studied by Han and Sato [7]. In particular, it was shown in [7] that the condition for positivity of $C0,f,\textscbl$ is the same as in the fixed-length case (with or without feedback) [12], namely:

\exists x, x^{'} \in X \forall y \in Y W (y | x) W (y | x^{'}) = 0.

(15)

In words, the zero-error feedback capacity of a DMC under bounded-length (or fixed-length) coding is positive if and only if there exist two non-confusable input letters.

In the following theorem we state the corresponding conditions for the positivity of the zero-error feedback capacity of SD-DMCs. As in the case of DMCs, the conditions for bounded-length coding are the same as those for fixed-length coding; however, the values of the zero-error capacities in the two settings are in general different (see Theorem 6 below).

Theorem 5.

For every $si \in SI$ , $Csi0,f,\textscbl>0$ if and only if $Csi0,f,\textscfl>0$ . In particular:

$C-,-0,f,\textscbl>0$ if and only if

$\exists x, x^{'} \in X \forall y \in Y (\forall s \in S W (y | x, s) = 0) \lor (\forall s \in S W (y | x^{'}, s) = 0) .$ (16)
$Csc,-0,f,\textscbl>0$ if and only if (16) holds.
$Cc,-0,f,\textscbl>0$ if and only if there exists a partition $Y_{0}, Y_{1}$ of $Y$ such that

$\forall s \in S \exists x, x^{'} \in X W (Y_{0} | x, s) = W (Y_{1} | x^{'}, s) = 1.$ (17)
$Cnc,-0,f,\textscbl>0$ if and only if

$\forall s, s^{'} \in S \exists x, x^{'} \in X \forall y \in Y W (y | x, s) W (y | x^{'}, s^{'}) = 0.$ (18)
$Csc,c0,f,\textscbl>0$ if and only if there exist two input letters that are non-confusable in each state:

$\exists x, x^{'} \in X \forall y \in Y, s \in S W (y | x, s) W (y | x^{'}, s) = 0.$ (19)
$Cc,c0,f,\textscbl>0$ if and only if in each state there exist two non-confusable input letters:

$\forall s \in S \exists x, x^{'} \in X \forall y \in Y W (y | x, s) W (y | x^{'}, s) = 0.$ (20)
$Cnc,c0,f,\textscbl>0$ if and only if (20) holds.
$Cnc,nc0,f,\textscbl>0$ if and only if (20) holds.

Proof:

We first show that fixed-length and bounded-length zero-error feedback capacities are either both positive or both zero. Since $Csi0,f,\textscbl≥Csi0,f,\textscfl$ , the “if part” is clear. Conversely, suppose that $Csi0,f,\textscbl>0$ . Then, for some $0 < k \leq n < \infty$ , there exists a zero-error code of cardinality at least $2$ , average length $k$ , and maximum length $n$

. By “zero-padding” the codewords we can then construct a fixed-length zero-error code of length

n

having the same cardinality, which implies that

Csi0,f,\textscfl≥1n>0

By the observation from the previous paragraph, we can focus on fixed-length codes in the rest of the proof.

(a)–(d) The conditions (16)–(18) for positivity of

Csi0,f,\textscfl

in cases when only the transmitter has state information were derived in [2, Thm 3, Thm 10, Rem. 17]. Note that (16) is the condition for positivity of the zero-error fixed-length feedback capacity of the DMC

˜ W (y | x) = \sum_{s \in S} Q (s) W (y | x, s)

; see (15). (e) The case

si = (sc, c)

is solved by using the standard technique of treating the output

Y

and the state

S

as a joint output

(Y, S)

of the DMC

¯¯¯¯¯¯W(y,s|x)\coloneqqQ(s)W(y|x,s)

with noiseless feedback. (Before the

n

’th channel use the transmitter obtains

Y^{n - 1}

through feedback and

S^{n - 1}

as side information, which is equivalent to saying that it obtains the previous outputs of

¯ ¯¯¯¯ ¯ W

, namely

(Y, S)^{n - 1}

, through feedback.) The condition (19) is the condition for positivity of the zero-error capacity of this DMC; see (15). It is therefore easy to see that this condition is sufficient for

Csc,c0,f,\textscfl>0

. To show that (19) is also necessary, suppose for the sake of contradiction that every two input letters

x, x^{'}

are confusable in some state

s_{x, x^{'}}

. Then, for every

n

, every two input sequences

x_{1} \dots x_{n}

x_{1}^{'} \dots x_{n}^{'}

can produce the same channel output if the state sequence realized in the channel happens to be

s_{x_{1}, x_{1}^{'}} \dots s_{x_{n}, x_{n}^{'}}

, meaning that error-free communication with fixed-length codes is impossible. Stated informally, since the transmitter does not know the current state, no matter which letter

x

it chooses to send in the current time slot there is a chance that the state will be one in which

x

is confusable with another letter

x^{'}

. Thus, error-free communication cannot be guaranteed, no matter how long the codewords are. (f) Consider now the case

si = (c, c)

. If for every state

s

there exists a pair of non-confusable inputs

x_{s}, x_{s}^{'}

, which is what the condition (20) means, then the transmitter and the receiver can agree beforehand for

x_{s}

to mean

0

and

x_{s}^{'}

to mean

1

. In this way, one bit can be transmitted error free in one channel use and so

Cc,c0,f,\textscfl>0

. Conversely, if (20) is not satisfied, then there exists a state

s

for which every two inputs are confusable. If this is the case, then for the state sequence

s^{n} = s \dots s

it is not possible to transmit one bit error-free in any number of channel uses

n

, and hence

Cc,c0,f,\textscfl=0

. (g), (h) The previous argument holds in these cases too. Namely, knowing the future states cannot help the encoder/decoder if these states remain unfavorable throughout the entire transmission.

∎

As in the (unbounded) variable-length case, whenever the zero-error feedback capacity under bounded-length coding is positive, it equals the vanishing-error capacity of the same channel.

Theorem 6.

Let $si \in SI$ . If $Csi0,f,\textscbl>0$ , then $Csi0,f,\textscbl=Csi↓$ .

Proof:

The proof is analogous to the proof of Theorem 3 for the variable-length case; the only difference is that the zero-error code that is used in the second phase of the protocol is now required to be of bounded length. ∎

V State Information at the Decoder Only

In this section we discuss SD-DMCs with state information available only at the decoder, the case that has been left out of the discussion thus far. As pointed out in Remark 1, in this channel model it is not quite clear how to define the code length for variable-length codes, i.e., the stopping time of the transmission (see Definition 1). Since the decoder makes a decision based on the outputs $Y^{n}$ and states $S^{n}$

, the encoder, not knowing the states, is not able to exactly simulate the decoding process and to determine the moment when the decision has been made. It can only provide an estimate of this moment based on the outputs

Y^{n}

which it obtains through feedback. As we shall see, this estimate is good enough when one considers coding with asymptotically vanishing error probability (in fact, it is not even necessary as the vanishing-error capacity can be achieved with fixed-length codes, with or without feedback). However, in the case of zero-error communication the encoder has to be certain that the decoding was successful before it stops transmitting a given message and starts transmitting the next message. It is in this case that the effects of the mismatch in state information at the two sides are most apparent.

V-a Vanishing-Error Capacity

We know from [5, Ch. 7.4] and (7) that:

C-,c↓,-,\textscfl=maxPXI(X;Y|S)=C% sc,c↓,f,\textscvl.

(21)

Note that the issue with the stopping time mentioned above does not arise when defining the two capacities in (21): for $C-,c↓,-,\textscfl$ because in the fixed-length setting the stopping time is fixed in advance, and for $Csc,c↓,f,\textscvl$ because in this case both sides have state information.

The quantity we are interested in here, $C-,c↓,f,\textscvl$ , has not been formally defined. However, it is clear that, for any reasonable definition, the following chain of inequalities must hold: $C-,c↓,-,\textscfl≤C-,c% ↓,f,\textscbl≤C-,%c↓,f,\textscvl≤Csc,c↓,f,\textscvl$ . From this and (21) we then conclude that

C-,c↓,f,\textscvl=C-,c↓,f,\textscbl=maxPXI(X;Y|S)=Csc,c↓.

(22)

In particular, the VLF capacity of the $(-, c)$ channel can be achieved by using fixed-length codes.

V-B Zero-Error Capacity: Bounded-Length Codes

We now turn to the zero-error problems and start with the bounded-length case. The following theorem gives a necessary and sufficient condition for positivity of the zero-error capacity in this setting.

Theorem 7.

The following statements are equivalent: (a) $C-,c0,f,\textscfl>0$ ; (b) $C-,c0,f,\textscbl>0$ ; (c) $Csc,c0,f,\textscbl>0$ ; (d) (19) holds.

Proof:

(a)

\Leftrightarrow

(b): This can be shown by “zero-padding” bounded-length codes to obtain fixed-length codes, as for the other models of state information availability (see Theorem 5). (b)

\Rightarrow

(c): This follows from

Csc,c0,f,\textscbl≥C-,c0,f,\textscbl

. (c)

\Leftrightarrow

(d): This was shown in Theorem 5(e). (d)

\Rightarrow

(a): Suppose that (d) holds, i.e., there exist two input letters

x, x^{'}

that are non-confusable in every state. Then, if the transmitter sends

x

for

0

and

x^{'}

for 1, the receiver will be able to tell from the output which of the two possibilities is the correct one because it knows the channel state. Therefore, one bit can be transmitted in one channel use, and hence

C-,c0,f,\textscfl≥1>0

∎

We now know from Theorem 7 that $C-,c0,f,\textscbl>0⇔Csc,%c0,f,\textscbl>0$ , from Theorem 6 that $Csc,c0,f,\textscbl=Csc,c↓$ whenever $Csc,c0,f,\textscbl>0$ , and from (22) that $C_{↓}^{sc, c} = C_{↓}^{-, c}$ . It is then natural to ask if it is also true that $C-,c0,f,\textscbl=C-,c↓$ whenever $C-,c0,f,\textscbl>0$ ? The corresponding equality for the other models of state information availability $si \in SI$ has been established in Theorem 6, but the proof technique used there does not apply when $si = (-, c)$ . The difficulty is precisely due to the fact we mentioned at the beginning of this section—in this model the transmitter cannot exactly simulate the decoding process because it does not know the states. Hence, the transmitter is in general not able to decide with certainty whether the receiver has decoded the received sequence correctly, and whether it should retransmit the same codeword or start sending the next codeword. Therefore, for $si = (-, c)$ it is not clear whether the vanishing-error capacity can always be achieved with zero-error bounded-length codes. We next give an example of a channel for which the answer to this question is affirmative.

Example 1.

Consider the following binary-input-binary-output channel with two states: in state $s_{0}$ the channel flips the input bit, $W (1 | 0, s_{0}) = W (0 | 1, s_{0}) = 1$ , and in state $s_{1}$ it leaves the bit intact, $W (0 | 0, s_{1}) = W (1 | 1, s_{1}) = 1$ .

Suppose that the transmitter sends $x$ in the $n$ ’th slot and that $y$ is produced at the channel output. After the transmitter obtains $y$ through the feedback link, it can easily determine the $n$ ’th state: the state is $s_{1}$ if and only if $x = y$ . This means that the transmitter effectively obtains (strictly causal) state information through feedback and therefore $C-,c0,f,\textscbl=Csc,c0,f,\textscbl=Csc,c↓$ , where the second equality holds because $Csc,c0,f,\textscbl>0$ (see (19)). $▲$

The main point in Example 1 is the following: if the states are uniquely determined by the channel inputs and outputs, then the problem of calculating $C-,c0,f,\textscbl$ is reduced to the (easier) problem of calculating $Csc,c0,f,\textscbl$ , which was solved in Theorems 5(e) and 6. Whether such a reduction is possible in general is an interesting question that we leave for future work.

We end this subsection with a remark that the problem of determining the value of $C-,c0,f,\textscbl$ is related to the problem of determining the zero-error capacity under bounded-length coding of DMC’s with noisy feedback. (The latter problem is also unsolved except in some special cases, see [9, 1].) Namely, as already noted in the proof of Theorem 5(e), the standard trick of dealing with state information at the decoder is to think of it as part of a joint channel output $(Y, S)$ of the DMC $¯¯¯¯¯¯W(y,s|x)\coloneqqQ(s)W(y|x,s)$ . Noiseless feedback in the DMC $¯ ¯¯¯¯ ¯ W$ would mean that the transmitter is given $(Y, S)^{n - 1}$ before the $n$ ’th channel use. This implies that an SD-DMC $W$ with $si = (sc, c)$ and noiseless feedback is equivalent to the DMC $¯ ¯¯¯¯ ¯ W$ with noiseless feedback. However, in the case $si = (-, c)$ this equivalence fails as the transmitter now obtains only a “degraded” version ( $Y^{n - 1}$ ) of the joint output ( $(Y, S)^{n - 1}$ ) through the feedback link.

V-C Zero-Error Capacity: Variable-Length Codes

For variable-length codes, we can only give a sufficient condition for positivity of the zero-error capacity at this point. The idea behind this condition is based on the fact that, for some channels, the transmitter obtains state information “for free” through the feedback link, in which case the $(-, c)$ model is effectively reduced to the $(sc, c)$ model (see Example 1). As we show in Theorem 8, it is in fact not necessary that all states be uniquely determined by inputs and outputs in order for this reduction to work—it is enough that only a group of states exists that contains a “disprover” and that is discernible from other states with positive probability.

Theorem 8.

A sufficient condition for $C-,c0,f,\textscvl>0$ is the following:

	$\exists x, x^{'}$	$\in X, y \in Y, S^{} \subseteq S, S^{} \neq \emptyset$		(23)
		$(\forall s \in S^{} W (y \| x^{'}, s) > 0 \land W (y \| x, s) = 0) \land (\forall s \in S ∖ S^{} W (y \| x^{'}, s) = 0) .$		(23)

Proof:

Let us first parse the condition (23). The meaning of the statement $W (y | x^{'}, s) > 0 \land W (y | x, s) = 0$ is the same as before: $y$ is a disprover for $x$ (in the group of states $S^{*}$ ). Further, we require the existence of an event ( $x^{'} \to y$ ) that has positive probability in the group of states $S^{*}$ , but is impossible in other states. The occurrence of this event can serve to the transmitter as an identifier of the group of states $S^{*}$ .

Now, assuming that (23) holds, one bit of information can be sent with zero error as follows. In the first two channel uses the transmitter sends $x, x^{'}$ for $0$ and $x^{'}, x$ for 1. If the letters obtained at the output are $\neg y, y$ and the state in the second slot is from $S^{*}$ , then the receiver concludes that $0$ must have been sent. The reason is the following: since the received symbol in the second slot is $y$ and the state is from $S^{*}$ (the receiver can see the states), the transmitted symbol must have been $x^{'}$ , and then it automatically follows that the symbol sent in the first slot is $x$ . Furthermore, the transmitter is also assured that the state in the second slot is from $S^{*}$ and that the receiver has received the bit correctly because the transition $x^{'} \to y$ is only possible in states from $S^{*}$ . Similarly, if the letters obtained at the output are $y, \neg y$ and the state in the first slot is from $S^{*}$ , then the receiver concludes that $1$ must have been sent, and the transmitter is assured that the receiver has received the bit correctly. In summary, if the channel output is either $\neg y, y$ or $y, \neg y$ and the state in the slot in which the output is $y$ is from $S^{*}$ , then the protocol stops and both parties agree on the value of the transmitted bit. If any other situation occurs, the procedure is repeated. The expected number of steps needed to complete the transmission of the bit is finite because the event that both parties are waiting for ( $x^{'}$ produces $y$ in a state from $S^{*}$ ) has positive probability. Therefore, $C-,c0,f,\textscvl>0$ . ∎

Note that the condition (10) for $C-,-0,f,\textscvl>0$ , together with (2), implies the condition (23). To see that it does, suppose that (10) holds and let $S^{*} = {s : W (y | x^{'}, s) > 0}$ . This is kind of a sanity check because clearly $C-,-0,f,\textscvl>0$ implies $C-,c0,f,\textscvl>0$ . The reverse implication does not hold because in (10) we require that $W (y | x, s) = 0$ in all states, rather than just in states from $S^{*}$ as in (23).

Note also that the sufficient condition for $C-,c0,f,\textscvl>0$ given in (23) is different from the necessary and sufficient condition for $Csc,c0,f,\textscvl>0$ given in (12). Based on Theorem 7, one might wonder whether it holds that $C-,c0,f,\textscvl>0⇔Csc,%c0,f,\textscvl>0$ , i.e., whether (12) is also necessary and sufficient for $C-,c0,f,\textscvl>0$ ? This is, however, not the case. In the proof of the following theorem we give an example of a channel for which $C-,c0,f,\textscvl=0$ and yet $Csc,c0,f,\textscvl>0$ . In other words, strictly causal state information at the transmitter can in some cases enable the parties to communicate error free, even if they were not able to do that in the absence of this information. This is a somewhat curious fact having in mind that in most other settings studied so far, strictly causal state information at the transmitter has been shown equivalent (in the sense of achievable rates) to no state information: $C_{↓}^{-, -} = C_{↓}^{% s c, -}$ , $C_{↓}^{-, c} = C_{↓}^{% s c, c}$ , $C-,-0,f,\textscbl=Csc,-0,f,\textscbl$ , $C-,-0,f,\textscvl=Csc,-0,f,\textscvl$ (see also Theorem 7).

Theorem 9.

There exists an SD-DMC for which $C-,c0,f,\textscvl=0$ and $Csc,c0,f,\textscvl>0$ .

Proof:

Consider the following binary-input-binary-output channel with two states: in state $s_{0}$ the channel is a BSC( $p$ ), where $0 < p < 1$ , i.e., $W (0 | 0, s_{0}) = W (1 | 1, s_{0}) = 1 - p$ , and in state $s_{1}$ the channel is noiseless, $W (0 | 0, s_{1}) = W (1 | 1, s_{1}) = 1$ .

By Theorem 2(e) we know that $Csc,c0,f,\textscvl>0$ . It is easy to see why this is the case—the transmitter and the receiver can agree on the codeword termination by using the noiseless state since they both know the states (the transmitter obtains the state information with a one-slot delay, but this can be circumvented by sending a dummy letter; see the proof of Theorem 2(e)).

However, if the transmitter has no state information, i.e., if $si = (-, c)$ , then it is not possible to communicate with zero-error in a finite expected number of channel uses. To see this, suppose that the transmitter is trying to send one bit through the channel by using a repetition code. The receiver can see the channel states and will therefore recover the bit correctly as soon as the state happens to be

	$\exists u \in U W^{'} (y \| u) = 0$	$\Leftrightarrow \exists u \in U \forall s \in S W (y \| u (s), s) = 0$		(13)
		$\Leftrightarrow \forall s \in S \exists x \in X W (y \| x, s) = 0,$		(14)

Error-Free Communication Over State-Dependent Channels with Variable-Length Feedback

Authors

Related Research

Feedback Capacity and Coding for the (0,k)-RLL Input-Constrained BEC

Variable-Length Coding for Zero-Error Channel Capacity

Zero-Error Feedback Capacity of Finite-State Additive Noise Channels for Stabilization of Linear Systems

Minimax Converse for Identification via Channels

Variable-Length Source Dispersions Differ under Maximum and Average Error Criteria

Real-Time Variable-to-Fixed Lossless Source Coding of Randomly Arriving Symbols

Algorithms for q-ary Error-Correcting Codes with Limited Magnitude and Feedback

This week in AI

I Introduction

I-a The Channel Model, Definitions, and Notation

Definition 1.

Remark 1.

Ii Vanishing-Error Capacity

Theorem 1.

Proof:

Iii Zero-Error Capacity: Variable-Length Codes

Theorem 2.

Proof:

Remark 2 (Shannon strategy).

Remark 3.

Theorem 3.

Proof:

Theorem 4.

Proof:

Iv Zero-Error Capacity: Bounded-Length Codes

Theorem 5.

Proof:

Theorem 6.

Proof:

V State Information at the Decoder Only

V-a Vanishing-Error Capacity

V-B Zero-Error Capacity: Bounded-Length Codes

Theorem 7.

Proof:

Example 1.

V-C Zero-Error Capacity: Variable-Length Codes

Theorem 8.

Proof:

Theorem 9.

Proof: