I Introduction
The noisy 20 questions problem (cf. [12, 2, 18, 10, 6, 4, 9]) arises when one aims to accurately estimate an arbitrarily distributed random variable by successively querying an oracle and using its noisy responses to form an estimate . A central goal in this problem is to find optimal query strategies that yield a good estimate for the unknown target random variable .
Depending on the query design strategy adopted, the 20 questions problem can either be adaptive or nonadaptive. In adaptive query procedures, the design of a subsequent query depends on all previous queries and noisy responses to these queries from the oracle. In nonadaptive query procedures, all the queries are designed independently in advance. For example, the bisection policy [6, Section 4.1] is an adaptive query procedure and the dyadic policy [6, Section 4.2]
is a nonadaptive query procedure. Compared with adaptive query procedures, nonadaptive query procedures have the advantage of lower computation cost, parallelizability and no need for feedback. Depending on whether or not the noisy responses depend on the queries, the noisy 20 questions problem is classified into two categories: querying with measurementindependent noise (e.g.,
[6, 4]); and querying with measurementdependent noise (e.g., [7, 9]). As argued in [7], measurementdependent noise better models practical applications. For example, for target localization in a sensor network, the noisy response to each query can depend on the size of the query region due to possible presence of clutter. Another example is in human query systems where personal biases abut the state may affect the response.In earlier works on the noisy 20 questions problem, e.g., [6, 16, 17], the queries were designed to minimize the entropy of the posterior distribution of the target variable . As pointed out in later works, e.g., [4, 3, 7, 9], other accuracy measures, such as the estimation resolution and the quadratic loss are often better criteria for localization, where the resolution is defined as the absolute difference between and its estimate , , and the quadratic loss is .
Motivated by the scenario of limited resources, computation and response time, we obtain new results on the nonasymptotic tradeoff among the number of queries , the achievable resolution and the excessresolution probability of optimal adaptive and nonadaptive query procedures for noisy 20 questions estimation of an arbitrarily distributed random variable taking values in the alphabet .
Our contributions for nonadaptive querying are as follows. First, we derive nonasymptotic bounds of optimal nonadaptive query procedures for any number of queries and any excessresolution probability
. Secondly, applying the BerryEsseen theorem, under mild conditions on the measurementdependent noise, we obtain a secondorder asymptotic approximation to the achievable resolution of optimal nonadaptive query procedures with a finite number of queries. As a corollary of our result, we establish a phase transition in the excessresolution probability as a function of the resolution decay rate for optimal nonadaptive query procedures. Finally, we specialize our secondorder analyses to measurementdependent versions of the binary symmetric channel.
Finally, we clarify the differences between our work and [7]. First of all, our results hold for arbitrary discrete channels under mild conditions, while the results in [7] focused only on a measurementdependent binary symmetric channel. Furthermore, our proof techniques are significantly different from [7]. The authors in [7] used large deviations analysis to prove the achievability part and the Fano’s inequality for the converse part. In contrast, our proofs use recent advances in finite blocklength information theory [11, 15]. It is important to note that our nonasymptotic bounds for nonadaptive query schemes are novel and there are no such comparable results in the previous literature including [7]. Finally, our secondorder asymptotic result in Theorem 3 refined [7, Theorem 1]. In particular, Theorem 3 provides an approximation to the performance of optimal query procedures employing a finite number of queries while [7, Theorem 1] only characterizes the asymptotic performance when the number of queries tends to infinity.
Ii Problem Formulation
Notation
Random variables and their realizations are denoted by upper case variables (e.g., ) and lower case variables (e.g., ), respectively. All sets are denoted in calligraphic font (e.g., ). Let
be a random vector of length
. We useto denote the inverse of the cumulative distribution function (cdf) of the standard Gaussian. We use
, and to denote the sets of real numbers, positive real numbers and integers respectively. Given any two integers , we use to denote the set of integers and use to denote . Given any , for any matrix by matrix , the infinity norm is defined as. The set of all probability distributions on a finite set
is denoted asand the set of all conditional probability distributions from
to is denoted as . Furthermore, we useto denote the set of all probability density functions on a set
. All logarithms are base unless otherwise noted. Finally, we use to denote the indicator function.Iia Noisy 20 Questions Problem
Let
be a continuous random variable defined on the unit interval
with arbitrary probability density function (pdf) . In the noisy 20 questions problem, a player aims to accurately estimate the value of the target random variable by posing a sequence of queries to an oracle knowing . After receiving the queries, the oracle finds binary answers and passes these answers through a measurementdependent channel with transition matrix yielding noisy responses . Given the noisy responses , the player uses a decoding function to obtain an estimate of the target variable . Throughout the paper, we assume that the alphabet for the noisy response is finite.A query procedure for the noisy 20 questions problem consists of the queries and the decoder . In general, these procedures can be classified into two categories: nonadaptive and adaptive querying. In a nonadaptive query procedure, the player needs to first determine the number of queries and then design all the queries simultaneously. In contrast, in an adaptive query procedure, the design of queries is done sequentially and the number of queries is a variable. In particular, when designing the th query, the player can use the previous queries and the noisy responses from the oracle to these queries, i.e., , to formulate the next query . Furthermore, the player needs to choose a stopping criterion, which may be random, determining the number of queries to make.
In subsequent sections, we clarify the notion of the measurementdependent channel including concrete examples and we present specific definitions of nonadaptive and adaptive query procedures.
IiB The MeasurementDependent Channel
In this subsection, we describe succinctly the measurementdependent channel scenario [7], also known as a channel with state [5, Chapter 7]. Given a sequence of queries , the channel from the oracle to the player is a memoryless channel whose transition probabilities are functions of the queries. Specifically, for any ,
(1) 
where denotes the transition probability of the channel which depends on the th query . Given any Lebesgue measurable query , define the size of as its Lebesgue measure, i.e., . Throughout the paper, we consider only Lebesgue measurable queries and assume that the measurementdependent channel depends on the query only though its size, i.e., is equivalent to a channel with state where the state .
For any , any and any subsets , and of with sizes , and , we assume the measurementdependent channel is continuous in the sense that there exists a constant depending on only such that
(2) 
A particular example of a measurementdependent channel satisfying the continuous constraint in (2) is given in Definition 1. See [19] for other examples.
Definition 1.
Given any , a channel is said to be a measurementdependent Binary Symmetric Channel (BSC) with parameter if and for any ,
(3) 
IiC NonAdaptive Query Procedures
A nonadaptive query procedure with resolution and excessresolution constraint is defined as follows.
Definition 2.
Given any , and , an nonadaptive query procedure for the noisy 20 questions problem consists of

queries ,

and a decoder
such that the excessresolution probability satisfies
(4) 
We remark that the definition of the excessresolution probability with respect to is inspired by ratedistortion theory [1, 8]. Our formulation differs from that of [7] where the authors constrained the dependent maximum excessresolution probability, where is the target variable.
Motivated by practical applications where the number of queries are limited (e.g., due to the high cost of queries and lowdelay requirement), we are interested in the following nonasymptotic fundamental limit on achievable resolution :
(5) 
Note that denotes the minimal resolution one can achieve with probability at least using a nonadaptive query procedure with queries. In other words, is the achievable resolution of optimal nonadaptive query procedures tolerating an excessresolution probability of . Dual to (5) is the sample complexity, determined by the minimal number of queries required to achieve a resolution with probability at least , i.e.,
(6) 
One can easily verify that for any ,
(7) 
Thus, it suffices to focus on the fundamental limit .
Iii Main Results
The proof of all results are omitted due to space limitation. Details are available in our extended version [19].
Iiia NonAsymptotic Bounds
We first present an upper bound on the error probability of optimal nonadaptive query procedures. Given any , let be the marginal distribution on
induced by the Bernoulli distribution
and the measurementdependent channel . Furthermore, define the following information density(8) 
Correspondingly, for any , we define
(9) 
as the mutual information density between and .
Theorem 1.
Given any , for any and any , there exists an nonadaptive query procedure such that
(10) 
where the tuple of random variables is distributed as with defined as the Bernoulli distribution with parameter (i.e., ).
Consider the measurementindependent channel where for all . It is straightforward to verify that for any , there exists an nonadaptive query procedure such that
(11) 
where the tuple of random variables is distributed as , the information density is defined as
(12) 
and is induced by and . Comparing the measurementindependent case (11) with the measurementdependent case (10), the nonasymptotic upper bound (10) in Theorem 1 differs from (11) in two aspects: an additional additive term and an additional multiplicative term in (10). As is made clear in the proof of Theorem 1, the additive term results from the atypicality of the measurementdependent channel and the multiplicative term appears due to the changeofmeasure we use to replace the measurementdependent channel with the measurementindependent channel .
We next provide a nonasymptotic converse bound to complement Theorem 1. For simplicity, for any query and any , we use to denote .
Theorem 2.
Set . Any nonadaptive query procedure satisfies the following. For any and any ,
(13) 
The proof of Theorem 2 is decomposed into two steps: i) we use the result in [7] which states that the excessresolution probability of any nonadaptive query procedure can be lower bounded by the error probability associated with channel coding over the measurementdependent channel with uniform message distribution, minus a certain term depending on ; and ii) we apply the nonasymptotic converse bound for channel coding [15, Proposition 4.4] by exploiting the fact that, given a sequence of queries, the measurementdependent channel is simply a time varying channel with deterministic states at each time point.
The nonasymptotic bounds in Theorems 1 and 2 lead to a secondorder asymptotic result in Theorem 3, which provides an approximation to the finite blocklength fundamental limit . Furthermore, the exact calculation of the upper bound in Theorem 2 is challenging. However, for sufficiently large, as shown in the proof of Theorem 3, the supremum in (13) can be achieved by queries where each query has the same size.
IiiB SecondOrder Asymptotic Approximation
In this subsection, we present the secondorder asymptotic approximation to the achievable resolution of optimal nonadaptive query procedures after queries subject to a worst case excessresolution probability of .
Given measurementdependent channels , the channel “capacity" is defined as
(14) 
where .
Let the capacityachieving set be the set of optimizers achieving (14). Then, for any , define the following “dispersion” of the measurementdependent channel
(15) 
The case of will be the focus of the sequel of this paper. We assume that for any
, the third absolute moment of
is finite. Under this assumption, we obtain the secondorder asymptotic result.Theorem 3.
For any , the achievable resolution of optimal nonadaptive query procedures satisfies
(16) 
where the remainder satisfies that .
We make the following remarks.
Firstly, Theorem 3
implies a phase transition in a machine learning sense
[14, 13], which we interpret in Figure 1.We remark that phase transition only appears in the secondorder asymptotic analysis and is not revealed by the firstorder asymptotic analysis, e.g., that developed in
[7, Theorem 1].Secondly, Theorem 3 refines [7, Theorem 1] in several directions. First, Theorem 3 is a secondorder asymptotic result that provides good approximation for the finite blocklength performance while [7, Theorem 1] only characterizes the asymptotic resolution decay rate with vanishing worstcase excessresolution probability, i.e., . Second, our results hold for any measurementdependent channel satisfying (2) while [7, Theorem 1] only considers the measurementdependent BSC.
Thirdly, the dominant event which leads to an excessresolution in noisy 20 questions estimation is the atypicality of the information density (cf. (9)). To characterize the probability of this event, we make use of the Berry–Esseen theorem and show that the mean
and the variance
of the information density play critical roles.Finally, we remark that any real number has the binary expansion . We can thus interpret the result in Theorem 3 as follows: using optimal nonadaptive query procedures, after queries, with probability of at least , one can extract the first bits of the binary expansion of the target variable .
IiiC Case of MeasurementDependent BSC
In the following, we specialize Theorem 3 to a measurementdependent BSC. Given any and any , let . For any , the information density of a measurementdependent BSC with parameter is
(17) 
It can be verified that the capacity of the measurementdependent BSC with parameter is given by
(18) 
where is the binary entropy function.
Depending on the value of , the set of capacityachieving parameters may or may not be singleton. In particular, for any , the capacityachieving parameter is unique. When , there are two capacityachieving parameters and where . It can be verified easily that . As a result, for any capacityachieving parameter of the measurementdependent BSC with parameter , the dispersion of the channel is
(19) 
Corollary 4.
Set any . If the channel from the oracle to the player is a measurementdependent BSC with parameter , then Theorem 3 holds with and for any .
We make the following observations.
Second, when one considers the measurementindependent BSC with parameter , then it can be shown that the achievable resolution of optimal nonadaptive query procedures satisfies
(21) 
To compare the performances of optimal nonadaptive query procedures under measurementdependent and measurementindependent channels, we plot in Figure 2 the secondorder approximation to the average number of bits (in the binary expansion of the target random variable ) extracted per query after queries, i.e., and for and different values of (the remainder is ignored). We observe an interesting phenomenon. When , optimal query procedures under a measurementdependent channel achieve a higher resolution than their counterpart in the measurementindependent case. Intuitively, this is because the probability of receiving wrong answers in the measurementdependent channel is smaller compared with the measurementindependent channel with the same parameter. However, when , we find that the relative performances can be reversed. The reasons for this phenomenon are two fold: i) BSC is a symmetric channel, thus under the measurementindependent setting, having a BSC with crossover probability is equivalent to having a BSC with parameter since one can easily flip all bits; ii) under the measurementdependent setting, since the probability of receiving wrong answers depends on the size of the query, this symmetric nature of BSC is lost.
Iv Numerical Illustration
In this section, we numerically illustrate the minimal achievable resolution of nonadaptive procedures over a measurementdependent BSC with parameter . We consider the case where the target random variable
is uniformly distributed over the alphabet
and set the target excessresolution probability . The simulation results for this case is provided in Figure 3, which demonstrate strong agreement with the theoretical result in Corollary 4.V Conclusion
We derived the minimal achievable resolution of nonadaptive query procedures for the noisy 20 questions problem where the channel from the oracle to the player is a measurementdependent discrete channel. In our extended version [19], we generalize our results to estimate a multidimensional target over the unit cube and to simultaneously estimate multiple targets. Furthermore, we establish a lower bound on the resolution gain associated with adaptive querying for estimating a target over the unit interval.
In this paper, we were interested in fundamental limits of optimal query procedures. It would be interesting to explore lowcomplexity practical query procedures and compare the performances of proposed query procedures to our derived benchmarks.
References
 [1] (1971) Ratedistortion theory. Wiley Online Library. Cited by: §IIC.
 [2] (1974) An interval estimation problem for controlled observations. Problemy Peredachi Informatsii 10 (3), pp. 51–61. Cited by: §I.
 [3] (2016) Sequential measurementdependent noisy search. In 2016 IEEE Information Theory Workshop (ITW), pp. 221–225. Cited by: §I.
 [4] (2018) Unequal error protection querying policies for the noisy 20 questions problem. IEEE Trans. Inf. Theory 64 (2), pp. 1105–1131. Cited by: §I, §I, §I.
 [5] (2011) Network information theory. Cambridge University Press. Cited by: §IIB.
 [6] (2012) Twenty questions with noise: bayes optimal policies for entropy loss. Journal of Applied Probability 49 (1), pp. 114–136. Cited by: §I, §I, §I.
 [7] (2018) Searching with measurement dependent noise. IEEE Trans. Inf. Theory 64 (4), pp. 2690–2705. Cited by: §I, §I, §I, §IIB, §IIB, §IIC, §IIIA, §IIIB, §IIIB, §IIIC.
 [8] (2013) Lossy data compression: nonasymptotic fundamental limits. Ph.D. Thesis, Department of Electrical Engineering, Princeton University. Cited by: §IIC.
 [9] (2018) Improved target acquisition rates with feedback codes. IEEE Journal of Selected Topics in Signal Processing 12 (5), pp. 871–885. Cited by: §I, §I, §I.
 [10] (2002) Searching games with errors—fifty years of coping with liars. Theoretical Computer Science 270 (12), pp. 71–109. Cited by: §I.
 [11] (2010) Channel coding: nonasymptotic fundamental limits. Ph.D. Thesis, Department of Electrical Engineering, Princeton University. Cited by: §I.
 [12] (1961) On a problem of information theory. MTA Mat. Kut. Int. Kozl. B 6, pp. 505–516. Cited by: §I.
 [13] (2016) Phase transitions in group testing. In Proceedings of the Twentyseventh Annual ACMSIAM Symposium on Discrete Algorithms, SODA ’16, Philadelphia, PA, USA, pp. 40–53. External Links: ISBN 9781611974331 Cited by: §IIIB.
 [14] (2017) Phase transitions in the pooled data problem. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), pp. 377–385. Cited by: §IIIB.
 [15] (2014) Asymptotic estimates in information theory with nonvanishing error probabilities. Foundations and Trends® in Communications and Information Theory 11 (1–2), pp. 1–184. Cited by: §I, §IIIA.
 [16] (2014) Collaborative 20 questions for target localization. IEEE Trans. Inf. Theory 60 (4), pp. 2233–2252. Cited by: §I.
 [17] (2015) On decentralized estimation with active queries. IEEE Transactions on Signal Processing 63 (10), pp. 2610–2622. Cited by: §I.
 [18] (1991) Adventures of a mathematician. Univ of California Press. Cited by: §I.
 [19] (2019) Resolution limits of noisy 20 questions estimation. arxiv 1909.12954. Cited by: §IIB, §III, §V.
Comments
There are no comments yet.