Large-scale distributed inference has taken a center stage in many machine learning tasks. In these settings, it is becoming increasingly critical to operate under limited resources at each player, where the players may be limited in their computational capabilities, communication capabilities, or may restrict the information about their data to maintain privacy. Our focus in this work will be on the last two constraints of communication and privacy, and, in general, on local information constraints on each player’s data.
We propose a general framework for distributed statistical inference under local information constraints. Consider a distributed model where players observe independent samples from an unknown distribution on , with player getting the sample . The players are constrained in the amount of information they can reveal about their observations in the following way: Player must choose a channel from a prespecified class of channels to report its observed sample to a central referee . In particular, player passes its observation as input to its chosen channel and receives the corresponding channel output . The central referee uses messages
from the players to complete an inference task such as estimation or testing on the underlying distribution. See Fig. 1 for an illustration of the setup.
The family of allowed channels serves as an abstraction of the information constraints placed on each player’s data. Before moving ahead, we instantiate our abstract setup with two important examples of local communication constraints and local privacy constraints by specifying the corresponding ’s.
Communication-Limited Inference. Each player can only send bits about their sample. This limitation can be captured by restricting to , the family of channels with output alphabet , i.e., for ,
Locally Differentially Private Inference. Each player seeks to maintain privacy of their own data. We adopt the notion of local differential privacy which, loosely speaking, requires that no output message from a player reveals too much about its input data. This is captured by restricting to , the family of -locally differentially private (-LDP) channels that satisfy the following (cf. [21, 35, 9, 19]): For ,
These specific cases of communication and privacy constraints have received a lot of attention in the literature, and we emphasize these cases separately in our results. Nonetheless, our results are valid for arbitrary families and can handle other examples from the literature such as the -step Markov transition matrices considered in .
Our proposed framework can be applied to inference for belonging to any family of distributions . For simplicity, however, in this work we restrict ourselves to a finite alphabet and consider the canonical inference problems of estimating and testing goodness-of-fit. Motivated by applications in distributed inference in a resource-constrained setting, we seek algorithms that enable the desired inference using the least number of samples, or equivalently, the least number of players. Our main results present a general approach for establishing lower bounds on the sample complexity of performing a given inference task under the aforementioned information-constrained setting. Underlying our lower bounds is a new quantitative characterization of contraction in chi-square distance between distributions of observations due to imposed information constraints.
We allow randomized selection of ’s from at each player and distinguish between private-coin protocols, where this randomized selection is done independently for each player, and public-coin protocols, where the players can use shared randomness. Interestingly, our chi-square contraction bounds provide a quantitative separation of sample complexity for private-coin and public-coin protocols, an aspect hitherto ignored in the distributed inference literature and which is perhaps the main contribution of our work.
We summarize our results below, after a formal description of our problem setting.
I-a Information-constrained inference framework
We begin by recalling standard formulations for learning and testing discrete distributions. Denote by the set of all distributions over . In this work, we consider observation alphabet and the set of unknown distributions . Let be independent samples from an unknown distribution . We focus on the following two inference tasks for .
Distribution Learning. In the -distribution learning problem, we seek to estimate a distribution in to within in total variation distance. Formally, a (randomized) mapping constitutes an -estimator if
where denotes the total variation distance between and (see Section II for definition of total variation distance). Namely, estimates the input distribution to within distance
with probability at least. Note that this choice of probability is arbitrary and can be replaced with any positive constant.
The sample complexity of -distribution learning is the minimum such that we can find an -estimator for . It is well-known that the sample complexity of distribution learning is and the empirical distribution estimate attains it.
Identity Testing. Given a known reference distribution , in the -identity testing problem, we seek to use samples from to test if equals or if it is -far from in total variation distance. Specifically, an -test is given by a (randomized) mapping such that
Namely, upon observing independent samples , the algorithm should “accept” with high constant probability if the samples come from the reference distribution and “reject” with high constant probability if they come from a distribution significantly far from . Note again that this choice of probability is arbitrary and can be replaced with any constant greater than .
The sample complexity of -identity testing is the minimum such that we can find an -test for . Clearly, this quantity will depend on the reference distribution . However, it is customary to consider sample complexity over the worst-case .111The sample complexity for a fixed has been studied under the “instance-optimal” setting (see [46, 11]): while the question is not fully resolved, nearly tight upper and lower bounds are known. In this worst-case setting, while it has been known for some time that the most stringent sample requirement arises for
set to the uniform distribution, a recent result of provides a formal reduction of arbitrary to the uniform distribution case. It is therefore enough to restrict to being the uniform distribution; identity testing for the uniform reference distribution is termed the -uniformity testing problem. The sample complexity of -uniformity testing was shown to be in .
Moving to our distributed setting, the estimator and the test must now be implemented by central referee using , where denotes the output of the channel (chosen by player from ) when the input is . The message constitutes communication from player to . Formally, we restrict to simultaneous message passing (SMP) protocols for communication, wherein the messages from all players are transmitted simultaneously to the central server, and no other communication is allowed. This restriction is motivated by applications in distributed inference where the players are users or nodes connected to a central server and there is no direct link for communication between these users. Although one could consider natural extensions to a more general adaptive communication setting, we restrict ourselves to the practically relevant SMP setting in this work.
Note that the SMP setting forbids communication between the players, but does allow them to a priori agree on a strategy to select different mappings from . In this context, the role of shared randomness available to the players is important, and motivates us to distinguish the settings of private-coin and public-coin protocols. In fact, as pointed earlier, a central theme of this work is to demonstrate the role of shared randomness available as public-coins in enabling distributed inference. We show that it is indeed a resource that can greatly reduce the sample complexity of distributed inference.222The distinction between public-coin and private-coin protocols is not so pronounced when multiple rounds of interaction between players are allowed. For instance, the first player may share the value of its private coins in the first round of communication, providing shared randomness. Thus, our results also imply a strict improvement in sample complexity of distributed inference by allowing multiple rounds of interaction.
Formally, the private- and public-coin SMP protocols are described as follows.
denote independent random variables, which are independent jointly of.333 In this work, we are not concerned with the amount of private or public randomness used. Thus, we can assume
’s to be discrete random variables, distributed uniformly over a domain of sufficiently large cardinality.In a private-coin SMP protocol, player is given access to and the channel is chosen as a function of . The central referee is given access to the random variables as well and its estimator and test can depend on .
Let be a random variable independent of . In a public-coin SMP protocol, all players are given access to , and they select their respective channels as a function of . The central referee is given access to the random variables as well and its estimator and test can depend on .
Hence, in a private-coin SMP protocol, the communication from player is a (randomized) function of . Note that since both and are generated from a product distribution, so is . In contrast, in a public-coin SMP protocol, the communication from player is a (randomized) function of and the ’s are not independent. They are, however, independent conditioned on the shared randomness .
Throughout we assume that some randomness is available to generate the output of the channel given its input . This randomness is assumed to be private as well, and, in fact, is assumed to be available only to each player – not even to . This distinction is important in the context of privacy where the information available to is seen as “leaked” and private randomness available only to the players is critical for enabling LDP channels.444Perhaps a more apt name for private-coin protocols in our setting would be pairwise-coin protocols; however, we have avoided this additional terminology for ease of presentation. Note that this assumption stands even for public-coin SMP protocols, implying the conditional independence of ’s given mentioned above.
The sample complexities of -distribution learning and -uniformity testing using in the distributed setting can now be defined analogously to the centralized setting by replacing with and , respectively, for public-coin and private-coin protocols. Note that since we are restricting to one sample per player, the sample complexity of these problems corresponds to the minimum number of players required to solve them. Our main objective in this line of work is the following:
Characterize the sample complexity for inference tasks using for public- and private-coin protocols.
I-B Summary of our results and contributions
We are initiating a systematic study of the distributed inference problem described in the previous section. In this paper, the first in our series, we shall focus on lower bounds. As is well-known from data-processing inequalities of information theory, the output distributions and for channel are “closer” than the corresponding input distributions and . At a high level, we derive lower bounds for distributed inference by providing a quantitative characterization of this reduction in distance for the chi-square distance, which we term chi-square contraction.
More technically, we consider probability distributions obtained by perturbing a nominal distribution. These perturbations are chosen so that in order to accomplish the given inference task, an algorithm must roughly distinguish the perturbed elements. In particular, we relate the difficulty of inference problems to the average chi-square distance of the perturbed distributions to the nominal distribution and the chi-square distance of the average perturbed distribution to the nominal distribution. For our distributed inference setting, we need to bound these two quantities for distributions induced at the outputs of the chosen channels from.
We provide bounds for these two quantities for channel output distributions in terms of two new measures of average distance in a neighborhood: the chi-square fluctuation for the average distance and the decoupled chi-square fluctuation for the distance to the average. The former notion has appeared earlier in the literature, albeit in different forms, and recovers known bounds for distributed distribution learning problems. The second quantity, the decoupled chi-square fluctuation, is the main technical tool introduced in this work, and leads to new lower bounds for distributed identity testing.
Observe that the general approach sketched above can be applied to any perturbations. We obtain lower bounds for public-coin protocols by a minmax evaluation of these bounds where the minimum is over perturbations and the maximum is over the choice of channels from . In contrast, we show that the performance of private-coin protocols is determined by a maxmin evaluation of these bounds. Remarkably, we establish that the maxmin evaluation is significantly smaller than minmax evaluation, leading to a quantitative separation in performance of private-coin and public-coin protocols for testing problems.
This separation has a heuristic appeal: On the one hand, in public-coin protocols players can use shared randomness to sample channels that best separate the current point in the alternative hypothesis class from the null. On the other hand, for a fixed private-coin protocol, one can identify a perturbation in a “direction” where the current choice of channels will face difficulty in distinguishing the perturbed distributions. Further, we remark that this separation only holds for testing problems. This, too, makes sense in light of the previous heuristic since learning problems require us to distinguish a neighborhood around the current hypothesis, without any preference for a particular “direction” of perturbation.
We develop these techniques systematically in Section III and Section IV. We begin by recasting the lower bounds for standard, centralized setting in our chi-square fluctuation language in Section III before extending these notions to the distributed setting in Section IV. Finally, we evaluate our general lower bounds for distribution learning and identity testing problems.
The resulting lower bounds involve matrices given by
Our bounds rely on the Frobenius norm and the nuclear norm of the matrix ; see Section II for definitions. These norms roughly measure how informative a particular channel is for distributed inference, and our bounds involve the maximum of these norms over in .
|Distribution Learning||Identity Testing|
|General channel family|
We summarize in Table I our sample complexity lower bounds for the -distribution learning and -identity testing problems using for public- and private-coin protocols. The form here is only indicative; formal statements for results for general channels are available in Sections IV-B, IV-B and IV-B in Section IV and implications for specific are given in Section V, with results on communication-limited setting in Theorems V.3, V.2 and V.1 and LDP setting in Theorems V.6, V.5 and V.4. The terms in each cell denotes the lower bound obtained by our approach. The first row contains our lower bounds for a general family . For comparison, we recall in the second row the results for the standard, centralized setting, and highlight the change in sample complexity lower bound as a multiplicative factor. Note that we have the same factor increase in the sample complexity for -distribution learning for both private- and public-coin protocols. We obtain the same factor increase for identity testing using private-coin protocols. On the other hand, we show that the factor increase for identity testing using public-coin protocols is , which in general can be much smaller.
As a corollary of these general bounds, we obtain and lower bounds for distribution learning using (the communication-limited setting) and (the LDP setting), respectively. These bounds have been previously obtained in other works as well and are known to be tight.
For identity testing, we obtain and lower bounds using and , respectively, for public-coin protocols. In the subsequent papers in this series ([3, 1]), we will present public-coin protocols to match these bounds, thereby establishing the optimality of these lower bounds.
Finally, for identity testing using private-coin protocols, we obtain and lower bounds using and , respectively, which too will be seen to be optimal among private-coin protocols in subsequent papers.
I-C Prior work
The statistical tasks of distribution learning and identity testing considered in this work have a rich history. The former requires no special techniques other than those used in parametric estimation problems with finite-dimensional parameter spaces, which are standard textbook material. The identity testing problem is the same as the classic goodness-of-fit problem. The latter goes beyond the discrete setting considered here, but often starts with a quantization to a uniform reference distribution (see [33, 36]). The focus in this line of research has always been on the relation of the performance to the support size (cf. ), with particular interest on the large-support and small-sample case where the usual normal approximations of statistics do not apply (cf. [37, 8]). Closer to our setting, Paninski  (see, also, ) established the sample complexity of uniformity testing, showing that it is sublinear in and equal to . As mentioned earlier, in this work we are following this sample complexity framework that has received attention in recent years. We refer the reader to surveys [17, 40, 14, 7] for a comprehensive review of recent results on discrete distribution learning and testing.
Distributed inference problems, too, have been studied extensively, although for the asymptotic, large-sample case and for simpler hypothesis classes. There are several threads here. Starting from Tsitsiklis , decentralized detection has received attention in the control and signal processing literature, with main focus on information structure, likelihood ratio tests and combining local decisions for global inference. In a parallel thread, distributed statistical inference under communication constraints was initially studied in the information theory community [5, 28, 29], with the objective to characterize the asymptotic error exponents as a function of the communication rate. Recent results in this area have focused on more complicated communication models [50, 49] and, more recently, on the minimum communication requirements for large sample sizes [41, 6].
Our focus is different from that of the works above. In our setting, independent samples are not available at one place, but instead information constraints are placed on individual samples. This is along the line of recent work on distributed mean estimation under communication constraints [54, 26, 42, 13, 51], although some of these works consider more general communication models than what we allow. The distribution learning problem under communication constraints has been studied in . However, in that paper the authors consider a blackboard model of communication and strives to minimize the total number of bits communicated, without placing any restriction on the number of bits per sample. A variant of the distribution testing problem is considered in  where players observe multiple samples and communicate their local test results to the central referee who is required to use simple aggregation rules such as AND. Interestingly, such setups have received a lot of attention in sensor network literature where fusion center combines local decisions using simple rules such as majority; see  for an early review.
Closest to our work and independent of it is , which studies the -distribution learning problem using bits of communication per sample. It was shown that the sample complexity for this problem is . This paper in turn uses a general lower bound from [31, 32], which yields lower bounds for distributed parametric estimation under suitable smoothness conditions. For this special case, our general approach reduces to a similar procedure as , which was obtained independently of our work.
Distribution learning under LDP constraints has been studied in [19, 34, 52, 4, 48], all providing sample-optimal schemes with different merits. Our lower bound when specialized for this setting coincides with the one derived in .
In spite of this large body of literature closely related to our work,
there are two distinguishing features of our approach. First, the methods
for deriving lower bounds under local information constraints in all
these works, while leading to tight bounds for distribution learning,
does not extend to identity testing. In fact, our decoupled
chi-square fluctuation bound fills this gap in the literature.
We remark that distributed uniformity testing under LDP constraints
has been studied recently in , but the lower bounds
derived there are significantly weaker than what we obtain. Second, our approach allows
us to prove a separation between the performances of public-coin and private-coin
protocols. This qualitative lesson – namely that shared public
randomness reduces the sample complexity – is in contrast to the
prescription of  which showed that shared
randomness does not help for simple hypothesis testing
problems.555 Identity testing is a composite hypothesis testing
problem with null hypothesis
Identity testing is a composite hypothesis testing problem with null hypothesisand alternative comprising all distributions that are -far from in total variation distance.
We observe that the unifying treatment based on chi-square distance is reminiscent of the lower bounds for learning under statistical queries (SQ) derived in [23, 22, 44]. On the one hand, the connection between these two problems can be expected based on the relation between LDP and SQ learning established in . On the other hand, this line of work only characterizes sample complexity up to polynomial factors. In particular, it does not lead to lower bounds we obtain using our decoupled chi-square fluctuation bounds.
We close with a pointer to an interesting connection to the capacity of an arbitrary varying channel (AVC). At a high level, our minmax lower bound considers the worst perturbation for the best channel. This is semantically dual to the expression for capacity of an AVC with shared randomness, where the capacity is determined by the maxmin mutual information, with maximum over input distributions and minimum over channels (cf. ).
We specify our notation in Section II and recall some basic inequalities needed for our analysis. This is followed by a review of the existing lower bounds for sample complexity of distribution learning and identity testing in Section III. In doing so, we introduce the notions of chi-square fluctuations which will be central to our work, and cast existing lower bounds under our general formulation. In Section IV, we generalize these notions to capture the information-constrained setting. Further, we apply our general approach to distribution learning and identity testing in the information-constrained setting. Then, in Section V, we instantiate these results to the settings of communication-limited and LDP inference and obtain our order-optimal bounds for testing and learning under these constraints. We conclude with pointers to schemes matching our lower bounds which will be reported in the subsequent papers in this series.
Ii Notation and preliminaries
Throughout this paper, we denote by the logarithm to the base and by the natural logarithm. We use standard asymptotic notation , , and for complexity orders.
Denote by for the set of integers . Given a fixed (and known) discrete domain of cardinality , we write for the set of probability distributions over , i.e.,
For a discrete set , we denote by the uniform distribution on and will omit the subscript when the domain is clear from context.
The total variation distance between two probability distributions is defined as
namely, is equal to half of the distance of and . In addition to total variation distance, we will extensively rely on the chi-square distance and Kullback–Leibler (KL) divergence between distributions , defined by
Using concavity of logarithms and Pinsker’s inequality, we can relate these two quantities to total variation distance as follows:
In our results, we will rely on the following norms for matrices. Given a real-valued matrix
with singular values, the Frobenius norm (or Schatten 2-norm) of is given by
Similarly, the nuclear norm (also known as trace or Schatten 1-norm) of is defined as
where is the (principal) square root of the positive semi-definite matrix . For any , the Frobenius and nuclear norms satisfy the following inequality
which can be seen to follow, for instance, from an / inequality and Cauchy–Schwarz inequality. Finally, the spectral radius of complex square matrix
with eigenvalues, is defined as .
Iii Perturbed families and chi-square fluctuations
To build basic heuristics, we first revisit the derivation of lower bounds for sample complexity of -distribution learning and -identity testing. As mentioned previously, it suffices to derive lower bound for -uniformity testing. For brevity, we will sometimes refer to distribution learning as learning and identity testing as testing. We present both proofs in a unifying framework which, in addition to its generality, will extend to our communication-constrained setting.666Although we restrict ourselves to the discrete setting here, the framework extends in a straightforward manner to more general parametric families.
Lower bounds for both learning and testing can be derived from a local view of the geometry of product distributions around the uniform distribution. Denote by the -fold product distribution with each marginal given by , the uniform distribution on . A typical lower bound proof entails finding an appropriate family of distributions close to for which it is information-theoretically difficult to solve the underlying problem. We call such a family a perturbed family and define it next.
For and a given -ary distribution , an -perturbed family around is a finite collection of distributions satisfying .
When is clear from context, we simply use the phrase perturbed family around .
As we shall see below, the bottleneck for learning distributions, which is a parametric estimation problem, arises from the difficulty in solving a multiple hypothesis testing problem with hypotheses given by the elements of a perturbed family around . Using Fano’s inequality, we can show that this difficulty is captured by the average KL divergence between and the elements of the perturbed family. In fact, for a unified treatment, we shall simply bound KL divergences by chi-square distances. This motivates the following definition.
Given a -ary distribution and a perturbed family around , the chi-square fluctuation of is given by
The aforementioned average divergence is bounded above by the chi-square fluctuation of , which will be used to obtain a lower bound for sample complexity of learning in the next section.
On the other hand, the bottleneck for testing, which is a composite hypothesis testing problem, arises from the difficulty in solving the binary hypothesis testing problem with as one hypothesis and a uniform mixture of the -fold product of elements of the perturbed family as the other. This difficulty is captured by the total variation distance between these two distributions on , for which a simple upper bound is . However, this bound turns out to be far from optimal.
Instead, an alternative bound derived using a recipe from Pollard  was shown to be tight in Paninski . To understand this bound, we parameterize the elements of the perturbed family as , for . Denoting by the normalized perturbation with entries given by
we can re-express as
is the second moment of the random variable(for drawn from ) and the outer expectation is over which is uniformly distributed over (independently of ). Using a technique of Pollard ( ), we can essentially replace in the previously mentioned upper bound by a quantity we term the decoupled chi-square fluctuation of , defined next.
Given a -ary distribution and a perturbed family around , the -fold decoupled chi-square fluctuation of is given by
where is the correlation inner product for drawn from and the outer expectation is over distributed uniformly over and an independent copy of .
While the quantities and are new, they are implicit in previous work. The abstraction here allows us to have a clear geometric view and lends itself to the more general local information-constrained setting. For completeness, we review the proofs of existing lower bounds using our chi-square fluctuations terminology.
In the sections below, we will present the proofs of lower bounds for sample complexity of learning and testing using a specific perturbed family and bring out the role of and in these bounds. In particular, both bounds will be derived using the -perturbed family around due to Paninski , consisting of distributions given by
The normalized perturbations for this perturbed family are given by
Note that this perturbed family is closely related to the standard one used in statistics where is proportional to for each ; the variant above ensures additionally that the probabilities of pairs of elements are preserved, whereby the perturbed family consists of elements of the -dimensional probability simplex.
Iii-a Chi-square fluctuation and the learning lower bound
For learning, we consider the multiple hypotheses testing problem where the hypotheses are , , given in Eq. 3. Specifically, denote by the random variable distributed uniformly on and by the random variable with distribution given . We can relate the accuracy of a probability estimate to the probability of error for the multiple hypothesis testing problem with hypotheses given by using the standard Fano’s method (cf. ). In particular, we can use a probability estimate to solve the hypothesis testing problem by returning as a that minimizes . The difficulty here is that the total variation distance may not be , and therefore, an -estimator may not return the correct hypothesis.
One way of circumventing this difficulty is to restrict to a perturbed family where pairwise-distances are . Note that for the perturbed family in (3)
where is the Hamming distance. This simple observation allows us to convert the problem of constructing a “packing” in total variation distance to that of constructing a packing in Hamming space. Indeed, a standard Gilbert–Varshamov construction of packing in Hamming space yields a subset with such that for every in . Using Fano’s inequality to bound the probability of error for this new perturbed family, we can relate the sample complexity of learning to . However, when later extending our bounds to the information-constrained setting, this construction would create difficulties in bounding for public-coin protocols. We avoid this complication by relying instead on a slightly modified form of the classic Fano’s argument from ; this form of Fano’s argument was used in  as well to obtain a lower bound for the sample complexity of learning under communication constraints.
Specifically, in view of (4), it is easy to see that for an estimate such that for all , we must have
where denotes the cardinality of Hamming ball of radius . Noting that
and combining the bounds above, we obtain
Therefore, to obtain a lower bound for sample complexity it suffices to bound from above. It is in this part that we bring in the role of chi-square fluctuations.
Indeed, we have
In fact, the argument above is valid for any perturbation with desired pairwise minimum total variation distance, namely any perturbed family satisfying an appropriate replacement for Eq. 5. In particular, it suffices to impose the following condition:
The foregoing arguments lead to the next result.
For and a -ary distribution , let be an -perturbed family around satisfying Eq. 8. Then, the sample complexity of -distribution testing must be at least
For Paninski’s perturbed family in Eq. 3, , , and an easy calculation yields
which with the previous result recovers the lower bound for sample complexity of learning.
Iii-B Decoupled chi-square fluctuation and the testing lower bound
As is the case with distribution learning, the pairwise hypothesis testing problems emerging from the perturbed family do not yield the desired dependence of sample complexity on . The bottleneck is obtained by realizing that the actual problem we end up solving is a composite binary hypothesis testing where the null hypothesis is given by and the alternative can be any of the , . In particular, any test for uniformity using samples will also constitute a test for versus for every random variable . Thus, another aspect of the geometry around that enters our consideration is the distance between and .
A simple approach to bound this quantity is to observe that by the convexity of the norm, we have
where the last identity is by (9). Thus, this upper bound of the distance between and in terms of the chi-square fluctuation only yields a sample complexity lower bound of , much lower than the bound that we strive for.
Instead, we bound this distance in terms of the decoupled chi-square fluctuation using a recipe from . To that end, we rely on the following result, which is a slight extension of a similar result in  (see, also, ). This crucial extension will allow us to handle local information constraints later; we include a proof for completeness.
Consider a random variable such that for each the distribution is defined as . Further, let be a fixed product distribution. Then,
where is an independent copy of , and with ,
where the expectation is over distributed according to .
Using the definition of chi-square distance, we have
where the outer expectation is for using the distribution . For brevity, denote by the random variable . The product in the expression above can be expanded as
whereby we get
Observe now that for every . Furthermore, is an independent copy of and and are independent for . Therefore, the expectation on the right-side above equals
which completes the proof. ∎
Proceeding as in , we obtain the following result which will be seen to yield the desired lower bound for sample complexity.
For and a -ary distribution , let be an -perturbed family around . Then, the sample complexity for -identity testing with reference distribution must satisfy
for some constant depending only on the probability of error.
The proof uses Le Cam’s two-point method. We note first that
and bound the right-side further using Section III-B with replaced by , , and to get
since . Now, to complete the proof, consider an -test . By definition, we have and for every , whereby
The left-hand-side above coincides with the Bayes error for test for the simple binary hypothesis testing problem of versus , which must be at least
Thus, we obtain , which together with (11) completes the proof. ∎
In particular, going back to Paninski’s perturbed family of Eq. 3, observe that
where are independent and distributed uniformly over , we can bound the decoupled chi-square fluctuation using Hoeffding’s Lemma ( ) as
Thus, Section III-B implies that samples are needed for testing (in particular, for uniformity testing).
We summarize the geometry captured by the bounds derived in this section in Fig. 2. This geometry is a local view in the neighborhood of the uniform distribution obtained using the perturbed family in Eq. 3. Each is at a total variation distance from . The mixture distribution we use is obtained by uniformly choosing the perturbation over .
The chi-square fluctuation of is whereby the average total variation distance to is . On the other hand, the decoupled chi-square fluctuation of is and thus the total variation distance of the mixture of to is . Note that for , the total variation distance between the mixture and is much smaller than the average total variation distance.
Iv Results: The chi-squared contraction bounds
We now extend our notions of chi-square fluctuation and decoupled chi-square fluctuation to the information-constrained setting. We follow the same notation as the previous section. Recall that in the information-constrained setting each player sends information about its sample by choosing a channel from a family to communicate to the central referee . The perturbed family will now induce a distribution on the outputs of the chosen channels . The difficulty of learning and testing problems will thus be determined by chi-square fluctuations for this induced perturbed family, extending the results of the previous section to the information-constrained setting. The difficulty of inference gets amplified by information constraints since the induced distributions are closer than the original ones and the chi-square fluctuation decreases.
As one of our main results in this section, we provide a bound for chi-square fluctuations of the induced perturbed family corresponding to Paninski’s perturbed family of Eq. 3, for a given . Underlying these bounds is a precise characterization of the contraction in chi-square fluctuation owing to information constraints. One can view this as a bound for the minmax chi-square fluctuation for an induced perturbed family, where the minimum is taken over perturbed families and the maximum over all channels in . We will see that for public-coin protocols, the bottleneck is indeed captured by this minmax chi-square fluctuation.
On the other hand, for private-coin protocols the bottleneck can be tightened further by designing a perturbation specifically for each choice of channels from . In other terms, we can in this case use a bound for maxmin chi-square fluctuation. Another main result of this section, perhaps our most striking one, is a tight bound for this maxmin chi-square fluctuation for the aforementioned induced perturbed family. This bound turns out to be more restrictive than the minmax chi-square fluctuation bound and leads to the separation for private- and public-coin protocols for the cases and considered in the next section.
We begin by noting that Section III-A and Section III-B extend to the information-constrained setting. Our extension involves the notions of an induced perturbed family and its chi-square fluctuations, defined next. Throughout we assume that the family of channels consists of channels where the input alphabet is and the output alphabet is finite, and the perturbed family can be parameterized as .
For a perturbed family and channels , the induced perturbed family comprises distributions on given by
To extend the notion of chi-square fluctuations to induced perturbed families, we need to capture the corresponding notion of normalized perturbation. Let and , respectively, be the output distributions for a channel with input distributions and . Then, for , we have