On Fundamental Limits of Robust Learning

1 Introduction

The last decade has witnessed a tremendous growth in the amount of data involved in machine learning tasks. In many cases, data volume has outgrown the capacity of memory of a single machine and it is increasingly common that learning tasks are performed in a

distributed fashion on many machines [2, 5, 25] or an online fashion on a single machine [15].

This gives rise to the following fundamental, yet rarely investigated, complexity questions for distributed and online learning algorithms. Considering a concrete example (our analysis is beyond this case): suppose one has the positive and negative examples stored on two separate machines, then how much communication is necessary to learn a good hypothesis to specified error rate? and how about for the case where more machines are involved? In addition, if the positive and negative examples arrives in a single machine with arbitrary order, how much memory must be used to store the necessary information to learn a good hypothesis? if multiple passes of the data are allowed, how does the memory cost scale then?

In this work, we investigate the above communication and space complexities of distributed and online probably approximately correct (PAC) learning

[19], in the presence of malicious errors (outliers) in the training examples. PAC learning with malicious errors was first proposed in [20] and then generalized in [10]. In this learning setting, there is a fixed probability

λ

of an error occurring independently on each request for a training example. This error may be of an arbitrary nature – in particular, it may be chosen by an adversary with unbounded computational resources, and exact knowledge of the target representation, the target distributions, and the current internal state of the learning algorithm. Though various robust learning algorithms have been proposed in literature to tackle the outliers in the data, the fundamental limits of these algorithms as well as of the problem itself, in terms of their communication and space complexities, are rarely investigated.

1.1 Our Contributions

We analyze and provide general lower bounds on the amount of communication and memory cost needed to learn a good hypothesis, from either distributed or streaming data, in the presence of malicious errors and outliers.

Our results reveal the connection between the communication complexity and the VC-dimension $d$ of the hypothesis to learn, as well as the outlier probability $λ$ . We first derive a lower bound of $Ω (\frac{1 - ϵ}{1 - λ} d)$ with $ϵ$ the specified error rate, for the simplest distributed learning protocol – only two machines are involved and only a single message is allowed to communicate. We then extend the lower bound analysis to more general distributed learning protocols, allowing $k$ -machine and $t$ -round communications, and obtain a general lower bound of $Ω (\frac{1 - ϵ}{1 - λ} \frac{d k}{t^{2}})$ . All these lower bounds present an additional factor proportional to $λ$ , which explains the extra communication expense brought by the outliers in the learning process.

We also analyze the $r$ -pass online robust PAC learning algorithms. We demonstrate their space complexity lower bounds can be conveniently deduced from the communication complexity results for distributed learning algorithms and hence obtain a space complexity lower bound of $Ω (\frac{1 - ϵ}{1 - λ} \frac{d}{r})$ .

Finally, we provide a communication efficient distributed robust PAC learning algorithm along with its performance guarantee. Experimental studies on synthetic and real data demonstrate the proposed algorithm consumes significant less communication to achieve comparable accuracy with a naive distributed learning algorithm, which aggregates all the examples to a single machine.

1.2 Related Works

Since its introduction in [23], communication complexity [13] has proven to be a powerful technique for establishing lower bounds in a variety of settings, including distributed [25, 7, 9] and streaming data models [15].

Several recent works on communication complexity for distributed PAC learning include [2, 5, 6]

and there are also some works about distributed statistical estimation

[25, 7]. In particular, Duchi et al. [8] demonstrated an analysis tool based on the information Fano inequalities. Garg et al. [9] investigated how the communication cost scales in the parameter dimensionality for distributed statistical estimation. Kremer et al. [12] provide a connection between communication complexity and VC-dimension. However, those works generally focus on the case no outliers or malicious errors are presented in the learning process.

On the other hand, studies on streaming algorithms focus on the scenario where input arrives very rapidly and there is limited memory to store the input [15]

, and investigate the necessary space cost under the most adversarial data orders for various data mining problems, include quantile query

[25], frequent elements query [14, 1], regression [22] and model monitoring [11]. Those algorithms do not consider the case with malicious errors either.

2 Problem Setup and Preliminaries

Robust PAC Learning

In a classical distribution-free learning setting [19], the probably approximately correct (PAC) learning model typically assumes the oracles Pos and Neg always faithfully return positive and negative examples from a sample domain $X$ , drawn according to target distributions respectively. However, in many real environments, it is possible that an erroneous or even adversarial example is given to the learning algorithm. We consider in this work such a generalized PAC learning problem, termed as robust PAC learning, where malicious outliers are possibly presented in training examples [10].

In particular, we consider following erroneous oracles in robust PAC learning. For a error probability $0 \leq λ < 1 / 2$ , we have access to following two noisy oracles – $\sc Posλ$ and $\sc Negλ$ – which behave as follows: when oracle $\sc Posλ$ is called, with probability $1 - λ$ , it returns positive examples as in the error free model. But with probability $λ$ , an example about which absolutely no assumptions can be made is returned. In particular, this example may be dynamically and maliciously chosen by an adversary. The oracle $\sc Negλ$ behaves similarly.

Formally, the task of a robust PAC learning algorithm is to find a hypothesis $h \in H$ , with access only to the above noisy oracles $\sc Posλ$ and $\sc Negλ$ and with predefined parameters $ϵ, δ$ and $λ$ , such that: for any input values $0 < ϵ, δ < 1$ and $λ < 1 / 2$ , the output hypothesis $h$ has bounded errors $e (h) < ϵ$ , with probability at least $1 - δ$ . Throughout the paper, we assume a constant failure probability $δ$ for the simplicity of analysis. Note that we always have $λ < ϵ$ as proved in [10].

Distributed and Online PAC Learning

As mentioned above, in this work we consider a realistic case where the entire data set is too large to store on a single machine. Thus following two learning frameworks become the choices to deal with the issues brought by the large scale of the dataset, which include:

Distributed robust learning. Data are distributed on $k$ machines. The problem of interest is how to communicate among the $k$ machines, and especially how much communication is necessary to produce a low-error hypothesis. Namely, we are concerned about the communication complexity of distributed robust learning.
Online robust learning. Data are revealed sequentially (in multiple passes) to a single machine. The problem we want to answer is how much memory cost is necessary to maintain the intermediate information, in order to obtain the low-error hypothesis, i.e., the space complexity of online robust learning.

We regard communication (the number of bits communicated) and memory as a limited resource, and therefore we not only want to optimize the hypothesis error $e (h)$ but also the resource cost of the whole learning procedure. We aim at providing a fundamental understanding on the necessary communication and space cost in order to learn a hypothesis $h$ with a specified error $ϵ$ , in presence of a constant outlier fraction $λ$ .

Communication Protocols and Complexity

In the above distributed robust learning setting, several different communication protocols among the machines can be employed. For instance, in a $1$ -round $2$ -machine public-randomness protocol between two machines $M_{1}$ and $M_{2}$ , denoted as $P^{M_{1} \to M_{2}}$ , $M_{1}$ is only allowed to send a single message to $M_{2}$ which must then be able to output the hypothesis $h$ . Similarly, when multiple-round communications are allowed between $M_{1}$ and $M_{2}$ , i.e., multiple messages can be communicated among the two machines, we are using a $t$ -round $2$ -machine protocol, which is denoted as $P^{M_{1} \leftrightarrow M_{2}}$ . Extending the above protocols to the cases involving $k$ machines gives the $1$ -round $k$ -machine $P^{M_{1} \to \dots \to M_{k}}$ and the $t$ -round $k$ -machine $P^{M_{1} \leftrightarrow \dots \leftrightarrow M_{k}}$ protocols respectively.

The communication complexity of a protocol is the minimal number of bits needed to exchange between machines for learning the hypothesis $h$ . Assume a specified error rate $0 < ϵ < 1$ of the learned hypothesis $h$ , we specifically consider following communication complexities $R_{ϵ}^{M_{1} \to M_{2}} (h)$ – the communication complexity of randomized $1$ -round $2$ -machine protocol, and $R_{ϵ}^{M_{1} \leftrightarrow \dots \leftrightarrow M_{k}} (h)$ – the complexity of the $P^{M_{1} \leftrightarrow \dots \leftrightarrow M_{k}}$ protocol.

We start the communication complexity analysis from the case when there are two machines, and the $1$ -round protocol $P^{M_{1} \to M_{2}}$ is employed. After demonstrating the lower bound on the complexity $R_{ϵ}^{M_{1} \to M_{2}} (h)$ , we proceed to analyze the complexity $R_{ϵ}^{M_{1} \leftrightarrow \dots \leftrightarrow M_{k}} (h)$ . The complexity lower bounds for other protocols, such as $2$ -machine $t$ -round and $k$ -machine $1$ -round protocols, are provided in the appendix.

3 Main Results on Distributed Robust PAC Learning

In this section we analyze the lower bounds on the communication cost for distributed robust PAC learning. We then extend the results to an online robust PAC learning setting in Section 4, through a one-direction equivalent lemma between distributed and online learning, in terms of communication complexity and space complexity.

3.1 Communication Models

For distributed learning, popular communication models include: (1) Blackboard model: any message sent by a machine is written on a blackboard visible to all machines; (2) Message passing model: a machine $p_{i}$ sending a message specifies another machine $p_{j}$ that will receive this message; (3) Coordinator model: there is an additional machine called the coordinator, who receives no input. Machines can only communicate with the coordinator, and not with each other directly.

We will focus on the message-passing model and the coordinator model, considering the blackboard model may introduce extra communication overload and is not so practical. Note that the coordinator model is almost equivalent to the message-passing model, up to a multiplicative factor of $2$ , since instead of machine $i$ sending the message to machine $j$ , machine $i$ can transmit message to the coordinator, and the coordinator forwards it to machine $j$ . Therefore, in the following sections, we mainly provide the complexity results for the message passing model, which can then be used to deduce the complexities of the coordinate model easily.

3.2 Lower Bound for $1$ -Round $2$ -Party Communication

From now on, we derive the main results of this work – the communication complexity lower bounds for distributed robust PAC learning: how much information communication is needed to learn a hypothesis $h$ with specified error rate $ϵ$ . Since we are mainly concerned about how communication complexity scales with $ϵ$ , outlier fraction $λ$ and VC-dimension of the hypothesis $d$ , in developing the results, we assume the failure probability $δ$ is fixed as a constant, such as $0.1$ .

We start with one simplest distributed learning setting: only two machines are involved and only a single message communication between them is allowed. We first construct a connection between the communication complexity and VC-dimension [21] of the hypothesis to learn, for this $1$ -round $2$ -machine communication protocol.

The result is basically obtained by following Yao’s principle [24] (Theorem 7 in the appendix) to investigate the communication complexity for a “difficult” distribution of the examples over the two machines, which then provides a lower communication complexity bound for general example distributions over the machines. The lower bound on the communication complexity for the specific “difficult” sample distribution is constructed by limiting the information capacity (depending on the VC-dimension $d$ ) of the communication channel between the two machines, and we show that with such limited capacity, the inevitable encoding error will result in a lowered bounded error rate $ϵ$ of the learned hypothesis $h$ . Based on the above technique, we obtain the lower bound on the complexity communication of the protocol $P^{M_{1} \to M_{2}}$ . More details on the tools and proofs for the following claim are provided in the appendix.

Theorem 1 (Communication Complexity Lower Bound for $1$ -Round $2$ -Machine Protocol).

For learning a hypothesis $h$ with a constant error $ϵ$ , where the oracles have an outlier rate of $0 \leq λ < ϵ$ , we have the following communication complexity lower bound for $1$ -round and $2$ -machine communication protocol,

R_{ϵ}^{M_{1} \to M_{2}} (h) = Ω (\frac{1 - ϵ}{1 - λ} d),

for a constant success probability. Here $d$ is the VC-dimension of the hypothesis $h$ .

From the above lower bound, one can observe that the necessary communicated bits between two machines is monotonically increasing with the outlier rate $λ$ when the error rate $ϵ$ is fixed, while monotonically decreasing with the specified error rate $ϵ$ . When there are no outliers in the samples, i.e., $λ = 0$ , the lower bound for learning a hypothesis with $e (h) < ϵ$ reduces to $Ω ((1 - ϵ) d)$ , which matches the result provided in [2] for faithful oracles.

Applying Theorem 1 directly provides a communication lower bound on learning the linear half spaces in the real space $R^{p}$ , whose VC-dimension is known to be $(p + 1)$ .

Corollary 1.

For the hypothesis $ℏ$ of linear half spaces in $R^{p}$ , applying the above theorem gives the one-round communication complexity between two machines to learn a half space is lower bounded as

R_{ϵ}^{M_{1} \to M_{2}} (ℏ) = Ω (\frac{1 - ϵ}{1 - λ} (p + 1)),

for a constant success probability.

In Section 3.5, we compare the above lower bound for half space learning with the bound given in [5], and we propose a communication-efficient distributed half-space learning algorithm, whose communication complexity is close to the above lower bound.

3.3 Lower Bound for $t$ -Round $2$ -Machine Communication

We then consider a more complicated and practical communication protocol, which allows the two machines to communicate $t > 1$ messages in turn. Such a protocol is called a $t$ -Round $2$ -Machine protocol. Here we assume, by convention, that one machine sends the first message and the recipient machine of the last message announces the learned hypothesis $h$ . To lower bound the complexity of multiple round communications, we need to investigate the size of the message sent in each round and invoke a technique of round elimination. Now we elaborate on how we can obtain the desired complexity bound.

We first provide a more detailed description on the multiple round communication protocol, which explicitly specifies the size of message sent in each round.

Definition 1.

A $[t; ℓ_{1}, \dots, ℓ_{t}]^{M_{1}}$ (resp. $[t; ℓ_{1}, \dots, ℓ_{t}]^{M_{2}}$ ) communication protocol is the protocol where the machine $M_{1}$ (resp. $M_{2}$ ) starts the communication, the size of the $i$ th message is $ℓ_{i}$ bits, and the communication goes on for $t$ rounds.

The communication complexity for such a protocol is then proportional to $\sum_{i = 1}^{t} ℓ_{i}$ . We employ a round elimination technique [18] to deduce the communication complexity lower bound for a $t$ -round protocol from a $(t - 1)$ -round protocol. After applying the round elimination for $(t - 1)$ times, it is shown that we only need to bound the complexity of a $1$ -round protocol, which has been provided in Theorem 1.

The round elimination, which is formally described by the following lemma, is based on the intuition that the existence of a “good” $t$ round protocol with $M_{1}$ starting, implied that there exists a “good” $(t - 1)$ -round protocol where $M_{2}$ communicates first.

Lemma 1 (Round Elimination, adapted from [18]).

Assume $h$ is a hypothesis to learn. Suppose the communication between two machines has a $[t; ℓ_{1}, \dots, ℓ_{t}]^{M_{1}}$ protocol which produces a hypothesis with error less than $ϵ$ . Then there is a $[t - 1; ℓ_{1} + ℓ_{2}, ℓ_{3}, \dots, ℓ_{t}]^{M_{2}}$ protocol for learning $h$ with error less than $ϵ + \sqrt{ℓ_{1} / d}$ . Here $d$ is also the VC-dimension of $h$ .

Applying the above round elimination lemma, we can reduce any $t$ -round communication protocol to a $1$ -round communication protocol. Thus applying the communication complexity for $1$ -round protocol in Theorem 1 gives the following lower bound.

Theorem 2 (Communication Complexity Lower Bound for $t$ -Round $2$ -Machine Protocols).

Let $ϵ$ be the specified error rate of $h$ . Let $λ$ be the outlier rate in the oracles with $λ < ϵ$ . Consider a $t$ -round communication protocol to learn hypothesis $h$ . Then the communication complexity is lower bounded as

R_{ϵ}^{M_{1} \leftrightarrow M_{2}} = Ω (\frac{1 - ϵ}{1 - λ} \frac{d}{t^{2}}),

for a constant success probability. Here $d$ is the VC-dimension of $h$ .

3.4 Lower Bound for $t$ -Round $k$ -Machine Communication

We now proceed to obtain the communication complexity lower bound for a more general protocol $P^{M_{1} \to \dots \to M_{k}}$ . To extend the above lower bound for $2$ -machine communication to the $k$ -machine case, we use the symmetrization technique [17] to reduce $k$ -machine communication to $2$ -machine communication. The symmetrization constructs a $2$ -machine protocol $P^{M_{1} \to M_{2}}$ from a $k$ -machine protocol $P^{M_{1} \to \dots \to M_{k}}$ with communication cost $E [R_{ϵ}^{M_{1} \to M_{2}}] = 2 R_{ϵ}^{M_{1} \to \dots \to M_{k}} / k$ , where the expectation is taken on the randomness of the samples returned by oracles. Then we can derive the bound for $R_{ϵ}^{M_{1} \to \dots \to M_{k}}$ from $R_{ϵ}^{M_{1} \to M_{2}}$ . The result is formally stated in the following theorem, whose proofs are given in the appendix.

Theorem 3 (Communication Complexity Lower Bound for $1$ -Round $k$ -Machine Protocols).

Let $ϵ$ be the specified error rate of learned hypothesis $h$ . Let $λ$ be the outlier rate in the oracles with $λ < ϵ$ . For $1$ -round $k$ -machine protocol, its communication complexity is lower bounded as

R_{ϵ}^{M_{1} \to \dots \to M_{k}} = Ω (\frac{1 - 2 ϵ}{1 - λ} d k),

for a constant success probability. Here $d$ is the minimal VC-dimension of the hypothesis $h$ .

In the above theorem, we observe that there is a factor of $2 ϵ$ instead of $ϵ$ , which appears due to the employed the symmetrization technique and introduces a gap with the lower bound for $2$ -machine protocol. In the future, we will further narrow such a gap.

Then similar to obtaining the results in Theorem 2 and Theorem 3, applying the round elimination and machine elimination by symmetrization together gives a general lower bound on the communication complexity for the $t$ -round $k$ -machine protocol, as stated in the following theorem.

Theorem 4 (Communication Complexity Lower Bound for $t$ -Round $k$ -Machine Protocols).

Let $ϵ$ be the specified error rate of learned $h$ . Let $λ$ be the outlier rate in the oracles with $λ < ϵ$ . Then for a $t$ -round $k$ -machine protocol, its communication complexity is lower bounded as

R_{ϵ}^{M_{1} \leftrightarrow M_{2} \leftrightarrow \dots \leftrightarrow M_{k}} = Ω (\frac{1 - 2 ϵ}{1 - λ} \frac{d k}{t^{2}}),

for a constant success probability. Here $d$ is the minimal VC-dimension of the hypothesis $h$ .

3.5 A Communication-Efficient Protocol: Weighted Sampling

We present an efficient distributed PAC learning algorithm, focusing on the $2$ -machine case, where the machines are denoted as $A$ and $B$ . The proposed algorithm is mainly based on communicating learned hypotheses and difficult examples, similar to boosting: each hypothesis learned by $A$ is communicated to and tested by $B$ . Then the difficult examples of $B$ are randomly sampled by a probability, proportional to how many times the hypothesis fails on them, and communicated back to $A$ for updating hypothesis. This two-machine protocol algorithm is developed based on the algorithm proposed in [5].

Following lemma [5] says the random weighted sampling can always find a good representing subset of the entire set on machine $B$ .

Lemma 2.

Let $B$ have a weighted set of points $X_{B} \subset R^{p}$ with weight function $w : X_{B} \to R^{+}$ . For any constant $0 < c < 1$ , machine $B$ can send a set $R_{B}$ of size $O (p / c)$

such that any hypothesis that correctly classifies all points in

R_{B}

will mis-classify points in

X_{B}

with a total weight at most

c \sum_{x \in X_{B}} w (x)

. The set

R_{B}

can be constructed by a weighted random sample from

(X_{B}, w)

as in Alg. 2, which succeeds with constant probability.

Input:

X_{A}, X_{B} \subset R^{p}

, parameters:

0 < ϵ < 1

, outlier fraction

0 \leq λ < 0.5

Initialization:

R_{B} = \emptyset, w_{i}^{0} = 1, \forall x_{i} \in X_{B}, ρ = 0.75, c = 0.2 (1 - λ) / (1 - 2 λ)^{2}

for

t = 1, \dots, T = 5 {log}_{2} (1 / ϵ)

X_{A} = X_{A} \cup R_{B}

;

h_{A}^{t} := Learn (X_{A})

;

send

h_{A}^{t}

B

;

R_{B} :=

mwu

(X_{B}, h_{A}^{t}, ρ, c)

(see Alg.2);

send

R_{B}

A

;

end for

h_{A B} = M a j o r i t y (h_{A}^{1}, h_{A}^{2}, \dots, h_{A}^{T})

Output:

h_{A B}

(hypothesis with

ϵ

-error on

X_{A} \cup X_{B}

Algorithm 1 Weighted Sampling (Ws)

Input:

h_{A}^{t}, X_{B} \subset R^{p}

, parameters:

0 < ρ, c < 1

for all

x_{i} \in X_{B}

h_{A}^{t} (x_{i}) \neq y_{i}

then

w_{i}^{t + 1} = w_{i}^{t} (1 + ρ)

;

else

w_{i}^{t + 1} = w_{i}^{t}

;

end if

end for

randomly sample

R_{B}

from

X_{B}

according to

w^{t + 1}

, whose size

| R_{B} | = min {p / c log (p / c), p / c^{2}}

Output:

R_{B}

Algorithm 2 mwu

(X_{B}, h_{A}^{t}, ρ, c)

Based on Lemma 2, we can obtain the following communication complexity result for Algorithm 1

Theorem 5.

The two-machine two-way protocol Weighted Sampling in Alg. 1 for linear separator in $R^{p}$ mis-classifies at most $ϵ | D |$ points after $T = O (log (1 / ϵ))$ rounds and uses $O (\frac{1 - λ}{(1 - 2 λ)^{2}} p^{2} log (1 / ϵ))$ bits of communication.

Comparing the communication complexity of Ws in Theorem 5 with the upper bound in Corollary 3 (where the sample description length is $b = O (p)$ and $| H | = 2^{p}$ ) provides that Ws reduces the communication complexity by a factor of $O (1 / log (1 / ϵ) ϵ)$ . This demonstrates Ws is much more communication efficient than naively aggregating sufficiently many samples.

Consider that VC-dimension of a linear classifier in $R^{p}$ is $(p + 1)$ . Applying Theorem 2 gives a communication complexity lower bound of learning linear classifiers from two machines to be $Ω (\frac{1 - ϵ}{1 - λ} (p + 1))$ . Omitting constant factors gives a ratio between this theoretical complexity lower bound and the practical complexity of Ws in Theorem 5 as $O (p)$ . This communication efficiency gap, linearly depending on $p$ , is significant when $p$ is large. The factor $p$ comes from the algorithm Mwu sampling a subset $R_{B}$ whose size depends on $p$ . This gap also demonstrates there is room to enhance communication efficiency for Ws algorithm, if we can construct a data subset from $B$ whose size independent of $p$ . This $2$ -machine communication protocol for distributed classifier learning can be straightforwardly generalized to $k$ -machine communication, using the coordinator communication model as follows. Fixing an arbitrary machine (say machine $1$ ) as the coordinator for the other $(k - 1)$ machines. Then machine $1$ runs the $2$ -machine communication protocol from the perspective of $A$ and the other machines serve jointly as the second machine $B$ . Each other machine reports the total weight $w (X_{i})$ of their data to machine $1$ , who then reports back to each machine what fraction of the total weight $w (X_{i}) / w (X)$ they own. Then each machine sends the coordinator a random sample of size $| R_{B} | w (D_{i}) / w (D)$ (for $| R_{B} |$ see Alg. 2). Party $1$ learns a classifier on the union of its own sample and the joint samples from other machines, and sends the updated classifier back to all the machines.

4 Online Robust PAC Learning

We briefly discuss here about the space complexity for online robust PAC learning. We use our results on communication complexity from the previous sections to derive lower bounds on space complexity for robust PAC learning in online setting. The space complexity has been extensively investigated in the context of streaming methods [15]. However, specific space complexity lower bound for robust learning is still absent.

Following lemma shows the connection between distributed learning algorithm and online learning algorithm, in particular from the perspective of the connection between the communication complexity and the space complexity bound in an online data model.

Lemma 3.

Suppose that we can learn a hypothesis $h$ using an online algorithm that has $s$ bits of working storage and makes $r$ passes over the samples. Then there is a $k$ -machine distributed algorithm for learning $h$ that uses $k r s$ bits of communications.

The above lemma allows us to deduce the space lower bound for the online learning, given the communication complexity of distributed learning algorithms.

Theorem 6 (Space Complexity Lower Bound for $r$ -Pass Online Learning).

Let $ϵ$ be the specified error rate of learned $h$ . Let $λ$ be the outlier rate in the oracles with $λ < ϵ$ . For a $r$ -pass online PAC learning algorithm, its space complexity is lower bounded as $Ω (\frac{1 - ϵ}{1 - λ} \frac{d}{r})$ , where $d$ is the VC-dimension of hypothesis $h$ .

For an online PAC learning algorithm, in order to learn a hypothesis with an error at most $ϵ$ , a space at the order of $\frac{1 - ϵ}{1 - λ} d$ must be maintained. However, if we allow the data to be passed to the learning algorithm for $r$ passes, then the lower bound on the space cost can be reduced by a factor of $r$ .

5 Simulations

In this section, we present simulation studies on the distributed learning algorithm, Ws, in Algorithm 1, for finding linear classifier in $R^{p}$ for two-machine and $k$ -machine scenarios. We empirically compare it with a Naive approach, which sends all samples from $(k - 1)$ machines to a coordinator machine $1$ and then learns the classifier at the coordinator. For any dataset, this accuracy is the best possible.

For the two compared methods, a linear SVM is used as the underlying classifier. We report their training accuracies and communication costs. The cost of communicating one sample or one linear classifier in $R^{p}$ is assumed to be $(p + 1)$ ( $p$ for describing the coordinates and $1$ for sign or offset). For instance, the total communication cost of the Naive method is the number of samples sent by $(k - 1)$ machines multiplied by the description length of each sample $(p + 1)$ , which is equal to $\sum_{i = 2}^{k} (p + 1) | X_{i} |$ . We always set the communication cost of the naive method as $1$ and report the communication cost ratio of Ws to the Naive method.

Dataset	Accuracy ( $λ = 0.1$ )		Accuracy ( $λ = 0.2$ )		Relative CC: Ws / Naive
Dataset	Naive	Ws	Naive	Ws	$λ = 0.1$	$λ = 0.2$
Syn.1	$85.63$	$84.97$	$75.52$	$71.26$	$0.42$	$0.64$
Syn.2	$85.23$	$83.13$	$75.05$	$72.33$	$0.43$	$0.69$
Syn.3	$84.22$	$80.94$	$78.52$	$69.97$	$0.21$	$0.34$
Syn.4	$83.40$	$83.61$	$75.05$	$71.64$	$0.20$	$0.33$
Cancer	$84.07$	$83.03$	$75.76$	$73.32$	$0.32$	$0.66$
Mushroom	$85.26$	$83.87$	$77.39$	$74.23$	$0.19$	$0.42$

Table 1: Classification accuracy of the two compared methods with different malicious error rates, and communication cost (CC) of Ws relative to the Cost of Naive method.

We report results for two-machine and four-machine protocols on both synthetic and real-world datasets. Four datasets, two each for two-machine and four-machine cases, are generated synthetically from mixture of two Gaussians $N (μ_{i}, Σ_{i})$ with $μ_{1} = - μ_{2}$ and $Σ_{1}, Σ_{2}$ two randomly generated diagonal matrices. Each Gaussian is carefully seeded to make sure the generated data from two components are separated well. Algorithms access training examples via noisy oracles $\sc Posλ$ and $\sc Negλ$ with a malicious error rate of $λ = 0.1$ and $0.2$ . That is, among the training examples, $10 %$ or $20 %$ of them come from a noisy distribution $N (0, I_{p})$ , which is significantly different from the above two Gaussian components. In addition, we conduct empirical studies on two real-world datasets from UCI repository¹¹1http://archive.ics.uci.edu/ml/, including the Cancer and Mushroom data sets. Statistics on the used data sets are given in Table 2 and corresponding results, including accuracies and relative communication costs (regarding the cost of Naive as $1$ ), are shown in Table 1. Observations on the results demonstrate: (1) the proposed multiplicative weighted sampling algorithm achieves prediction accuracy matching the best results provided by naive algorithm, and (2) increasing $λ$ (error rate of oracles) generally brings extra communication cost to achieve comparable performance with the naive algorithm.

6 Conclusions

In this work, we provided several theoretic results regarding fundamental limits of the communication complexity for distributed PAC learning and space complexity for online PAC learning, in the presence of malicious errors in the training examples. We demonstrated how the complexities increase along with the malicious error rates for various distributed learning settings, from simplest two-machine one-round communication protocols to the general multi-machine multi-round communication protocols. A connection between online learning and distributed learning was presented, which gives the space complexity for various online learning protocols. We also provided a boosting flavor distributed robust linear classifier learning algorithm Ws, which presented significantly higher communication efficiency than the naive distributed learning algorithm with negligible classification accuracy loss.

Appendix A Simulations

More details on the used datasets for the simulation evaluation are summarized in Table 2.

Dataset	total # examples	# examples per machine	# dimension $p$
Syn. 1	$2 \times 10^{4}$	$1 \times 10^{4}$	$100$
Syn. 2	$4 \times 10^{4}$	$2 \times 10^{4}$	$100$
Syn. 3	$2 \times 10^{4}$	$0.5 \times 10^{4}$	$100$
Syn. 4	$4 \times 10^{4}$	$1 \times 10^{4}$	$100$
Cancer	$683$	$342$	$10$
Mushroom	$8,124$	$4,062$	$112$

Table 2: Summary of the used datasets. For synthetic datasets Syn.3 and Syn.4, four machines (machines) are used. For the other four datasets, two machines (machines) are used.

Appendix B Communication Complexity Lower Bounds for Other Communication Protocols

b.1 Lower Bound for $1$ -Round $k$ -Machine Communication

Now we consider a slightly more general communication protocol than the $1$ -round $2$ -machine one, where $k$ machines are involved, and only one-round communication is allowed for two neighboring machines in the sequence of communication.

We first note that the multi-machine one-round protocol is not more efficient than the two-machine protocol [3], in terms of the maximal size of message communicated. The total communication complexity lower bound for $1$ -round $k$ -machine protocol is provided in 3.4.

To show this, we associate with every disjoint sample partition $I_{1}, \dots, I_{k}$ over $k$ machines a disjoint sample partition for two machines: for $j = 1, \dots, k - 1$ , $J_{1}^{j} = ⋃_{i = 1}^{j} I_{i}$ and $J_{2}^{j} = ⋃_{i = j + 1}^{k} I_{i}$ , and we have following results.

Lemma 4.

For every hypothesis $h : X \to {0, 1}$ , every disjoint input partition $I_{1}, \dots, I_{k}$ dividing $X$ into ${X_{I_{1}}, \dots, X_{I_{k}}}$ , and every specified error rate $ϵ > 0$ ,

R_{ϵ}^{I_{1} \to \dots \to I_{k}, max} (h) \geq max j R_{ϵ}^{J_{1}^{j} \to J_{2}^{j}} (h),

where the superscript $max$ in the communication complexity denotes the length of the longest message.

Proof.

The two machines can simulate the $k$ -machine protocol as follows: $M_{1}$ simulates the first $j$ machines; it then transmits what the $j$ -th machine would have sent to the $(j + 1)$ -st machine to $M_{2}$ ; $M_{2}$ completes the computation by simulating the last $(k - j)$ machines. ∎

The result also holds for the $μ$ -distributional complexity for any input distribution $μ$ . It therefore follows from Theorem 1 that:

Corollary 2 (Communication Complexity Lower Bound on Maximum Message Size for $1$ -Round $k$ -Machine Protocols).

Let $λ$ be the outlier rate in the oracles. Let $ϵ$ be the specified error rate of $h$ . For every hypothesis $h : X \to {0, 1}$ , every disjoint input partition $I_{1}, \dots, I_{k}$ , we have

R_{ϵ}^{M_{1} \to \dots \to M_{k}, max} (h) = Ω (\frac{1 - ϵ}{1 - λ} d^{'}),

for a constant success probability. Here $d^{'}$ is the maximal VC-dimension among the hypothesis $h_{X_{I_{j}}}$ learned from $X_{I_{j}}$ .

Appendix C Proofs of Main Results

c.1 Tools

The following essential theorem is a consequence of the minmax theorem in [16], which states the randomized public-coin communication complexity is always lower bounded by the distribution complexity.

Theorem 7 (Yao’s principle, [24]).

For every function $f : X \times Y \to {0, 1}$ , and for every $0 < ϵ < 1$ ,

R_{ϵ}^{A \to B, pub} (h) = max μ D_{ϵ}^{A \to B, μ} (h),

where $μ$ ranges over all distributions on $X \times Y$ .

This important result provides us with a convenient way to lower bound the communication complexity, through finding a “hard” distribution $μ$ w.r.t. which the communication complexity is easy to evaluate. Throughout the paper, we usually take a product distribution, i.e., $X$ and $Y$ are independent of each other, as the hard distribution $μ$ .

c.2 Proof of Communication Complexity Lower Bound for $1$ -Round $2$ -Machine Protocol, Theorem 1

To prove the above lower bound, we need the following lemma, which says a lower bounded decoding error is inevitable when the communication channel capacity is limited. Define $d i s t (z, U)$ to be the minimum hamming distance between $z \in {0, 1}^{d}$

and a vector

u \in U \subset {0, 1}^{d}

Lemma 5.

Suppose $d \geq 2$ . For every set $U \subset {0, 1}^{d}$ , $| U | \leq 2^{c d}$ , where $c = \frac{8 (1 - ϵ)}{15 (1 - λ)}$ with $0 \leq λ < ϵ \leq 1$ ,

E [d i s t (z, U)] \geq \frac{ϵ - λ}{1 - λ} d,

where the expectation is taken with respect to the uniform distribution over

z \in {0, 1}^{d}

Proof.

For each $u \in U$ , let $N_{u} = {z \in {0, 1}^{d} | d i s t (z, u) \leq \frac{2 (ϵ - λ)}{1 - λ} d}$ . Denote $t = \frac{2 (ϵ - λ)}{1 - λ}$ , and we have $0 \leq t < 1$ . For each $u \in U$ ,

| N_{u} | = t d \sum i = 0 (\frac{d}{i}) \leq (e d / (t d))^{t d} = 2^{t d log (e / t)} .

Thus,

\sum u | N_{u} | < 2^{c d + t d log (e / t)} \leq 2^{d - 1} .

The last inequality is from the assumption that $d \geq 2$ . We show that $∣ ∣ ⋃_{u} N_{u} ∣ ∣ \leq \sum_{u} | N_{u} | \leq 2^{d - 1}$ , and hence $E [d i s t (z, U)] > \frac{ϵ - λ}{1 - λ} d$ by Markov inequality. ∎

Now we proceed to prove Theorem 1.

Proof.

According to Yao’s theorem (Theorem 7), $R_{ϵ}^{A \to B} (h) = {max}_{μ} D_{ϵ}^{A \to B, μ} (h)$ , where the maximum is taken over all distributions $μ$ .

In order to prove this lower bound, we describe a product distribution $μ = μ_{1} \cup μ_{2}$ for which $D_{ϵ}^{A \to B, μ} (h) = Ω (d)$ , where $d$ is the VC-dimension of $h_{X}$ . By definition of the VC-dimension, there exists a set $S \subseteq Y$ of size $d$ which is shattered by $h_{X}$ – the hypothesis learned from $X$ . Namely, for every subset $R \subseteq S$ there exits $x_{R} \in X$ , such that $\forall y \in S, h_{x_{R}} (y) = 1$ iff $y \in R$ . For each $R \subseteq S$ , fix such an $x_{R}$ . Let $μ_{1}$ be the uniform distribution over the set of pairs ${(x_{R}, y) | R \subseteq S, y \in S}$ . Let $μ_{2}$ be the outliers.

Let $P$ be a single round deterministic protocol for learning $h \in H$ whose cost is at most $c = \frac{8 (1 - ϵ)}{15 (1 - λ)} d$ . Thus, $P$ induces two mappings. $P_{1} : {0, 1}^{d} \to {0, 1}^{c}$ , determines which $c$ bits $M_{1}$ should send to $M_{2}$ for every given $x_{R}$ , and $P_{2} : {0, 1}^{c} \to {0, 1}^{d}$ determines the value of $h$ computed by $M_{2}$ for every $y \in S$ , given the $c$ bits sent by $M_{1}$ . Combining these two mappings together, $P$ induces a mapping $P_{1, 2} ≜ P_{1} \circ P_{2}$ from ${0, 1}^{d}$ into a set $U \subset {0, 1}^{d}$ , where $| U | \leq 2^{c}$ . The expected error of $h$ is

ϵ (h) = (1 - λ) \frac{1}{d 2^{d}} \sum z \in {0, 1}^{d} d i s t (z, P_{1, 2} (z)) + λ,

where $d i s t (\cdot, \cdot)$ denotes the hamming distance between the vectors. Then Lemma 5 gives us

ϵ (h) \geq (1 - λ) \frac{1}{d 2^{d}} \sum z \in {0, 1}^{d} \frac{ϵ - λ}{1 - λ} d + λ = ϵ .

Therefore, if the communication complexity is limited to less than $c = \frac{8 (1 - ϵ)}{15 (1 - λ)} d$ , the error of the learned hypothesis will be always greater than specified error $ϵ$ . Thus, we get the lower bound in Theorem 1. ∎

c.3 Proof of Communication Complexity Lower Bound for $t$ -Round $2$ -Machine Protocols, Theorem 2

Proof.

Suppose the hypothesis $h$ can be learned with $ϵ$ -error following a $t$ -round randomized $[t; ℓ_{1}, \dots, ℓ_{t}]$ protocol $P$ . Let $h_{t}$ denote the learned hypothesis. Applying the round elimination Lemma 1 to $P$ repeatedly for $(t - 1)$ times, we see that $h_{1}$ has a protocol $Q$ with communication complexity $c (Q) \leq ℓ_{1} + \dots + ℓ_{t}$ and

			$e r r (Q)$
		$\leq$	$ϵ + \sqrt{\frac{ℓ_{1}}{d}} + \sqrt{\frac{ℓ_{1} + ℓ_{2}}{} d} + \dots + \sqrt{\frac{ℓ_{1} + \dots + ℓ_{t - 1}}{d}}$
		$\leq$	$ϵ + t \sqrt{\frac{ℓ_{1} + \dots + ℓ_{t}}{d}} .$

Suppose the communication complexity of $P$ satisfies $c (P) \leq \frac{8 (1 - ϵ)}{15 (1 - λ)} \frac{d}{t^{2}}$ . Then $ℓ_{1} + \dots + ℓ_{t} \leq \frac{8 (1 - ϵ)}{15 (1 - λ)} \frac{d}{t^{2}}$ , so $e r r (Q) \leq 2 ϵ$ with communication cost less than $\frac{8 (1 - ϵ)}{15 (1 - λ)} d$ . This is a contradiction to Theorem 1. Therefore, the communication complexity is at least $\frac{8 (1 - ϵ)}{15 (1 - λ)} \frac{d}{t^{2}}$ . ∎

c.4 Proof of Communication Complexity Lower Bound for $1$ -Round $k$ -Machine Protocols, Theorem 3

Proof.

To extend the above lower bound for $2$ -machine communication to the $k$ -machine case, we use the symmetrization technique [17] to reduce $k$ -machine communication to $2$ -machine communication.

The symmetrization is conducted as follows. Consider a protocol $P$ for this $k$ -machine problem, which works on this distribution, communicates $c (P)$ bits in expectation. We build from $P$ a new protocol $P^{'}$ for a $2$ -machine problem. In the $2$ -machine problem, suppose that $M_{1}$ gets input $x$ and $M_{2}$ gets input $y$ , where $x, y \in {0, 1}^{d}$ are independent random vectors. Then $P^{'}$ works as follows: $M_{1}$ and $M_{2}$ randomly choose two distinct index $i, j \in {1, \dots, k}$ using the public randomness, and they simulate the protocol $P$ , where $M_{1}$ plays machine $i$ and lets $x_{i} = x$ , $M_{2}$ plays machine $j$ and lets $x_{j} = y$ , and they both play all of the rest of the machines; the inputs of the rest of the machines is chosen from shared randomness. $M_{1}$ and $M_{2}$ begin simulating the running of $P$ . Every time machine $i$ should speak, $M_{1}$ sends to $M_{2}$ the message that machine $i$ was supposed to communicate, and vice versa. When any other machine $p_{r}$ ( $r \neq i, j$ ) should speak, both $M_{1}$ and $M_{2}$ know his input so they know what it should communicate, thus no communication is actually needed. A key observation is that the inputs of the $k$ machines are uniform and independent and thus entirely symmetrical, and since the index $i$ and $j$ were chosen uniformly at random, then the expected communication performed by the protocol $P^{'}$ is $E [c (P^{'})] = 2 c (P) / k$ . Since $c (P^{'}) \geq \frac{16 (1 - 2 ϵ)}{15 (1 - λ)} d$ (Theorem 1), we have $c (P) \geq \frac{8 (1 - 2 ϵ)}{15 (1 - λ)} d k$ .

∎

c.5 Proof on Equivalence between Distributed and Online Algorithms, Lemma 3

Proof.

An $r$ -pass, $s$ -space online algorithm for learning a hypothesis $h$ on a set $X$ yields an $r$ -round, $p$ -machine communication protocol for learning $h (X)$ when $X$ is partitioned into $p$ subsets $S_{1}, \dots, S_{p}$ and the $i$ th machine receives $S_{i}$ : the $i$ th machine randomly permutes $S_{i}$ to generate stream $s_{i}$ and the machines emulate the online algorithm on the concatenated stream $⟨ s_{1} | s_{2} | \dots | s_{p} ⟩$ . The emulation requires $O (r p s)$ bits of communication. ∎

Appendix D Proof on The Ws Algorithm, Theorem 5

Proof.

At the start of each round $t$ , let $ϕ_{t}$ be the potential function given by the sum of weights of all points in that round. Initially, $ϕ_{1} = \sum_{x_{i} \in D_{B}} w_{i} = n$ since by definition for each point $x_{i} \in D_{B}$ we have $w_{i} = 1$ .

Then in each round, $A$ constructs a hypothesis $h_{A}^{t}$ at $B$ to correctly classify the set of points that accounts for at least $1 - c^{'} = 1 - c \frac{1 - λ}{(1 - 2 λ)^{2}}$ fraction of the total weight by Lemma 2. All other misclassified points are upweighted by $(1 + ρ)$ . Hence, for round $(t + 1)$ we have $ϕ^{t + 1} \leq ϕ^{t} ((1 - c^{'}) + c^{'} (1 + ρ)) = ϕ^{t} (1 + c^{'} ρ) = n (1 + c^{'} ρ)^{t}$ .

Let us consider that weight of the points in the set $S \subset D_{B}$ that have been misclassified by a majority of the $T$ classifiers (after the protocol ends). This implies every point in $S$ has been misclassified at least $T / 2$ number of times and at most $T$ number of times. So the minimum weight of points in $S$ is $(1 + ρ)^{T / 2}$ and the maximum weight is $(1 + ρ)^{T}$ .

Let $n_{i}$ be the number of points in $S$ that has weight $(1 + ρ)^{i}$ where $i \in [T / 2, T]$ . The potential function value of $S$ after $T$ rounds is $ϕ_{S}^{T} = \sum_{i = T / 2}^{T} n_{i} (1 + ρ)^{i}$ . Our claim is that $\sum_{i = t}^{T} n_{i} = | S | \leq ϵ n$ . Each of these at most $| S |$ points have a weight of at least $(1 + ρ)^{T / 2}$ . Hence we have

ϕ_{S}^{T} = T \sum i = T / 2 n_{i} (1 + ρ)^{i} \geq (1 + ρ)^{T / 2} T \sum i = T / 2 n_{i} = (1 + ρ)^{T / 2} | S | .

Relating these two inequalities we obtain the following,

| S | (1 + ρ)^{T / 2} \leq ϕ_{S}^{T} \leq ϕ^{T} = n (1 + c ρ)^{T} .

Hence (using $T = 5 {log}_{2} (1 / ϵ)$ )

			$\| S \| \leq n {(\frac{1 + c^{'} ρ}{(1 + ρ)^{1 / 2}})}^{T}$
		$=$	$n {(\frac{1 + c^{'} ρ}{(1 + ρ)^{1 / 2}})}^{5 {log}_{2} (1 / ϵ)}$
		$=$	$n (1 / ϵ)^{5 {log}_{2} (\frac{1 + c^{'} ρ}{(1 + ρ)^{1 / 2}})} .$

Setting $c^{'} = 0.2$ , which means $c = 0.2 (1 - 2 λ)^{2} / (1 - λ)$ and $ρ = 0.75$ , we get $5 {log}_{2} (\frac{1 + c^{'} ρ}{(1 + ρ)^{1 / 2}}) < - 1$ and thus $| S | < n (1 / ϵ)^{- 1} < ϵ n$ , as desired since $ϵ < 1$ . Thus each round uses $O (p)$ points, each requiring $p$ bits of communication, yielding a total communication of $O (\frac{1 - λ}{(1 - 2 λ)^{2}} p^{2} log (1 / ϵ))$ . ∎

Appendix E Upper Bound for Communication Complexity

We first establish an upper bound on the communication complexity for distributed PAC learning, which is simply derived from the sample complexity result for PAC learning with malicious errors [10]. Before stating the sample complexity result, let us recall the Occam algorithm [4], whose existence is a sufficient condition for the learnability of a PAC learning problem [19].

Definition 2 (Occam Algorithm).

An algorithm $A$ is called an Occam algorithm if it draws $m$ training examples and outputs a hypothesis $h$ such that $h$ is consistent with these examples.

In the presence of malicious outliers (with $λ$ error rate) in the training examples, we desire to develop a $λ$ -tolerant algorithm, defined as follows [10].

Definition 3 ( $λ$ -tolerant Algorithm).

Given a malicious error rate $0 \leq λ \leq 1 / 2$ for the oracles. If an algorithm $A$ with access to the oracles outputs a hypothesis $h$ with error probability $e (h) \leq ϵ$ with a probability of $1 - δ$ , the algorithm $A$ is a $λ$ -tolerant algorithm.

Basically, a $λ$ -tolerant algorithm is able to output a hypothesis with bounded error even in presence of a fraction of $λ$ outliers in the training examples. Similarly, we call an Occam algorithm $A$ as a $λ$ -tolerant Occam algorithm if $A$ outputs a hypothesis $h$ whose error $e (h) \leq ϵ$ over $m$ provided training examples containing malicious outlier with a fraction of $λ$ .

Following theorem demonstrates that if we have a $λ$ -tolerant Occam algorithm, we can always develop a $λ$ -tolerant algorithm for learning any target representation with sufficiently many training examples.

Theorem 8 (Sample Complexity for Robust PAC Learning).

Suppose an algorithm $A$ is a $λ$ -tolerant Occam algorithm for $c \in C$ by $H$ . Then $A$ is also a $λ$ -tolerant algorithm for $c \in C$ by $H$ , and the sample size required by $A$ is $m = O (\frac{(1 - λ)}{(1 - 2 λ)^{2}} (1 / ϵ ln 1 / δ + 1 / ϵ ln | H |))$ , for achieving an error rate of $ϵ$ with a success probability at least $1 - δ$ .

Proof.

Let $h \in H$ be such that its error rate on positive examples $e^{+} (h) \geq ϵ$ . Then the probability that $h$ agrees with a point received from the oracle $\sc Posλ$ is bounded above by

(1 - λ) (1 - ϵ) + λ = 1 - (1 - λ) ϵ .

Thus the probability that $h$ agrees with at least a fraction $1 - ϵ / 2$ of $m$ examples received from $\sc Posλ$ is bounded above by

e^{- m ϵ (1 - 2 λ)^{2} / 8 (1 - λ)},

by Chernoff bound. From this it follows that the probability that some $h \in H$ with $e^{+} (h) \geq ϵ$ agrees with a fraction $1 - ϵ / 2$ of the $m$ examples is at most $| H | e^{- m ϵ (1 - 2 λ)^{2} / 8 (1 - λ)}$ . Solving $| H | e^{- m ϵ (1 - 2 λ)^{2} / 8 (1 - λ)} \leq δ / 2$ , we obtain $m \geq \frac{8}{ϵ} \frac{(1 - λ)}{(1 - 2 λ)^{2}} (ln | H | + ln \frac{δ}{2})$ . ∎

Based on the above sample complexity result, a simplest communication protocol is to just have each machine out of $k$ machines send a random sample of size $O (\frac{1}{k} \frac{1 - λ}{(1 - 2 λ)^{2}} (\frac{1}{ϵ} ln \frac{1}{δ} + \frac{1}{ϵ} ln | H |))$ to a specific machine, which then performs the learning, and there is just one round of communication. Therefore, we can immediately obtain the following upper bound on the communication complexity.

Corollary 3 (Communication Complexity Upper Bound).

Assume a $λ$ -tolerant Occam algorithm always exists. Any target representation $c \in C$ can be learned from $H$ to error $ϵ$ using $1$ round and $O (\frac{(1 - λ)}{(1 - 2 λ)^{2}} (\frac{1}{ϵ} ln \frac{1}{} δ + \frac{1}{ϵ} ln | H |))$ total examples communicated. Suppose each example is represented by $b$ bits, then the total communication complexity is upper bounded by $O (b \frac{(1 - λ)}{(1 - 2 λ)^{2}} (\frac{1}{ϵ} ln \frac{1}{δ} + \frac{1}{ϵ} ln | H |))$ .

References

[1] Noga Alon, Yossi Matias, and Mario Szegedy.
The space complexity of approximating the frequency moments.
In
Proceedings of the twenty-eighth annual ACM symposium on Theory of computing
, pages 20–29. ACM, 1996.
[2] Maria-Florina Balcan, Avrim Blum, Shai Fine, and Yishay Mansour. Distributed learning, communication complexity and privacy. arXiv preprint arXiv:1204.3514, 2012.
[3] Ziv Bar-Yossef, TS Jayram, Ravi Kumar, and D Sivakumar. Information theory methods in communication complexity. In Computational Complexity, 2002. Proceedings. 17th IEEE Annual Conference on, pages 72–81. IEEE, 2002.
[4] Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. Occam’s razor. Information processing letters, 24(6):377–380, 1987.
[5] Hal Daumé III, Jeff M Phillips, Avishek Saha, and Suresh Venkatasubramanian. Efficient protocols for distributed classification and optimization. pages 154–168. Springer, 2012.
[6] Hal Daume III, Jeff M Phillips, Avishek Saha, and Suresh Venkatasubramanian. Protocols for learning classifiers on distributed data. arXiv preprint arXiv:1202.6078, 2012.
[7] John C Duchi, Michael I Jordan, Martin J Wainwright, and Yuchen Zhang. Information-theoretic lower bounds for distributed statistical estimation with communication constraints. arXiv preprint arXiv:1405.0782, 2014.
[8] John C Duchi and Martin J Wainwright. Distance-based and continuum fano inequalities with applications to statistical estimation. arXiv preprint arXiv:1311.2669, 2013.
[9] Ankit Garg, Tengyu Ma, and Huy Nguyen. On communication cost of distributed statistical estimation and dimensionality. In NIPS, pages 2726–2734, 2014.
[10] Michael Kearns and Ming Li. Learning in the presence of malicious errors. SIAM Journal on Computing, 22(4):807–837, 1993.
[11] Flip Korn, S Muthukrishnan, and Yunyue Zhu. Checks and balances: Monitoring data quality problems in network traffic databases. In Proceedings of the 29th international conference on Very large data bases-Volume 29, pages 536–547. VLDB Endowment, 2003.
[12] Ilan Kremer, Noam Nisan, and Dana Ron. On randomized one-round communication complexity. Computational Complexity, 8(1):21–49, 1999.
[13] E Kushilevitz and N Nisan. Communication Complexity. Cambridge University Press, Cambridge, 1996.
[14] Ahmed Metwally, Divyakant Agrawal, and Amr El Abbadi. Efficient computation of frequent and top-k elements in data streams. In Database Theory-ICDT 2005, pages 398–412. Springer, 2005.
[15] Shanmugavelayutham Muthukrishnan. Data streams: Algorithms and applications. Now Publishers Inc, 2005.
[16] J v Neumann. Zur theorie der gesellschaftsspiele. Mathematische Annalen, 100(1):295–320, 1928.
[17] Jeff M Phillips, Elad Verbin, and Qin Zhang. Lower bounds for number-in-hand multiparty communication complexity, made easy. In SODA, pages 486–501. SIAM, 2012.
[18] Pranab Sen. Lower bounds for predecessor searching in the cell probe model. In Computational Complexity, 2003. Proceedings. 18th IEEE Annual Conference on, pages 73–83. IEEE, 2003.
[19] Leslie G Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134–1142, 1984.
[20] Leslie G Valiant. Learning disjunction of conjunctions. In IJCAI, pages 560–566, 1985.
[21] Vladimir N Vapnik and A Ya Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability & Its Applications, 16(2):264–280, 1971.
[22] Angeline Wong, Leejay Wu, Phillip B Gibbons, and Christos Faloutsos. Fast estimation of fractal dimension and correlation integral on stream data. Information Processing Letters, 93(2):91–97, 2005.
[23] Andrew Chi-Chih Yao. Some complexity questions related to distributive computing (preliminary report). In Proceedings of the eleventh annual ACM symposium on Theory of computing, pages 209–213. ACM, 1979.
[24] Andrew Chi-Chin Yao. Probabilistic computations: Toward a unified measure of complexity. In FOCS, pages 222–227. IEEE, 1977.
[25] Yuchen Zhang, John Duchi, Michael Jordan, and Martin J Wainwright. Information-theoretic lower bounds for distributed statistical estimation with communication constraints. In Advances in Neural Information Processing Systems, pages 2328–2336, 2013.

On Fundamental Limits of Robust Learning

Authors

The Optimal Sample Complexity of PAC Learning

Outlier Robust Online Learning

Improved Algorithms for Collaborative PAC Learning

Fundamental Limits of Online Learning: An Entropic-Innovations Viewpoint

On the Sample Complexity of Adversarial Multi-Source PAC Learning

Distributed and Streaming Linear Programming in Low Dimensions

On Communication Complexity of Classification Problems

This week in AI

1 Introduction

1.1 Our Contributions

1.2 Related Works

2 Problem Setup and Preliminaries

Robust PAC Learning

Distributed and Online PAC Learning

Communication Protocols and Complexity

3 Main Results on Distributed Robust PAC Learning

3.1 Communication Models

3.2 Lower Bound for 1-Round 2-Party Communication

Theorem 1 (Communication Complexity Lower Bound for 1-Round 2-Machine Protocol).

Corollary 1.

3.3 Lower Bound for t-Round 2-Machine Communication

Definition 1.

Lemma 1 (Round Elimination, adapted from [18]).

Theorem 2 (Communication Complexity Lower Bound for t-Round 2-Machine Protocols).

3.4 Lower Bound for t-Round k-Machine Communication

Theorem 3 (Communication Complexity Lower Bound for 1-Round k-Machine Protocols).

Theorem 4 (Communication Complexity Lower Bound for t-Round k-Machine Protocols).

3.5 A Communication-Efficient Protocol: Weighted Sampling

Lemma 2.

Theorem 5.

4 Online Robust PAC Learning

Lemma 3.

Theorem 6 (Space Complexity Lower Bound for r-Pass Online Learning).

5 Simulations

6 Conclusions

Appendix A Simulations

Appendix B Communication Complexity Lower Bounds for Other Communication Protocols

b.1 Lower Bound for 1-Round k-Machine Communication

Lemma 4.

Proof.

Corollary 2 (Communication Complexity Lower Bound on Maximum Message Size for 1-Round k-Machine Protocols).

Appendix C Proofs of Main Results

c.1 Tools

Theorem 7 (Yao’s principle, [24]).

c.2 Proof of Communication Complexity Lower Bound for 1-Round 2-Machine Protocol, Theorem 1

Lemma 5.

Proof.

Proof.

c.3 Proof of Communication Complexity Lower Bound for t-Round 2-Machine Protocols, Theorem 2

Proof.

c.4 Proof of Communication Complexity Lower Bound for 1-Round k-Machine Protocols, Theorem 3

Proof.

c.5 Proof on Equivalence between Distributed and Online Algorithms, Lemma 3

Proof.

Appendix D Proof on The Ws Algorithm, Theorem 5

Proof.

Appendix E Upper Bound for Communication Complexity

Definition 2 (Occam Algorithm).

Definition 3 (λ-tolerant Algorithm).

Theorem 8 (Sample Complexity for Robust PAC Learning).

Proof.

Corollary 3 (Communication Complexity Upper Bound).

References

3.2 Lower Bound for $1$ -Round $2$ -Party Communication

Theorem 1 (Communication Complexity Lower Bound for $1$ -Round $2$ -Machine Protocol).

3.3 Lower Bound for $t$ -Round $2$ -Machine Communication

Theorem 2 (Communication Complexity Lower Bound for $t$ -Round $2$ -Machine Protocols).

3.4 Lower Bound for $t$ -Round $k$ -Machine Communication

Theorem 3 (Communication Complexity Lower Bound for $1$ -Round $k$ -Machine Protocols).

Theorem 4 (Communication Complexity Lower Bound for $t$ -Round $k$ -Machine Protocols).

Theorem 6 (Space Complexity Lower Bound for $r$ -Pass Online Learning).

b.1 Lower Bound for $1$ -Round $k$ -Machine Communication

Corollary 2 (Communication Complexity Lower Bound on Maximum Message Size for $1$ -Round $k$ -Machine Protocols).

c.2 Proof of Communication Complexity Lower Bound for $1$ -Round $2$ -Machine Protocol, Theorem 1

c.3 Proof of Communication Complexity Lower Bound for $t$ -Round $2$ -Machine Protocols, Theorem 2

c.4 Proof of Communication Complexity Lower Bound for $1$ -Round $k$ -Machine Protocols, Theorem 3

Definition 3 ( $λ$ -tolerant Algorithm).