Stochastic Tverberg theorems and their applications in multi-class logistic regression, data separability, and centerpoints of data

1 Introduction

This paper shows how methods from stochastic convex geometry can be successfully used in the foundations of data science. Before we discuss the geometric results, we discuss their implications:

Logistic regression is perhaps the most widely used non-linear model in multivariate statistics and supervised learning

[12]. Statistical inference for this model relies on the theory of maximum likelihood estimation. In the binary classification case, given

n

independent observations

(x_{i}, y_{i}), i = 1, \dots, n

, logistic regression links the response

y_{i} \in {- 1, 1}, i = 1, \dots, n

to the covariates

x_{i} \in R^{d}

via the logistic model

P (y_{i} = 1 | x_{i}) = σ (x_{i}^{'} β), σ (t) := \frac{e^{t}}{1 + e^{t}};

here $β \in R^{p}$

is the unknown vector of regression coefficients. In this model, the

log-likelihood is given by

l (β) = n \sum i = 1 - log (1 + exp (- y_{i} x_{i}^{'} β))

and, by definition, the maximum likelihood estimate (MLE) is any maximizer of this functional. The basic intuition behind this method is as follows: we seek coefficients $β$ so that $P (y_{i} = 1 | x_{i})$ corresponds as closely as possible with the observations $(x_{i}, y_{i})$ . For example, if $x_{i}^{'} β$ and $y_{i}$ have different signs, there is a larger “penalty” expressed in the log-likelihood, since in that case $exp (- y_{i} x_{i}^{'} β) > 1$ . See [10] for further discussion and examples.

One difficulty arising in machine learning is that the MLE does not exist in all situations. In fact, given two data classes, say one of red points (where

y_{i} = 1

), and one of blue points (where

y_{i} = - 1

), it is well-known that an MLE exists if and only if the convex hulls of the blue points intersects the convex hull of the red points [1, 18]. Although an appealing criterion for existence, this geometric characterization leads to another question: How much training data do we need, as a function of the dimension of the covariates of the data, before we expect an MLE to exist with high probability?

The seminal work of Cover [6] (adapting a technique originally due to Schläfli [17]) provides an answer in a special case. When applied to logistic regression, Cover’s main result states the following: assume that the $x_{i}$

’s are drawn i.i.d. from a continuous probability distribution

F

and that the class labels are independent from

x_{i}

, and have equal marginal probabilities; i.e.,

P (y_{i} = 1 | x_{i}) = 1 / 2

. Then Cover showed that as

d

and

n

grow large in such a way that

d / n \to k

, the convex hulls of the data points asymptotically overlap - with probability tending to one - if

k < 1 / 2

, whereas they are separated - also with probability tending to one - if

k > 1 / 2

. When the class labels are not independent from the

x_{i}

, the problem is more difficult. In this case, Candès and Sur [3]

proved that a similar phase transition occurs, and is parameterized by two scalars measuring the overall magnitude of the unknown sequence of regression coefficients.

Tukey introduced a notion of depth for a point $p$ relative to a data cloud $S$ , as the smallest number of data points in a closed half-space with boundary through $p$ (see [21, 16] and references therein). We say a point $p$ has half-space depth $k$ in $S$ if that every half-space containing $p$ contains at least $k$ points in $S$ . A centerpoint of an $n$ point data set $S \subset R^{d}$ is a point $p$ such that every half-space containing $p$ has at least $\frac{n}{d + 1}$ points in $S$ , thus it is a point of depth at least $\frac{n}{d + 1}$

. In a way a centerpoint is a generalization of the notion of median for high-dimensional data. Centerpoints are useful in a variety of applications (see e.g.,

[7] for references). Unfortunately, obtaining a centerpoint is difficult, and the current best randomized algorithm constructs a centerpoint in time

O (n^{d + 1} + n log n)

[4, 13]. Thus finding an approximate centerpoint of a set is of interest.

Consequences of our geometric results

The first contribution of our paper is to further develop the connection between geometric probability (Cover’s result), discrete geometry (Tverberg-type results), and the conditions for the existence of MLEs. Our paper discusses the generalization of Cover’s stochastic separation problem to more than two colors by studying so-called Tverberg partitions- a partition of a data set into classes so that the intersection of all the convex hulls of the classes is nonempty.

Each of our stochastic-geometric theorems has a nice implication. Table 1 summarizes our theorems (middle column) as well as their consequences to the existence of the Maximum-likelihood estimator in terms of the size of the data set (right column).

Deterministic version	Stochastic version	Likely MLE Existence
Radon	Cover’s Theorem[6]	pair of data classes (mentioned above)
Tverberg	Thms 4,5	all pairs of data classes
		(Theorem 2 part 1.)
Radon with tolerance	Thm 7	pair of data classes with outliers removed
		(Theorem 2 part 2.)
Tverberg with tolerance	Thms 2,6,[19]	all pairs of data classes with outliers removed
		(Theorem 2 part 2.)

Table 1: Stochastic analogues of Tverberg’s theorem and their implications for existence of MLEs. By “Likely MLE Existence”, we mean that one can bound below the probability of MLE existence as a function of the number of input data points, according to the corresponding theorems in the “Stochastic” column.

There are two common approaches to extend binary classification to multi-class classification: “one-vs-rest” and “one-vs-one”. Suppose the data has $C > 2$ classes. In “one-vs-rest”, we train $C$

separate binary classification models. Each classifier

f_{c}

for

c \in {1, \dots, C}

is trained to determine whether or not an example is part of class

c

or not. To predict the class for a new sample

x

, we run all

C

classifiers on

x

and choose the class with the highest score:

^y=argmaxc∈{1,…,C}fc(x).

In “one-vs-one” regression, we train

(\frac{C}{2})

separate binary classification models, one for each possible pair of classes. To predict the class for a new sample

x

, we run all

(\frac{C}{2})

classifiers on

x

and choose the class with the most votes.

To apply “one-vs-one” multinomial logistic regression, we would like to ensure that the MLE exists between the data corresponding to every pair of labels. The next theorem applies our stochastic Tverberg theorem to give a sufficient condition for all these MLEs to exist with high probability (a sequence of events $X_{n}, n \geq 1$ , occurs with with high probability if ${lim}_{n \to \infty} P (X_{n}) = 1$ .)

Theorem 1 (Stochastic Tverberg theorems applied to multinomial regression)

Fix $ϵ > 0$ . Assume that the $x_{i}$ ’s are drawn i.i.d. from a centrally symmetric continuous probability distribution $F$ on $R^{d}$ and that the class labels are independent from $x_{i}$ , and have equal marginal probabilities; i.e., $P (y_{i} = k | x_{i}) = 1 / m$ for all $k \in {1, \dots, m}$ .
Then

Letting the number of data points $f (m)$ grow as a function of the number of labels $m$ , the MLE exists between the data corresponding to every pair of labels with high probability as long as

$f (m) ≫ (1 + ϵ) m {log}_{2} (m) ln (ln (m)) .$
Suppose the number of data points is $g (t)$ , where we fix $m$ the number of labels, and $g (t)$ is a function of $t$ - the number of outliers to be removed from the data set. Then the MLE exists between the data corresponding to every pair of labels with high probability if any $t$ points are removed, so long as $t ≪ g (t) / m$ .

$_{□}$

The same bound applies to “one-vs-rest” logistic regression, since MLE existence in that case is a weaker condition. The various special cases of Stochastic Tverberg theorems are thus useful in different kinds of classification problems, and these observations are summarized in Table 1.

The last two rows of the table of summarizing our results was motivated by the challenge of dealing with outlier data and seeking robust classification of data, we rely on an additional parameter: tolerance. A $t$ -tolerant partition, which will be defined formally later, but it is a notion of “robust” intersection in the sense that the intersection of the convex hulls of the subsets remains non-empty even after any $t$ points are removed. See Figure 1 for an example of a $1$ -tolerant partition.

Figure 1: A partition in three data classes with tolerance one. All three convex hulls intersect even after any one point is removed.

The parameter of tolerance is also significant in studying MLE existence. A natural observation is that $t$ tolerant partitions correspond to robust MLE existence. Any $t$ points, possibly corrupted or outlier data, can be removed and still the convex hulls of the data with each label intersect.

In fact, the parameter of tolerance is also similar to an important parameter used to guarantee to the convergence speed of first order methods for finding MLEs. Recently, when studying binomial logistic regression, Freund, Grigas and Mazumunder [9] introduced the following notion to quantify the extent that a dataset is non-separable (where $a^{-} := - min {a, 0}$ denotes the negative part of $a$ ):

DegNSEP* := min β \in R^{p} \frac{1}{n} n \sum i = 1 [y_{i} β^{T} x_{i}]^{-}

s.t. ∥ β ∥ = 1.

DegNSEP* is thus the smallest (over all normalized models $β$ ) average misclassification error of the model $β$ over the $n$ observations. They showed that the condition number DegNSEP* informs the computational properties and guarantees of the standard deterministic first-order steepest descent solution method for logistic regression. Let us now briefly discuss how the parameter of tolerance for Radon partitions (Tverberg 2-partitions) can be viewed as a discrete analogue of DegNSEP*.

Define PertSEP* as the smallest (or more precisely, the infimum thereof) perturbation $Δ X$ of the feature data $X$ which will render the perturbed problem instance $(X + Δ X, y)$ separable. Namely,

PertSEP* := inf Δ X \frac{1}{n} ∥ Δ ∥_{\cdot, 1}

s.t. (X + Δ X, y) is separable.

In Proposition 2.4 of [9] it is shown that DegNSEP* = PertNSEP*.

In this paper we introduce a new parameter ${PertSEP*}_{0}$ simply defined as the $L_{0}$ norm of the smallest perturbation of the feature data $X$ which will render the perturbed problem instance $(X + Δ X, y)$ separable. In other words, it is the minimal number of data points we could move to make the data set separable, normalized by the total number of data points. Namely,

{PertSEP*}_{0} := inf Δ X \frac{1}{n} ∥ Δ ∥_{\cdot, 0}

s.t. (X + Δ X, y) is separable.

The following theorem shows that the tolerance of a Radon partition is given by ${PertSEP*}_{0}$ :

Theorem 2

Suppose that $X = X_{1} \cup X_{2}$ , $| X | = n$ is a Radon partition with tolerance precisely equal to $t$ . Then viewing $X$ as a labeled dataset (with $(X, y) = {(x, - 1) : x \in X_{1}} \cup {(x, 1,) : x \in X_{2}}$ ), we have that

{PertSEP*}_{0} = t / n .

$_{□}$

Theorem 2, combined with a result of Soberón, has a corollary, stated precisely in the next section, which roughly says that ${PertSEP*}_{0}$ of a randomly bi-partitioned point set asymptotically approaches $1 / 2$ . This is the highest possible value one could hope for since, by definition, ${PertSEP*}_{0}$ of any two class data set is bounded above by $1 / 2$ . In fact, this result extends easily to the multi-class setting. In other words, for a large randomly $m$ -partitioned data set, we expect ${PertSEP*}_{0}$ of every pair of data classes to be close to $1 / 2$ - independent of both the dimension of the covariates, as well as the number of classes $m$ .

For further discussion of PertSEP* and $DegNSEP *$ for two-class data, including more probabilistic aspects of these condition numbers and many interesting implications for steepest descent algorithms, see [9].

We also discuss how our geometric probability results are related to the problem of computing approximations to centerpoints of datasets. Table 2 and the discussion that follows summarize our contributions. Tverberg’s theorem implies that every data set has a centerpoint, as the Tverberg intersection point of a Tverberg partition must be a point of half-space depth one in each of the $m = ⌈ \frac{n}{d + 1} ⌉$ color classes. Hence an effective version is desirable as a method to obtain centerpoints. The proof of Radon’s lemma is constructive and, in fact, one of the most notable randomized algorithms for computing approximate centerpoints works by repeatedly replacing subsets of $d + 2$ points by their Radon point. In contrast, no known polynomial time algorithm exists for computing exact Tverberg points. Thus, fast algorithms for approximate Tverberg points have been introduced in [5, 14, 15]. If one is interested in probabilistic algorithms for finding Tverberg partitions, the main results of our paper can be used to give expected performance of algorithms where we obtain Tverberg partitions by random choice, so long as the points come from a balanced distribution.

In particular, our Theorem 5 suggests a trivial algorithm for finding a Tverberg partition among a set of i.i.d. points drawn from a distribution which is balanced about a point $p$ . According to Theorem 5, a random equipartition of $m$ such points into less than $m / l o g_{2} (m)$ sets should produce a Tverberg partition with high probability. This trivial non-deterministic algorithm was also suggested by Soberón, except using a random allocation rather than equi-partition. Our asymptotic results improve the bounds on expected performance of Soberón’s proposed algorithm (random allocation) for points from a balanced distribution as well. We summarize the performance and time complexity of various algorithms for obtaining Tverberg partitions, including our own (last two rows), in Table 2.

Method	Number of Colors	Time complexity
Tverberg	$⌊ (m + 1) / (d + 1) ⌋$	PPAD (unknown if polynomial)
Mulzer, Werner [14]	$m / (4 d + 1)^{3}$	$d^{O (log d)} m$
Rolnick, Soberón [15]	$m / d (d + 1)^{2}$ with error prob. $ϵ$	weakly poly. in $m$ , $d$ and $log (1 / ϵ)$
Random equi-partition	$O (\frac{m}{{log}_{2} (m)})$	$O (m)$
Random Allocation	$O (\frac{m}{{log}_{2} (m) (ln (ln (m)))})$	$O (m)$

Table 2: Approximate Tverberg Partitions for balanced distributions.

Section 2 presents our geometric tools and results, and Section 3 contains the proofs of our new results.

2 Our geometric methods: Stochastic Tverberg-type theorems

We begin by remembering Tverberg’s celebrated theorem [22] which generalizes Radon’s lemma to $m$ -partitions (see [2, 7] for references and the importance of this theorem):

Theorem (Theorem: (H. Tverberg 1966))

Every set $S$ with at least $(d + 1) (m - 1) + 1$ points in Euclidean $d$ -space has at least one $m$ -Tverberg partition (with tolerance zero). $_{■}$

The notion of “tolerant Tverberg theorems” was pioneered by Larman [11] and refined over the years, such as in the following result due to Soberón and Strausz [20]. Here is the precise definition:

Definition 1

Given a set $S \subset R^{d}$ , a Tverberg $m$ -partition of $S$ with tolerance $t$ is a partition of $S$ into $m$ subsets $S_{1}, \dots S_{m}$ with the property that all $m$ convex hulls of the $S_{i}$ intersect after any $t$ -points are removed. In other words, for all ${x_{1}, \dots, x_{t}} \in S$ , we have

⋂ i \in [m] conv (S_{i} ∖ {x_{1}, \dots, x_{t}}) \neq \emptyset .

$_{□}$

Theorem (Theorem: (Soberón, Strausz 2012))

Every set $S$ with at least $(t + 1) (m - 1) (d + 1) + 1$ points in $R^{d}$ has at least one Tverberg $m$ -partition with tolerance $t$ . In other words, $S$ can be partitioned into $m$ parts $S_{1}, \dots, S_{m}$ so that for all ${x_{1}, \dots, x_{t}} \in S$ , we have

⋂ i \in [m] conv (S_{i} ∖ {x_{1}, \dots, x_{t}}) \neq \emptyset .

$_{■}$

More recently, P. Soberón proved the following bound [19]. Let $N$ denote the smallest positive integer such that a Tverberg $m$ -partition with tolerance $t$ exists among any $N$ points in dimension $d$ . Then $N = m t + O (\sqrt{t})$ for fixed $m$ and $d$ . The proof of this result relies on the probabilistic method and, as Soberón remarked, can in fact be used to prove a Stochastic Tverberg-type result, which we will revisit later.

Prior Stochastic Tverberg theorems

Before stating our main results, we introduce two models for random partitioned data point sets. In both models will use the term colors instead of subsets, for ease of notation. Hereafter, when we refer to a continuous distribution on $R^{d}$ , we mean continuous with respect to the Lebesgue measure on $R^{d}$ . We defer proofs of the new results stated until the next section.

Our first model is a so-called random equi-partition model i.e., we ensure that every color has the same number of points. More specifically, given integers $m$ and $n$ and a continuous probability distribution $D$ on $R^{d}$ , we let $E_{m, n, D}$ denote a random equi-partitioned point set with $m n$ points, consisting of $m$ colors, and $n$ points of each color, distributed independently according to $D$ .

Our second model is a random allocation model: Given integers $k$ and $m$ and a continuous probability distribution $D$ on $R^{d}$ , we let $R_{m, k, D}$ denote a random point set with $k$ points i.i.d. according to $D$ , which are randomly colored one of $m$ colors with uniform probability ( $1 / m$ for each color).

For example, using these models we can state Cover’s result as follows:

Theorem (Theorem: (T. Cover 1965))

If $D$ is a continuous probability distribution on $R^{d}$ , then

P (R_{2, n, D} is Radon) = 1 - 2^{- n + 1} d \sum k = 0 (\frac{}{n - 1}) k .

In particular, we have

P (R_{2, 2 (d + 1), D} is Radon) = 1 / 2.

Furthermore, for any $ϵ > 0$ and any sequence of continuous probability distributions ${D_{i}}, i \in Z_{+}$ where each $D_{d}$ is a distribution on $R^{d}$ , we have

lim i \to \infty P (R_{2, (1 + ϵ) 2 i, D_{i}} % is Radon) = 1

and

lim i \to \infty P (R_{2, (1 - ϵ) 2 i, D_{i}} % is Radon) = 0

$_{■}$

To the best of the authors’ knowledge, the first generalization of Cover’s 1964 result to more than two colors appeared only recently in Soberón’s paper [19]:

Theorem (Theorem: P. Soberón 2018)

Let $N, t, d, m$ be positive integers and let $ϵ > 0$ be a real number. Given $N$ points in $R^{d}$ , a random allocation of them into $m$ parts is a Tverberg partition with tolerance $t$ with probability at least $1 - ϵ$ , as long as

t + 1 \leq N / m - \sqrt{\frac{1}{2} [(d + 1) (m - 1) N ln (N m) + N ln (\frac{1}{ϵ})]} .

$_{■}$

This result is quite remarkable. For any fixed $m, d,$ and $δ$ , it shows that the probability of a random allocation of of $N$ points in $R^{d}$ in $m$ colors having tolerance at least $(1 - δ) N / m$ approaches one as $N$ goes to infinity. On the other hand, by pigeonhole principle, any allocation of $N$ points into $m$ colors must have one color with at most $N / m$ points. Thus, for a fixed number of colors $m$ , the tolerance of a random partition is asymptotically as high as it could possibly be! By Theorem 2, this result yields the following corollary.

Corollary 1

For any sequence ${R_{(2, k, D)}}, k \in N$ of partitioned point sets with $D$ a distribution on $R^{d}$ , and any $ϵ > 0$ , we have $| {PertSEP*}_{0} (R_{(2, k, d)}) - 1 / 2 | < ϵ$ with high probability.

$_{□}$

In fact, for fixed $d$ and $m$ , Corollary 1 can be extended to the multi-class setting. In other words, for a large randomly $m$ -partitioned data set, we expect ${PertSEP*}_{0}$ of every pair of data points to be close to $1 / 2$ :

Theorem 3

Fix $ϵ > 0$ . For any distribution $D$ on $R^{d}$ and any sequence ${R_{(m, k, D)}}, k \in N$ of $m$ -partitioned point sets $R_{(m, k, D)} = {X_{1}, \dots, X_{m}}$

we have

lim k \to \infty (min X_{i}, X_{j} \in R_{(m, k, D)} % {P e r t S E P *}_{0} (X_{i} \cup X_{j}) = 1 / 2)

with high probability. $_{□}$

Our new stochastic geometric theorems

Our first theorem is a geometric probability result similar to Soberón’s and Cover’s. It yields a Stochastic Tverberg theorem for equi-partitions (without tolerance).

Theorem 4 (Stochastic Tverberg theorem for equi-partitions)

Suppose $D$ is a probability distribution on $R^{d}$ that is balanced about some point $p \in R^{d}$

, in the sense that every hyperplane through

p

partitions

D

into two sets of equal measure. Then

{(1 - (\frac{1}{2^{n - 1}} d - 1 \sum k = 0 (\frac{n - 1}{k})))}^{m} \leq P (E_{m, n, D} is Tverberg) \leq {(2 (1 - 2^{- n})^{m} - (1 - 2^{- n + 1})^{m})}^{d} .

$_{□}$

In fact, the previous theorem is asymptotically tight in the number of colors $m$ . This is shown by our next theorem, which establishes an interesting threshold phenomenon for Tverberg partitions.

Theorem 5 (Tverberg Threshold Phenomena for equi-partitions)

Let $D$ be a continous probability distribution in $R^{d}$ balanced about some point $p \in R^{d}$ . Consider the sequence of random equi-partitioned point sets $E_{m, f (m), D}$ , where $m \in N$ , and $n = f (m)$ depends on $m$ . Then $E_{m, f (m), D}$ is Tverberg with high probability if $f (m) ≫ {log}_{2} (m)$ , and $E_{m, f (m), D}$ is not Tverberg with high probability if $f (m) ≪ {log}_{2} (m)$ .

$_{□}$

Remark: It is also interesting to consider the same problem from the “box convexity” setting where the convex hull of a set of points is defined to be the smallest box (with sides parallel to the coordinate axes) enclosing those points. Since checking convex hull membership is easier in the box convexity setting, this set up may be more relevant in certain applications. Our method of proof of Theorem 4 also works in box convexity setting, and we obtain the same bounds.

We note that the number of points needed to reach the conclusion in Theorem 5 is independent of the dimension, as in the aforementioned result of Soberón [19].

The next two theorems adapt both Cover’s result and Theorem 4 to the setting of tolerance.

Theorem 6 (Stochastic Tverberg with tolerance for equi-partition)

Suppose $D$ is a probability distribution on $R^{d}$ that is balanced about some point $p \in R^{d}$ .

$_{□}$

For the case of random bi-partitions, we can adapt Cover’s result to obtain a Stochastic Radon theorem with Tolerance.

Theorem 7 (Stochastic Radon with tolerance for random allocation)

If $D$ is a continuous probability distribution on $R^{d}$ , then

In particular, we have

P (R_{2, k, D} is Radon with tolerance ⌊ k / (4 d + 4) ⌋) \geq 1 / 2.

$_{□}$

Remark: Theorem 7 yields a weaker expected tolerance than Soberóns result, but the proof is shorter and more elementary.

For random allocations with more than two colors, we will use some developments on random allocation problems, including the following notation. If balls are thrown into $m$ urns uniformly and independently, let $N_{m} (n)$ equal the number of throws necessary to obtain at least $m$ balls in each urn.

Corollary 2 (Stochastic Tverberg for random allocation)

Suppose $D$ is a probability distribution on $R^{d}$ that is balanced about some point $p \in R^{d}$ .

Then

$P (R_{m, k, D} is Tverberg with tolerance t) \geq P (N_{n} (m) \leq k) {(1 - 2^{- ⌊ n / 2 d ⌋} k \sum i = 1 (\frac{}{⌊ n / 2 d ⌋}) i)}^{m} .$
For the case of Tverberg without tolerance, we also have

$P (R_{m, k, D} is Tverberg) \geq P (N_{n} (m) \leq k) {(1 - (2^{- n + 1} d - 1 \sum k = 0 (\frac{n - 1}{k})))}^{m} .$
Suppose $R_{m, f (m), D}$ , $m \in N$ is a sequence of random partitioned point sets, where $n = f (m)$ depends on $m$ .

Then $R_{m, f (m), D}$ is Tverberg with high probability if $f (m) ≫ m {log}_{2} (m) ln (ln (m))$ .

$_{□}$

These results are improvements on Soberón’s bound when the number of colors is large relative to the desired tolerance.

3 Proofs of our stochastic results

Proof (Proof of Theorem 2)

Let $M$ denote the minimal number of points perturbed among any perturbation that makes $(X, y)$ separable, and $N$ denote the minimal number of points needing to be removed from $(X, y)$ to make $(X, y)$ separable. Then ${PertSEP*}_{0} (X, y)$ is equal to $M / n$ , and the tolerance $t$ of $X_{1}$ , $X_{2}$ is equal to $N$ . It suffices to show that $M = N$ . To see that $M \geq N$ , note if $x_{1}, \dots x_{M}$ in $X$ are moved so that the resulting set $(X^{'}, y^{'})$ is separable, then $(X ∖ {x_{1}, x_{2}, \dots, x_{N}}, y ∖ {y_{1}, \dots y_{N}})$ is also separable. To see that $M \leq N$ , suppose that $(X ∖ {x_{1}, x_{2}, \dots, x_{M}}, y ∖ {y_{1}, \dots y_{M}})$ is separable by a hyperplane. Then moving $x_{1}, \dots x_{M}$ to the appropriate sides of the hyperplane determined by $h$ , we can construct a separable dataset $(X^{'}, y^{'})$ , obtained from moving $M$ points from $(X, y)$ . $_{■}$

Proof (Proof of Theorem 3)

For fixed $m, d,$ and $δ$ , let $E_{0} (N)$ denote the event that a random allocation of $N$ points in $R^{d}$ in $m$ colors has tolerance at least $(1 - δ) N / m$ . By Soberón’s theorem above, $E_{0} (N)$ asymptotically approaches one as $N$ goes to infinity. Now, for fixed $m$ and $ϵ$ , let $E_{i} (N)$ denote the event that a random allocation of $N$ points into $m$ colors has between $(1 - ϵ) N / m$ and $(1 + ϵ) N / m$ points of color $i$ , where $i \in [m]$

. By the law of large numbers,

E_{i} (N)

approaches one as

N

goes to infinity. As the

m + 1

events

E_{j} (N)

, where

j = {0, 1, \dots, m}

, all have probability approaching one, the probability of the intersection of all these events also approaches one. This can be seen by applying the union bound to their complements. Thus there exists

N^{'} \in N

such that the

E_{j} (N^{'})

, where

j = {0, 1, \dots, m}

, simultaneously occur with probability

(1 - ϵ_{2})

. Therefore with probability

(1 - ϵ_{2})

, each pair of colors has at most

(1 + ϵ) 2 N^{'} / m

points, and is a Radon partition of tolerance at least

(1 - δ) N^{'} / m

(the tolerance of each bi-partition is a priori bounded below by the tolerance of the

m

-partition). By theorem 2,

{PertSEP*}_{0}

of each pair is at least

(1 - δ) / 2 (1 + ϵ)

with probability

(1 - ϵ_{2})

. Since

δ

and

ϵ

were arbitrary, this completes the proof.

_{■}

Proof (Proof of the lower bound in Theorem 4)

After a possible translation, can assume without loss of generality that $D$ is balanced about the origin. We will prove that

{(1 - (2^{- n + 1} d - 1 \sum k = 0 (\frac{n - 1}{k})))}^{m} \leq P (P_{m, n, D} is Tverberg)

by bounding from below the probability that the origin is a Tverberg point. We may assume without loss of generality that none of the randomly selected points are the origin. Furthermore we can radially project the points onto a sphere of radius smaller than the minimal norm of the projected points, since that will not affect whether the origin is a Tverberg point. After this projection, we may assume the points are uniformly sampled on a small sphere centered at the origin. The origin is then a Tverberg point as long as the points from each color contain the origin in their convex hull. This is equivalent to showing no color has all of its points contained in one hemisphere. For a fixed color, the probability of the $n$ points of that color being contained in one hemisphere was computed by Wagner and Welzl [23] (generalizing the celebrated result of Wendel [24] addressing the case when $D$ is rotationally invariant about the origin) as

(2^{- n + 1} d - 1 \sum k = 0 (\frac{n - 1}{k})) .

(1)

Using this to compute the probability that none of the $m$ color classes is contained in one hemisphere we obtain the desired bound above.

$_{■}$

Proof (Proof of the upper bound in Theorem 4)

Again, we assume without loss of generality that $D$ is balanced about the origin. We will first treat the case $d = 1$ , and then explain how to obtain the bound for arbitrary $d$ . To bound the probability of a Tverberg partition from above, we bound the probability of the complement below. We let $E$ denote the event that the convex hulls have empty intersection. In dimension one, $E$ is contained in the event that there is at least one color class with all points less than zero, and at least one color class with all points greater than zero. Since we assume that the origin equipartitions $D$ , we can rephrase this as the probability that among $m$ people each flipping $n$ fair coins, there is at least one person with all heads and at least one person with all tails. In other words, denoting by $H$ and $T$ the events that at least one person gets all heads or tails respectively, we have $P (E) \geq P (H \cap T)$ . We have

P (H \cap T) = P (H) + P (T) - P (H \cup T) = P (H) + P (T) - (1 - P ((H \cup T)^{c}) .

Since $P (H) = P (T) = (1 - 2^{- n})^{m}$ and $P ((H \cup T)^{c}) = 1 - (1 - 2^{- n + 1})^{m}$ , this yields

P (H \cap T) = 1 + (1 - 2^{- n + 1})^{m} - 2 (1 - 2^{- n})^{m} .

The probability of a Tverberg partition is thus bounded as follows

P (P_{m, n, D} is Tverberg) \leq 1 - P (E) \leq 1 - P (H \cap T) = 2 (1 - 2^{- n})^{m} - (1 - 2^{- n + 1})^{m} .

This proves the desired bound for dimension one. For higher dimensions, we note that if we let $p_{i}$ , denote the projection onto the $i$ -th axis for $i \leq d$ , we have that the signs of $p_{1} (x), \dots, p_{d} (x)$

are independent Bernoulli random variables with probability

1 / 2

(as the hyperplane orthogonal to the

i

-th axis equipartitions

D

by the assumption that

D

is balanced about the origin). Thus to have a Tverberg partition, we must have that no pair of the color classes are separated by the origin after projecting onto the

d

coordinate axes. Since these

d

events are independent, the probability of this happening is bounded as follows.

P (P_{m, n, D} is Tverberg) \leq {(2 (1 - 2^{- n})^{m} - (1 - 2^{- n + 1})^{m})}^{d} .

$_{■}$

Proof (Proof of Theorem 5)

We will show that $P_{m, f (m), D}$ is Tverberg with high probability if $f (m) > ln (m) / ln (2)$ . Fix an $ϵ > 0$ . We set $n = (1 + ϵ) {log}_{2} (m)$ and apply the lower bound in Theorem 4 to deduce that

P (P_{m, n, D} is Tverberg) \geq {(1 - (2^{- (1 + ϵ) * {log}_{2} (m) + 1} d - 1 \sum k = 0 (\frac{n - 1}{k})))}^{m}

= {(1 - (2 m^{- (1 + ϵ)} d - 1 \sum k = 0 (\frac{n - 1}{k})))}^{m} .

Choosing a constant $K$ so that $K n^{d} \geq 2 \sum_{k = 0}^{d - 1} (\frac{n - 1}{k})$ , we have

(1 - K n^{d} m^{- (1 + ϵ)})^{m} \leq P (P_{m, n, D} is % Tverberg) .

We will show that the limit as $m$ approaches infinity of the left hand side is bigger than $e^{- δ}$ for any $δ > 0$ . Fix $δ > 0$ . As $n^{d} \sim O (ln (m)^{d})$ , there exists an $M$ such that $K n^{d} m^{- ϵ} < δ$ for all $m \geq M$ . Consequently $(1 - K n^{d} m^{- (1 + ϵ)})^{m} > (1 - δ m^{- 1})^{m}$ for all $m \geq M$ . Thus

lim m \to \infty (1 - K n^{d} m^{- (1 + ϵ)})^{m} \geq lim m \to \infty (1 - δ m^{- 1})^{m} = e^{- δ} .

Since $δ$ was arbitrary, we see that the probability of a Tverberg partition tends to 1.

Now we show that $P_{m, f (m), D}$ is not Tverberg with high probability if $f (m) < {log}_{2} (m)$ . As before, we fix an $ϵ$ greater than zero apply the upper bound in Theorem 4 with $n = (1 - ϵ) l o g_{2} (m)$ to obtain

P (P_{m, n, D} is Tverberg) \leq {(2 (1 - m^{- 1 + ϵ})^{m} - (1 - 2 m^{- 1 + ϵ})^{m})}^{d} .

For any $γ > 0$ , when $m$ is large, both terms inside the parentheses are smaller than $(1 - γ m^{- 1})^{m}$ . Since ${lim}_{m \to \infty} (1 - γ m^{- 1})^{m} = e^{- γ}$ , the probability of a Tverberg partition converges to zero as $m$ approaches infinity.

$_{■}$

Proof (Proof of Theorem 6)

Again, we assume without loss of generality that $D$ is balanced about $0$ . Let $S$ denote the set of points of some fixed color. Then we assume that $| S | = n$ , and we can partition $S$ into $⌊ n / 2 d ⌋$ subsets $S_{1}, \dots, S_{⌊ n / 2 d ⌋}$ with $S_{i} \geq 2 d$ for each $i$ . By Wagner and Welzl’s result (Equation 3 above), for each $i$ , $conv (S_{i})$ contains the origin with probability at least $1 / 2$ . By independence, the probability that less than $t + 1$ of the $S_{i}$ contain the origin is less than $2^{- ⌊ n / 2 d ⌋} \sum_{i = 1}^{t} (\frac{⌊ n / 2 d ⌋}{i})$ . On the other hand, if at least $t + 1$ of the $conv (S_{i})$ contain the origin, then by pigeonhole principle $conv (S ∖ {x_{1}, \dots, x_{t}})$ contains the origin for any $x_{1}, \dots x_{t} \in S$ . Thus, with probability at least $1 - 2^{- ⌊ n / 2 d ⌋} \sum_{i = 1}^{t} (\frac{⌊ n / 2 d ⌋}{i})$ , we have that $conv (S) ∖ {x_{1}, \dots, x_{t}})$ contains the origin. Since this probability is independent for each of the $m$ colors, the result follows. $_{■}$

Using a similar strategy combined with Cover’s result, we give the proof of Theorem 7 below.

Proof (Proof of Theorem 7)

Given $k$ points in $R^{d}$ colored red and blue by random allocation, we arbitrarily partition them into $⌊ k / (2 d + 2) ⌋$ groups of size at least $2 d + 2$ . By Cover’s result, for each fixed group, the convex hulls (of the red and blue points) in that group intersect with probability at least $1 / 2$ . For each of the $⌊ k / (2 d + 2) ⌋$ groups, we think of the event that the convex hulls in that group intersect as a “success”. Then the probability that at least $t + 1$ groups have intersecting convex hulls is bounded below by the probability that a binomial process with $⌊ k / (2 d + 2) ⌋$ trials and success probability $1 / 2$ has at least $t + 1$ total successes. Computing this binomial probability yields the theorem. (If at least $t + 1$ groups have intersecting convex hulls, then removing at most $t$ points leaves at least one group with intersecting convex hulls. ) $_{■}$

Proof (Proof of Corollary 2)

We split the proof according to the three respective parts of the statement.

The probability that a random allocation of $k$ points into $m$ colors is an $m$ -Tverberg partition with tolerance $t$ is bounded below by the probability that a random allocation of $k$ points into $m$ colors has at least $n$ points per color, times the probability that an equipartition of $n m$ points into $m$ colors is Tverberg with tolerance $t$ . The result for Tverberg with tolerance then follows from Theorem 6.
The result for the special case of Tverberg without tolerance then follows the same reasoning as part (1), except using Theorem 4 in place of Theorem 6.
To show the asymptotic result, we use a result on urn models due to Erdős and Renyi [8] saying that

$lim n \to \infty P (\frac{N_{m} (n)}{n} < log (n) + (m - 1) log (log (n)) + x) = e x p (- \frac{e^{- x}}{(m - 1)!}) .$

This implies that for any $ϵ > 0$ and sequence of $log (log (m)) l o g_{2} (m) (1 + ϵ)$ points allocated into $m$ urns, we have at least $l o g_{2} (m) (1 + ϵ / 2)$ points in each urn with high probability. Then we apply Theorem 5, which says that any equi-partition of a point set into $m$ colors and $l o g_{2} (m) (1 + ϵ / 2)$ points per color is Tverberg with high probability.

$_{■}$

4 Acknowledgments

This work was partially supported by NSF grants DMS-1522158 and DMS-1818169. We are grateful to David Rolnick and Pablo Soberón for their comments.

References

[1] A. Albert and J. A. Anderson, On the existence of maximum likelihood estimates in logistic regression models, Biometrika, 71 (1984), pp. 1–10, https://doi.org/10.1093/biomet/71.1.1, http://dx.doi.org/10.1093/biomet/71.1.1, https://arxiv.org/abs//oup/backfile/content_public/journal/biomet/71/1/10.1093/biomet/71.1.1/2/71-1-1.pdf.
[2] I. Bárány and P. Soberón, Tverberg’s theorem is 50 years old: A survey, Bull. Amer. Math. Soc., 55 (2018), pp. 459–492, https://doi.org/https://doi.org/10.1090/bull/1634.
[3] E. J. Candès and P. Sur, The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression, 2019. to appear Annals of Statistics.
[4] T. M. Chan, An optimal randomized algorithm for maximum Tukey depth, in Proceedings of the Fifteenth Annual ACM-SIAM Symposium on Discrete Algorithms, ACM, New York, 2004, pp. 430–436.
[5] K. L. Clarkson, D. Eppstein, G. Miller, C. Sturtivant, and S. Teng, Approximating center points with iterative Radon points, International Journal of Computational Geometry & Applications, 6 (1996), pp. 357–377.
[6] T. M. Cover,
Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition
, Electronic Computers, IEEE Transactions on, EC-14 (1965), pp. 326 – 334, https://doi.org/10.1109/PGEC.1965.264137.
[7] J. A. De Loera, X. Goaoc, F. Meunier, and N. H. Mustafa, The discrete yet ubiquitous theorems of Carathéodory, Helly, Sperner, Tucker, and Tverberg, Bull. Amer. Math. Soc. (N.S.), 56 (2019), pp. 415–511, https://doi.org/10.1090/bull/1653, https://doi.org/10.1090/bull/1653.
[8] P. Erdős and A. Rényi,
On a classical problem of probability theory
, Magyar Tud. Akad. Mat. Kutató Int. Közl., 6 (1961), pp. 215–220.
[9] R. M. Freund, P. Grigas, and R. Mazumder, Condition Number Analysis of Logistic Regression, and its Implications for Standard First-Order Solution Methods, arXiv e-prints, (2018), arXiv:1810.08727, p. arXiv:1810.08727, https://arxiv.org/abs/1810.08727.
[10] G. James, D. Witten, T. Hastie, and R. Tibshirani, An introduction to statistical learning, vol. 103 of Springer Texts in Statistics, Springer, New York, 2013, https://doi.org/10.1007/978-1-4614-7138-7, https://doi.org/10.1007/978-1-4614-7138-7. With applications in R.
[11] D. G. Larman, On sets projectively equivalent to the vertices of a convex polytope, Bulletin of the London Mathematical Society, 4 (1972), pp. 6–12.
[12] P. McCullagh and J. A. Nelder, Generalized linear models, Monographs on Statistics and Applied Probability, Chapman & Hall, London, 1989, https://doi.org/10.1007/978-1-4899-3242-6, https://doi.org/10.1007/978-1-4899-3242-6. Second edition [of MR0727836].
[13] G. L. Miller and D. R. Sheehy, Approximate centerpoints with proofs, Comput. Geom., 43 (2010), pp. 647–654, https://doi.org/10.1016/j.comgeo.2010.04.006, https://doi.org/10.1016/j.comgeo.2010.04.006.
[14] W. Mulzer and D. Werner, Approximating Tverberg points in linear time for any fixed dimension, Discrete & Computational Geometry, 50 (2013), pp. 520–535.
[15] D. Rolnick and P. Soberón, Algorithmic aspects of Tverberg’s theorem, CoRR, abs/1601.03083 (2016), http://arxiv.org/abs/1601.03083, https://arxiv.org/abs/1601.03083.
[16] P. J. Rousseeuw and A. Struyf, Computing location depth and regression depth in higher dimensions, Statistics and Computing, 8 (1998), pp. 193–203, https://doi.org/10.1023/A:1008945009397, https://doi.org/10.1023/A:1008945009397.
[17] L. Schläfli, Gesammelte mathematische Abhandlungen. Band I, Verlag Birkhäuser, Basel, 1950.
[18] M. J. Silvapulle, On the existence of maximum likelihood estimators for the binomial response models, Journal of the Royal Statistical Society. Series B (Methodological), 43 (1981), pp. 310–313, http://www.jstor.org/stable/2984941.
[19] P. Soberón, Robust Tverberg and colourful Carathéodory results via random choice, Combinatorics, Probability and Computing, 27 (2018), p. 427–440, https://doi.org/10.1017/S0963548317000591.
[20] P. Soberón and R. Strausz, A generalisation of Tverberg’s theorem, Discrete Comput. Geom., 47 (2012), pp. 455–460, https://doi.org/10.1007/s00454-011-9379-z, https://doi.org/10.1007/s00454-011-9379-z.
[21] J. W. Tukey, Mathematics and the picturing of data, Proceedings of the International Congress of Mathematicians, Vancouver, 1975, 2 (1975), pp. 523–531, https://ci.nii.ac.jp/naid/10029477185/en/.
[22] H. Tverberg, A generalization of Radon’s theorem, J. London Math. Soc., 41 (1966), pp. 123–128.
[23] U. Wagner and E. Welzl, A continuous analogue of the upper bound theorem, Discrete Comput. Geom., 26 (2001), pp. 205–219, https://doi.org/10.1145/336154.336176, https://doi.org/10.1145/336154.336176. ACM Symposium on Computational Geometry (Hong Kong, 2000).
[24] J. G. Wendel, A problem in geometric probability., Mathematica Scandinavia, 11 (1962), pp. 109–112, https://doi.org/10.7146/math.scand.a-10655, https://www.mscand.dk/article/view/10655.

Stochastic Tverberg theorems and their applications in multi-class logistic regression, data separability, and centerpoints of data

A note on "MLE in logistic regression with a diverging dimension"

Stochastic DCA for minimizing a large sum of DC functions with application to Multi-class Logistic Regression

Sparse Representations of Positive Functions via Projected Pseudo-Mirror Descent

Bayesian isotonic logistic regression via constrained splines: an application to estimating the serve advantage in professional tennis

DS-MLR: Exploiting Double Separability for Scaling up Distributed Multinomial Logistic Regression

On functional logistic regression via RKHS's

Linear and Fisher Separability of Random Points in the d-dimensional Spherical Layer

1 Introduction

Consequences of our geometric results

Theorem 1 (Stochastic Tverberg theorems applied to multinomial regression)

Theorem 2

2 Our geometric methods: Stochastic Tverberg-type theorems

Theorem (Theorem: (H. Tverberg 1966))

Definition 1

Theorem (Theorem: (Soberón, Strausz 2012))

Prior Stochastic Tverberg theorems

Theorem (Theorem: (T. Cover 1965))

Theorem (Theorem: P. Soberón 2018)

Corollary 1

Theorem 3

Our new stochastic geometric theorems

Theorem 4 (Stochastic Tverberg theorem for equi-partitions)

Theorem 5 (Tverberg Threshold Phenomena for equi-partitions)

Theorem 6 (Stochastic Tverberg with tolerance for equi-partition)

Theorem 7 (Stochastic Radon with tolerance for random allocation)

Corollary 2 (Stochastic Tverberg for random allocation)

3 Proofs of our stochastic results

Proof (Proof of Theorem 2)

Proof (Proof of Theorem 3)

Proof (Proof of the lower bound in Theorem 4)

Proof (Proof of the upper bound in Theorem 4)

Proof (Proof of Theorem 5)

Proof (Proof of Theorem 6)

Proof (Proof of Theorem 7)

Proof (Proof of Corollary 2)

4 Acknowledgments

References

Stochastic Tverberg theorems and their applications in multi-class logistic regression, data separability, and centerpoints of data

Related Research

A note on "MLE in logistic regression with a diverging dimension"

Stochastic DCA for minimizing a large sum of DC functions with application to Multi-class Logistic Regression

Sparse Representations of Positive Functions via Projected Pseudo-Mirror Descent

Bayesian isotonic logistic regression via constrained splines: an application to estimating the serve advantage in professional tennis

DS-MLR: Exploiting Double Separability for Scaling up Distributed Multinomial Logistic Regression

On functional logistic regression via RKHS's

Linear and Fisher Separability of Random Points in the d-dimensional Spherical Layer

1 Introduction

Consequences of our geometric results

Theorem 1 (Stochastic Tverberg theorems applied to multinomial regression)

Theorem 2

2 Our geometric methods: Stochastic Tverberg-type theorems

Theorem (Theorem: (H. Tverberg 1966))

Definition 1

Theorem (Theorem: (Soberón, Strausz 2012))

Prior Stochastic Tverberg theorems

Theorem (Theorem: (T. Cover 1965))

Theorem (Theorem: P. Soberón 2018)

Corollary 1

Theorem 3

Our new stochastic geometric theorems

Theorem 4 (Stochastic Tverberg theorem for equi-partitions)

Theorem 5 (Tverberg Threshold Phenomena for equi-partitions)

Theorem 6 (Stochastic Tverberg with tolerance for equi-partition)

Theorem 7 (Stochastic Radon with tolerance for random allocation)

Corollary 2 (Stochastic Tverberg for random allocation)

3 Proofs of our stochastic results

Proof (Proof of Theorem 2)

Proof (Proof of Theorem 3)

Proof (Proof of the lower bound in Theorem 4)

Proof (Proof of the upper bound in Theorem 4)

Proof (Proof of Theorem 5)

Proof (Proof of Theorem 6)

Proof (Proof of Theorem 7)

Proof (Proof of Corollary 2)

4 Acknowledgments

References