Active Learning in the Overparameterized and Interpolating Regime

1 Overparameterized Models and Active Learning

The success of deep learning systems has sparked interest in understanding how and why overparameterized models that interpolate the training data often display surprisingly good generalization properties

[27, 11, 38, 12, 8, 1, 10, 23]. Notably, it is now understood that mininum norm solutions have the potential to generalize well in the overparameterized, interpolating regime [11, 9, 23, 25]. These findings suggest that the theory of active learning also requires careful reconsideration. This paper introduces a new framework for active learning based on the notion of minimum norm interpolators. We propose a new active learning heuristic for kernel machines inspired by the recent breakthroughs mentioned above.

Active learning algorithms adaptively select examples for labeling based on two general strategies [15]: 1) selecting examples that rule-out as many (incompatible) classifiers as possible at each step; and 2) discovering cluster structure in unlabeled data and labeling representative examples from each cluster. We show that our new active learning approach based on a minimum norm heuristic automatically exploits both these strategies: Figure 1 depicts where samples are selected using the new criterion based on minimum norm interpolators.

Figure 1: Active learning using minimum norm interpolators strategically selects examples for labeling (red points). (a) reduces to binary search in simple 1-d threshold problem setting; (b) labeling is focused near decision boundary in multidimensional setting; (c) automatically discovers clusters and labels representative examples from each.

1.1 Review of Active Learning

Active learning involves selectively labeling examples based on some measure of informativeness and/or label uncertainty; see [34, 35] for a broad overview of the field. Here we discuss the key ideas exploited in active learning and put our contribution in context. We emphasize that this is not a comprehensive literature review.

Perhaps the simplest example of active learning is binary search or bisection to identify a single decision threshold in a one-dimensional classification setting. More general “version space” approaches [13, 3, 31, 22] extend this idea to arbitrary classes. Alternatively, the one-dimensional binary search problem naturally generalizes to the problem of learning a multi-dimensional linear classifier [4, 17, 5] and its kernel-based extensions [37]. These methods are “margin-based,” selecting labeling examples in the vicinity of the learned decision boundary. However, in the overparameterized and interpolating regime, potentially every labeled example is near the learned decision boundary. This calls for a more refined sample selection criterion. Roughly speaking, our new selection criteria favor examples close to the current decision boundary and closest to oppositely labeled examples. This yields optimal bisection sampling in simple low-dimensional settings.

Exploiting the geometric structure of data (e.g., clustering) is also a powerful heuristic for active learning [15]

. Graph-based active learning inspired by semi-supervised learning

[41, 14]

and hierarchical clustering methods

[16] are two common geometric approaches to active learning. Bayesian methods based on notions of information-gain can also exploit distributional structure [18, 20, 24, 26], but require appropriate specifications of prior distributions. The new active learning criteria proposed in this paper are capable of automatically exploiting geometric structure in data, such as clusters, without specifying priors, forming a graph, or clustering.

Inspired by the past theory and methods discussed above, several proposals have been recently made for deep active learning [33, 19, 39, 36]. None of this work, however, directly addresses overparameterized and interpolating active learners.

2 A New Active Learning Criterion

At each iteration of the active learning algorithm, looking at the currently labeled set of samples, a new unlabeled point is selected to be labeled. The criterion we are proposing to pick the samples to be labeled is based on a ‘max-min’ operator. We will describe the criterion in its most general form along with the intuition behind this choice of criterion. In the remaining of the paper, we will go through some theoretical results about the properties of variations of this criterion in various setups along with some additional descriptive numerical evaluations and simulations.

Let $L = {x_{1}, \dots, x_{n}}$ with labels ${y_{1}, \dots, y_{n}}$ be the set of labeled examples so far. We assume binary valued labels, but our approach can be generalized to the multiclass setting. Let $U$ be the set of unlabeled samples. In other words, we have a partially labeled training set. In the interpolating regime, the goal is to correctly label all the points in $U$ so that the training error is zero. Passive learning generally requires labeling every point in $U$ . Active learning sequentially selects points in $U$ for labeling, with the aim of learning a correct classifier without necessarily labeling all of $U$ . Our setting can be viewed as an instance of pool-based active learning.

Let $H$ be a rich collection of functions capable of interpolating the training data. For example, $H$ could be a nonparametric infinite dimensional Reproducing Kernel Hilbert Space (RKHS) or an overparameterized neural network representation. Let $f \in H$ be the minimum norm function that interpolates the labeled examples. Note that $f$ depends on the set $L$ and their labels, but we do not show that dependency in our notation for brevity. Define $f_{t}^{u} (x)$ is the minimum norm interpolating function based on $L$ and the point $u \in U$ with label $t$ . For each $u \in U$ , we select a label $t (u)$ according to one of the following criteria:

t^{(1)} (u) = {a r g m i n}_{t \in {- 1, + 1}} ∥ f_{t}^{u} (x) ∥, t^{(2)} (u) = {\begin{matrix} + 1 if f (u) \geq 0 - 1 if f (u) < 0 \end{matrix}

(1)

The rationale for the definition of $t^{(1)} (u)$ is that operating in the interpolating regime, we select the label that yields the minimum norm interpolant (i.e., the “smoother” of the two interpolants). Intuitively, one could argue that our best guess of the label of $u$ is $t^{(2)} (u)$ (where $f (x)$ is the interpolation function based only on $L$ ). We will show if $f$ is constrained to be an element of an RKHS, then $t^{(1)} (u) = t^{(2)} (u)$ for all $u \in U$ (assuming the Hilbert norm is used in Eq. (1)). In general, the definition $t^{(1)} (u)$ is more flexible since one can choose an arbitrary norm in; e.g., a variety of norms could be considered if $f$ is represented by a neural network. Define $f^{u} (x) = f_{t (u)}^{u} (x) .$ The norm of the minimizer $f^{u} (x)$ or the norm of the difference $f^{u} (x) - f (x)$ are viewed as score functions for the point $u \in U$ :

{s c o r e}^{(1)} (u) = ∥ f^{u} (x) ∥,

{s c o r e}^{(2)} (u) = ∥ f^{u} (x) - f (x) ∥,

(2)

Then, the next point to be labeled by our active learning algorithm is $u^{*} = {a r g m a x}_{u \in U} s c o r e (u) .$ We show in Section 3.2 that if $f$ is an element of RKHS and the Hilbert norm is used, the two above selection criteria are equivalent. These selection criteria are motivated by the aim of interpolating all the data.

Interpolating functions $f^{u}$ with larger scores are less smooth (in terms of norm) or have greater changes from the previous interpolator ( ${s c o r e}^{(1)}$ and ${s c o r e}^{(2)}$ , respectively). The intuition is that attacking the most challenging points in the input space first may eliminate the need to label other “easier” examples later. The distinction between the two definitions of the $s c o r e$ function in Eq. (2) is as follows. Scoring unlabeled points according to the definition ${s c o r e}^{(1)}$ priotorizes labeling the examples which result in minimum norm interpolating functions with largest norm. Since the norm of the function can be associated with its smoothness, roughly speaking, this means that this criteria picks the points which give the least smooth interpolating functions. We expect that these point are the most “informative" of samples with some measure of informativeness. In Section 3 we will show some theoretical results supporting this intuition. Scoring unlabeled points according to the definition ${s c o r e}^{(2)}$ priotorizes labeling the samples which result in minimum norm interpolating functions with largest change. Again, we expect that these point are the most “informative" of samples with some measure of information. In Section 3 we will show some theoretical results supporting this intuition.

We did not specify the norms in Equations (1) and (2) purposefully to have freedom to pick different norms in each of them in different variations of the definition of $s c o r e$ function. If these norms are the same, using ${s c o r e}^{(1)}$ function, $u^{*}$ is chosen such that

∥ f^{u^{*}} (x) ∥ = max u \in U min t \in {- 1, + 1} ∥ f_{t}^{u} (x) ∥ .

This the reason we call this a ‘max-min’ operator.

3 Interpolating Active Learners in an RKHS

In this section, we will focus on minimum norm interpolating functions in a Reproducing Kernel Hilbert Space (RKHS). We present theoretical properties for general RKHS settings, detailed analytical results in the one-dimensional setting, and numerical studies in multiple dimensions. Broadly speaking, we establish the following properties: the proposed score functions

$∙$ tend to select examples near the decision boundary of $f$ , the current interpolator;

$∙$ the score is largest for unlabeled examples near the decision boundary and close to oppositely labeled examples, in effect searching for the boundary in the most likely region of the input space;

$∙$ in one dimension the interpolating active learner coincides with an optimal binary search procedure;

$∙$ using data-based function norms, rather than the RKHS norm, the interpolating active learner executes a tradeoff between sampling near the current decision boundary and sampling in regions far away from currently labeled examples, thus exploiting cluster structure in the data.

3.1 Kernel Methods

A Hilbert space $H$ is associated with an inner product: $⟨ f, g ⟩_{H}$ for $f, g \in H$ . This induces a norm defined by $∥ f ∥_{H} = {\sqrt{⟨ f, f ⟩}}_{H}$ . A symmetric bivariate function $K : X \times X \to R$ is positive semidefinite if for all $n \geq 1$ , and elements ${x_{i}}_{i = 1}^{n}$ the matrix K with element $K_{i, j} = K (x_{i}, x_{j})$ is positive semidefinite (PSD). These functions are called PSD kernel functions. A PSD kernel constructs a Hilbert space, $H$ of functions on $f : X \to R$ . For any $x \in X$ and any $f \in H$ , the function $K (\cdot, x) \in H$ and $⟨ f, K (\cdot, x) ⟩_{H} = f (x)$ .

For $x_{1}, \dots, x_{n} \in X$ and $y_{1}, \dots, y_{n} \in {- 1, + 1}$ , we define $f (x) \in H$ to be the interpolating function (i.e., $f (x_{i}) = y_{i}$ ) with minimum norm (according to the norm associated with $H$ ). Define K to be $n$ by $n$ matrix such that $K_{i, j} = K (x_{i}, x_{j})$ and $y = [y_{1}, \dots, y_{n}]$

. Then, one can show that there exists a vector the vector

α = [α_{1}, \dots, α_{n}]^{T}

such that

f (x)

can be decomposed as

f (x) = n \sum i = 1 α_{i} K (x_{i}, x), where α = K^{- 1} y .

(3)

Using the property $⟨ K (x_{i}, \cdot), K (x_{j}, \cdot) ⟩ = K (x_{i}, x_{j})$ , then $∥ f (x) ∥_{H} = α^{T} K α = y^{T} {% K}^{- 1} y .$

3.2 Properties of General Kernels for Active Learning

We first show that using RKHS for interpolation and Hilbert norm in (1), $t^{(1)} (u) = t^{(2)} (u)$ . We then show that the two score functions in (2) are equivalent in the RKHS setting.

Proposition 1.

For $u \in X$ and $t \in {-, +}$ , define $f (x)$ and $f_{t}^{u} (x)$ according to the definitions in Section 2. Then, $t^{(1)} (u) = t^{(2)} (u)$ .

Proof.

Let ${˜ y}_{t} = [y_{1}, \dots, y_{n}, t]^{T}$ , $a = [K (x_{1}, u), \dots, K (x_{n}, u)]^{T}$ and $b = K (u, u)$ . Let K be the kernel matrix for the elements in $L$ and $˜ K$ be the kernel matrix for the elements in $L \cup {u}$ (Hence, $˜ K$ can be constructed by $K, a$ and $b$ ). Then, for $t \in {- 1, + 1}$

	$∥ f_{t}^{u} (x) ∥$	$= {˜ y}_{t}^{T} ˜ K {˜ y}_{t} (a) = y^{T} (K - \frac{a a^{T}}{b})^{- 1} y - 2 t {y%}^{T} (K - \frac{a a^{T}}{b})^{- 1} \frac{}{a} b + (b - a^{T} K^{- 1} a)^{- 1}$
		$(b) = y^{T} K^{- 1} y + \frac{(1 - t y^{T} K^{- 1} a)^{2}}{b - a^{T} K^{- 1} a} (c) = ∥ f (x) ∥_{H} + \frac{[1 - t f (x)]^{2}}{b - a^{T} K^{- 1} % a} .$

where Schur’s complement formula gives (a) and Woodbury Identity with a bit algebra gives (b). (c) uses (3) for the minimum norm interpolating function. Hence, $∥ f_{+}^{u} (x) ∥ \geq ∥ f_{-}^{u} (x) ∥$ iff $f (x) \leq 0$ . ∎

Proposition 2.

If the norm used in definition of ${s c o r e}^{(1)} (u)$ and ${s c o r e}^{(2)} (u)$ is the Hilbert norm, then ${s c o r e}^{(1)} (u) = {s c o r e}^{(2)} (u) + C$ for all $u \in U$ . Hence, ${s c o r e}^{(1)} (u)$ and ${s c o r e}^{(2)} (u)$ can be used interchangeably.

We omit the proof for brevity. It is based on some simple algebra which shows that for any $u$ and $t \in {- 1, + 1}$ ,

∥ f_{t}^{u} (x) - f (x) ∥_{H} = ∥ f_{t}^{u} (x) ∥_{H} - ∥ f (x) ∥_{H} .

3.3 Laplace Kernel in One Dimension

To develop some intuition, we consider active learning on one-dimension. The sort of target function we have in mind is a multiple threshold classifier. Optimal active learning in this setting coincides with binary search. We now show that the proposed score function with Hilbert norm and the Laplace kernel results in an optimal active learning in one dimension (proof in Appendix A).

Proposition 3.

[Max Min criteria in one dimension] Define $K (x, y) = exp (- | x - y | / h)$ to be the Laplace kernel in one dimension. Then the following statements hold for any value of $h > 0$ :

Let $x_{1}$ and $x_{2}$ be two neighboring labeled points in $L$ . Then for all $x_{1} < u < x_{2}$ $s c o r e (\frac{x_{1} + x_{2}}{2}) \geq s c o r e (u)$ .
Let $x_{1} < x_{2}$ and $x_{3} < x_{4}$ be two pairs of neighboring labeled points such that $x_{2} - x_{1} \geq x_{4} - x_{3}$ , then
- if $y_{1} \neq y_{2}$ and $y_{3} = y_{4}$ . Then $s c o r e (\frac{x_{1} + x_{2}}{2}) \geq s c o r e (\frac{x_{3} + x_{4}}{2})$ .
- if $y_{1} = y_{2}$ and $y_{3} \neq y_{4}$ . Then $s c o r e (\frac{x_{1} + x_{2}}{2}) \leq s c o r e (\frac{x_{3} + x_{4}}{2})$ .
- if $y_{1} \neq y_{2}$ and $y_{3} \neq y_{4}$ . Then $s c o r e (\frac{x_{1} + x_{2}}{2}) \leq s c o r e (\frac{x_{3} + x_{4}}{2})$ .
- if $y_{1} = y_{2}$ and $y_{3} = y_{4}$ . Then $s c o r e (\frac{x_{1} + x_{2}}{2}) \geq s c o r e (\frac{x_{3} + x_{4}}{2})$ .

The key conclusion drawn from these properties is that the midpoints between the closest oppositely labeled neighboring examples have the highest score. If there are no oppositely labeled neighbors, then the score is largest at the midpoint of the largest gap between consecutive samples. Thus, the score results in a binary search for the thresholds definining the classifier. Using the proposition above, it is easy to show the following result, proved in the Appendix B.

Corollary 1.

Consider $N$

points uniformly distributed in the interval

[0, 1]

labeled according to a

k

-piecewise constant function

g (x)

so that

y_{i} = g (x_{i}) \in {- 1, + 1}

and length of the pieces are roughly on the order of

Θ (1 / K)

. Then by running the proposed active learning algorithm with Laplace Kernel and any bandwidth, after

O (k log N)

queries the sign of the resulting interpolant

f

correctly labels all

N

examples (i.e., the training error is zero).

3.4 General Radial-Basis Kernels in One Dimension

For $p > 1$ , we define $ℓ_{p}$ norm of $d$ dimensional vector $x$ to be $∥ x ∥_{ℓ_{p}} = (\sum_{i = 1}^{d} x_{i}^{p})^{1 / p}$ . We define radial basis kernel with parameter $p > 1$ and bandwidth $h$ to be

K_{p} (x, y) = exp (- \frac{1}{h} ∥ x - y ∥_{ℓ_{p}}^{p})

(4)

n the next proposition, we look at the special case of radial basis kernels applied to one dimensional functions with only three initial points. We next show how maximizing ${s c o r e}^{(1)}$ is equivalent to picking the zero-crossing point of our current interpolator.

Proposition 4 (One Dimensional Functions with Radial Basis Kernels).

Assume that for any pair of samples $x, x^{'} \in L$ we have $| x - x^{'} | \geq Δ$ . Assume $Δ h^{- 1 / p} \geq C$ for a constant value of $C$ . Let $x_{1} < x_{2} < x_{3} \in R$ , $y_{1} = y_{2} = + 1$ and $y_{3} = - 1$ . For $u$ such that $x_{2} + Δ / 2 < u < x_{3} - Δ / 2$ , we have ${s c o r e}^{(1)} (u) \leq {s c o r e}^{(1)} (u^{*})$ where $u^{*}$ is the point satisfying $f (u^{*}) = 0$ .

The proof is rather tedious and appears in Appendix C. But the idea is based on showing that with small enough bandwidth, $∥ f_{+}^{u} ∥$ is increasing in $u$ in the interval $[x_{2} + Δ / 2, x_{3} - Δ / 2]$ and $∥ f_{-}^{u} ∥$ is decreasing in $u$ in the same interval. This shows that ${max}_{u} {min}_{t \in {- 1, + 1}} ∥ f_{t}^{u} ∥$ occurs at $u^{*}$ such that $∥ f_{+}^{u^{*}} ∥ = ∥ f_{-}^{u^{*}} ∥$ . We showed that this is equivalent to the condition $f (u^{*}) = 0$ .

3.5 Data-Based Norm

As mentioned above, one can consider norms other than the RKHS norm. One natural option is to consider the $2$ -norm of the output map. Consider

{s c o r e}^{(2)} (u) = ∥ f^{u} (x) - f (x) ∥_{U} := \sqrt{\sum v \in U (f^{u} (v) - f (v))^{2}} .

(5)

Intuitively, this score measures the expected change in the squared norm over all unlabeled examples if $u \in U$ is selected as the next point. This norm is sensitive to the particular distribution of the data, which is important if the data are clustered. This behavior will be demonstrated in the multidimensional setting discussed next.

3.6 Multidimensional Setting

The properties and behavior found in the one dimensional setting carry over to higher dimensions. In particular, the max-min norm criterion tends to select unlabeled examples near the decision boundary and close to oppositely labeled examples, This is illustrated in Figure 2 below. The inputs points (training examples) are uniformly distributed in the square $[- 1, 1] \times [- 1, 1]$ . We trained an Laplace kernel machine to perfectly interpolate four training points with locations and binary labels as depicted in Figure 2(a). The color depicts the magnitude of the learned interpolating function: dark blue is $0$ indicating the “decision boundary” and bright yellow is approximately $3.5$ . Figure 2(b) denotes the score for selecting a point at each location based on RKHS norm criterion. Figure 2(c) denotes the score for selecting a point at each location based on data-based norm criterion discussed above. Both criteria select the point on the decision boundary, but the RKHS norm favors points that are closest to oppositely labeled examples whereas the data-based norm favors points on the boundary further from labeled examples.

Figure 2: Data selection of Laplace kernel active learner. (a) Magnitude of output map kernel machine trained to interpolate four data points as indicated (dark blue is $0$ indicating the learned decision boundary). (b) Max-Min RKHS norm selection of next point to label. Brightest yellow is location of highest score and selected example. (c) Max-Min selection of next point to label using data-based norm. Both select the point on the decision boundary, but the RKHS norm favors points that are closest to oppositely labeled examples.

Next we present a modified scenario in which the examples are not uniformly distributed over the input space, but instead concentrated only in certain regions indicated by the magenta highlights in Figure 3(a). In this setting, the example selection criteria differ more significantly for the two norms. The weight norm selection criterion remains unchanged, but is applied only to regions where there are examples. Areas with out examples to select are indicated by dark blue in Figure 3(b)-(c). The data-based norm is sensitive to the non-uniform input distribution, and it scores examples near the lower portion of the decision boundary highest.

Figure 3: Data selection of Laplace kernel active learner. (a) Unlabeled examples are only available in magenta shaded regions. (b) Max-Min selection map using RKHS norm. (c) Max-Min selection map using data-based norm defined in Equation (5).

The distinction between the max-min selection criterion using the RKHS vs. data-based norm is also apparent in the experiment in which a curved decision boundary in two dimensions is actively learned using a Laplace kernel machine, as depicted in Figure 4 below. ${s c o r e}^{(1)}$ is the max-min RKHS norm criterion at progressive stages of the learning process (from left to right). The data-based norm is used in ${s c o r e}^{(2)}$ defined in Equation (5). Both dramatically outperform a passive (random sampling) scheme and both demonstrate how active learning automatically focuses sampling near the decision boundary between the oppositely labeled data (yellow vs. blue). However, the data-based norm does more exploration away from the decision boundary. As a result, the data-based norm requires slightly more labels to perfectly predict all unlabeled examples, but has a more graceful error decay, as shown on the right of the figure.

Figure 4: Uniform distribution of samples, smooth boundary, Laplace Kernel, Bandwidth $= .1$ . On left, sampling behavior of ${s c o r e}^{(1)}$ and ${s c o r e}^{(2)}$
at progressive stages (left to right). On right, error probabilities as a function of number of labeled examples.

Before moving on to active neural network learners, we demonstrate how the data-based norm also tends to automatically select representive examples from clusters when such structure exists in the unlabeled dataset. Figure 5 compares the behavior of selection based on ${s c o r e}^{(1)}$ with the RKHS norm and ${s c o r e}^{(2)}$ with data-based norm, when data are clusters and each cluster is homogeneously labeled. We see that the data-based norm quickly identifies the clusters and labels a representative from each, leading to faster error decay as shown on the right.

Figure 5: Uniform distribution of samples, smooth boundary, Laplace Kernel, Bandwidth $= .1$ . On left, sampling behavior of ${s c o r e}^{(1)}$ and ${s c o r e}^{(2)}$ at progressive stages (left to right). On right, error probabilities as a function of number of labeled examples.

4 Interpolating Neural Network Active Learners

Here we briefly examine the extension of the max-min criterion and its variants to neural network learners. Neural network complexity or capacity can be controlled by limiting magnitude of the network weights [6, 29, 40]. A number of weight norms and related measures have been recently proposed in the literature [30, 7, 21, 2, 28]

. For example, ReLU networks with a single hidden layer and minimum

ℓ_{2}

norm weights coincide with linear spline interpolation [32]. With this in mind, we provide empirical evidence showing that defining the max-min criterion with the norm of the network weights yields a neural network active learning algorithm with properties analagous to those obtained in the RKHS setting.

Consider a single hidden layer network with ReLU activation units trained using MSE loss. In Figure 6

we show the results of an experiment implemented in PyTorch in the same settings considered above for kernel machines in Figures

2 and 3. We trained an overparameterized network with

100

hidden layer units to perfectly interpolate four training points with locations and binary labels as depicted in Figure 6(a). The color depicts the magnitude of the learned interpolating function: dark blue is

0

indicating the “decision boundary” and bright yellow is approximately

3.5

. Figure 6(b) denotes the

{s c o r e}^{(1)}

with the weight norm (i.e., the

ℓ_{2}

norm of the resulting network weights when a new sample is selected at that location). The brightest yellow indicates the highest score and the location of the next selection. Figure 6(c) denotes the

{s c o r e}^{(2)}

with the data-based norm defined in Section 3.5. In both cases, the max occurs at roughly the same location, which is near the current decision boundary and closest to oppositely labeled points. The data-based norm also places higher scores on points further away from the labeled examples. Thus, the data selection behavior of the neural network is analagous to that of the kernel-based active learner (compare with Figure 2).

Next we present a modified scenario in which the examples are not uniformly distributed over the input space, but instead concentrated only in certain regions indicated by the magenta highlights in Figure 7(a). In this setting, the example selection criteria differ more significantly for the two norms. The weight norm selection criterion remains unchanged, but is applied only to regions where there are examples. Areas without examples to select are indicated by dark blue in Figure 7(b)-(c). The data-based norm is sensitive to the non-uniform input distribution, and it scores examples near the lower portion of the decision boundary highest. Again, this is quite similar to the behavior of the kernel active learner (compare with Figure 3).

Figure 6: Data selection of neural network active learner. (a) Magnitude of output map of single hidden layer ReLU network trained to interpolate four data points as indicated (dark blue is $0$ indicating the learned decision boundary). (b) Max-Min selection of next point to label using network weight norm. (c) Max-Min selection of next point to label using data-based norm. Both select the point on the decision boundary that is closest to oppositely labeled examples.

Figure 7: Data selection of neural network active learner. (a) Unlabeled examples are only available in magenta shaded regions. (b) Max-Min selection map using network weight norm. (c) Max-Min selection map using data-based norm.

5 MNIST Experiment

Here we illustrate the performance of the proposed active learning method on the MNIST dataset. The goal of the classifier is detecting whether an image belongs to the set of numbers greater or equal to $5$ or not. We used Laplace kernel with bandwidth $10$ on the vectorized version of a dataset of $500$ images. Hence, the dimensionality of each sample is $400$ . For comparison, we implemented a passive learning algorithm which selects the points randomly, the active learning algorithm using ${s c o r e}^{(1)}$ with RKHS norm, and the active learning algorithm using ${s c o r e}^{(2)}$ with data-based function norm described in Section 3.5. Figure 8 depicts the decay of probability of error for each of these algorithms. The behavior is similar to the theory and experiments above. Both the RKHS norm and the data-based norm, score $^{(1)}$ and score $^{(2)}$ respectively, dramatically outperform the passive learning baseline and the data-based norm results in somewhat more graceful error decay. Note that the purpose of this experiment is comparing the performance of various selection algorithms. Hence, we are training our classification algorithm on a small subset of the MNIST dataset and that is the reason for nonnegligible error level in the test set.

Figure 8: Probability of error for learning a classification task on MNIST data set. The performance of three selection criteria for labeling the samples: random slection, active selection based on ${s c o r e}^{(1)}$ , and active selection based on ${s c o r e}^{(2)}$ . The left curve depicts the probability of error on the training set and the right curve is the probability of error on the test set.

References

[1] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. arXiv preprint arXiv:1901.08584, 2019.
[2] Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In
International Conference on Machine Learning
, pages 254–263, 2018.
[3] Maria-Florina Balcan, Alina Beygelzimer, and John Langford. Agnostic active learning. Journal of Computer and System Sciences, 75(1):78–89, 2009.
[4] Maria-Florina Balcan, Andrei Broder, and Tong Zhang. Margin based active learning. In
International Conference on Computational Learning Theory
, pages 35–50. Springer, 2007.
[5] Maria-Florina Balcan and Phil Long. Active and passive learning of linear separators under log-concave distributions. In Conference on Learning Theory, pages 288–316, 2013.
[6] Peter L Bartlett. For valid generalization the size of the weights is more important than the size of the network. In Advances in neural information processing systems, pages 134–140, 1997.
[7] Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. In Advances in Neural Information Processing Systems, pages 6240–6249, 2017.
[8] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning and the bias-variance trade-off. arXiv preprint arXiv:1812.11118, 2018.
[9] Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. arXiv preprint arXiv:1903.07571, 2019.
[10] Mikhail Belkin, Daniel J Hsu, and Partha Mitra. Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate. In Advances in Neural Information Processing Systems, pages 2300–2311, 2018.
[11] Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. In International Conference on Machine Learning, pages 540–548, 2018.
[12] Mikhail Belkin, Alexander Rakhlin, and Alexandre B Tsybakov. Does data interpolation contradict statistical optimality? arXiv preprint arXiv:1806.09471, 2018.
[13] David Cohn, Les Atlas, and Richard Ladner. Improving generalization with active learning. Machine learning, 15(2):201–221, 1994.
[14] Gautam Dasarathy, Robert Nowak, and Xiaojin Zhu. S2: An efficient graph based active learning algorithm with application to nonparametric classification. In Conference on Learning Theory, pages 503–522, 2015.
[15] Sanjoy Dasgupta. Two faces of active learning. Theoretical computer science, 412(19):1767–1781, 2011.
[16] Sanjoy Dasgupta and Daniel Hsu. Hierarchical sampling for active learning. In Proceedings of the 25th international conference on Machine learning, pages 208–215. ACM, 2008.
[17] Sanjoy Dasgupta, Adam Tauman Kalai, and Claire Monteleoni.
Analysis of perceptron-based active learning.
In International Conference on Computational Learning Theory, pages 249–263. Springer, 2005.
[18] Yoav Freund, H Sebastian Seung, Eli Shamir, and Naftali Tishby. Selective sampling using the query by committee algorithm. Machine learning, 28(2-3):133–168, 1997.
[19] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1183–1192. JMLR. org, 2017.
[20] Daniel Golovin, Andreas Krause, and Debajyoti Ray. Near-optimal bayesian active learning with noisy observations. In Advances in Neural Information Processing Systems, pages 766–774, 2010.
[21] Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory, pages 297–299, 2018.
[22] Steve Hanneke et al. Theory of disagreement-based active learning. Foundations and Trends® in Machine Learning, 7(2-3):131–309, 2014.
[23] Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. arXiv preprint arXiv:1903.08560, 2019.
[24] Ashish Kapoor, Kristen Grauman, Raquel Urtasun, and Trevor Darrell. Active learning with gaussian processes for object categorization. In
2007 IEEE 11th International Conference on Computer Vision
, pages 1–8. IEEE, 2007.
[25] Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel" ridgeless" regression can generalize. arXiv preprint arXiv:1808.00387, 2018.
[26] Michael Lindenbaum, Shaul Markovitch, and Dmitry Rusakov. Selective sampling for nearest neighbor classifiers. Machine learning, 54(2):125–152, 2004.
[27] Siyuan Ma, Raef Bassily, and Mikhail Belkin. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning, pages 3331–3340, 2018.
[28] Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally-normalized margin bounds for neural networks. arXiv preprint arXiv:1707.09564, 2017.
[29] Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614, 2014.
[30] Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, pages 1376–1401, 2015.
[31] Robert D Nowak. The geometry of generalized binary search. IEEE Transactions on Information Theory, 57(12):7893–7906, 2011.
[32] Pedro Savarese, Itay Evron, Daniel Soudry, and Nathan Srebro. How do infinite width bounded norm networks look in function space? arXiv preprint arXiv:1902.05040, 2019.
[33] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489, 2017.
[34] Burr Settles. Active learning literature survey. Technical report, University of Wisconsin-Madison Department of Computer Sciences, 2009.
[35] Burr Settles. Active learning.
Synthesis Lectures on Artificial Intelligence and Machine Learning
, 6(1):1–114, 2012.
[36] Yanyao Shen, Hyokun Yun, Zachary C Lipton, Yakov Kronrod, and Animashree Anandkumar. Deep active learning for named entity recognition. arXiv preprint arXiv:1707.05928, 2017.
[37] Simon Tong and Daphne Koller. Support vector machine active learning with applications to text classification. Journal of machine learning research, 2(Nov):45–66, 2001.
[38] Sharan Vaswani, Francis Bach, and Mark Schmidt. Fast and faster convergence of sgd for over-parameterized models and an accelerated perceptron. arXiv preprint arXiv:1810.07288, 2018.
[39] Keze Wang, Dongyu Zhang, Ya Li, Ruimao Zhang, and Liang Lin. Cost-effective active learning for deep image classification. IEEE Transactions on Circuits and Systems for Video Technology, 27(12):2591–2600, 2017.
[40] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.
[41] Xiaojin Zhu, John Lafferty, and Zoubin Ghahramani. Combining active learning and semi-supervised learning using gaussian fields and harmonic functions. In ICML 2003 workshop on the continuum from labeled to unlabeled data in machine learning and data mining, volume 3, 2003.

Appendix A Max Min criteria in one dimension with Laplace Kernel

Proof of Proposition 1.

Let us look at the Gram matrix for three neighboring points $x_{1} < x_{2} < x_{3}$ according to the Laplace Kernel.

K

= ⎡ ⎢ ⎣ \begin{matrix} 1 & e^{- (x_{2} - x_{1}) / h} & e^{- (x_{3} - x_{1}) / h} e^{- (x_{2} - x_{1}) / h} & 1 & e^{- (x_{3} - x_{2}) / h} e^{- (x_{3} - x_{1}) / h} & e^{- (x_{3} - x_{2}) / h} & 1 \end{matrix} ⎤ ⎥ ⎦

where we define $A = e^{- (x_{2} - x_{1}) / h}$ and $B = e^{- (x_{3} - x_{2}) / h}$ . It can be shown that with the above structure

K^{- 1}

= ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ \begin{matrix} \frac{1}{1 - A^{2}} & \frac{- A}{1 - A^{2}} & 0 \frac{- A}{1 - A^{2}} & \frac{1}{1 - A^{2}} + \frac{1}{1 - B^{2}} - 1 & \frac{- B}{1 - B^{2}} 0 & \frac{- B}{1 - B^{2}} & \frac{1}{1 - B^{2}} \end{matrix} ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦

In general, if we look into the Gram matrix of $x_{1} < x_{2} < \dots < x_{n}$ , and define $d_{i} = e^{- (x_{i + 1} - x_{i}) / h}$ for $1 \leq i \leq n - 1$ , we observe:

	$(K^{- 1})_{1, 1}$	$= \frac{1}{1 - d_{1}^{2}}, (K^{- 1})_{n, n} = \frac{1}{1 - d_{n - 1}^{2}}$
	$(K^{- 1})_{i, i}$	$= \frac{1}{1 - d_{i - 1}^{2}} + \frac{1}{1 - d_{i}^{2}} - 1, \forall 1 < i < n$
	$(K^{- 1})_{i, i + 1}$	$= (K^{- 1})_{i + 1, i} = \frac{- d_{i}}{1 - d_{i}^{2}}, \forall 1 \leq i < n$

and the remaining elements of matrix of matrix $K^{- 1}$ is zero, i.e., $(K^{- 1})_{i, j}$ for $| j - 1 | \geq 2$ .

Looking at three neighboring points $x_{1} < x_{2} < x_{3}$ labeled $y = {y_{1}, y_{2}, y_{3}}$ we can compute $∥ f (x) ∥^{2}$ :

	$∥ f (x) ∥^{2}$	$= y^{T} K^{- 1} y$
		$= t r a c e (K^{- 1}) + 2 y_{1} y_{2} (K^{- 1})_{1, 2} + 2 y_{2} y_{3} (K^{- 1})_{2, 3}$

Consider two neighboring points $x_{1} < x_{3}$ labeled $y = {-, +}$ . Let $u$ be such that $x_{1} < u < x_{3}$ and $t = y_{3}$ . Define $A = e^{- (u - x_{1}) / h}$ and $B = e^{- (x_{3} - u) / h}$ . Then,

$∥ f_{t}^{u} (x) ∥^{2}$ $= t r a c e (K^{- 1}) + 2 t [(K^{- 1})_{2, 3} - (K^{- 1})_{1, 2}]$

$= t r a c e (K^{- 1}) + 2 t [\frac{A}{1 - A^{2}} - \frac{B}{1 - B^{2}}]$

Hence,

$t (u)$ $= {a r g m i n}_{t} ∥ f_{t}^{u} (x) ∥^{2} = {\begin{matrix} + 1, if u > \frac{x_{1} + x_{3}}{2} or B > A - 1, if u \leq \frac{x_{1} + x_{3}}{2} or B \leq A \end{matrix}$

${s c o r e}^{(1)} (u)$ $= min t ∥ f_{t}^{u} (x) ∥^{2} = t r a c e (K^{- 1}) - 2 ∣ ∣ \frac{A}{1 - A^{2}} - \frac{B}{1 - B^{2}} ∣ ∣$

Without loss of generality, assume $A \geq B$ or $u \leq \frac{x_{1} + x_{3}}{2}$ , then

${s c o r e}^{(1)} (u)$ $= 2 {\frac{1}{1 - A^{2}} + \frac{1}{1 - B^{2}} - \frac{A}{1 - A^{2}} + \frac{B}{1 - B^{2}}} - 1$

$= 2 {\frac{1}{1 + A} + \frac{1}{1 - B}} - 1$

Note that for all $u$ such that $x_{1} < u < x_{3}$ , we have $A B = e^{- (x_{3} - x_{1}) / h} = C$ and is constant. So

$max u \leq \frac{x_{1} + x_{3}}{2} {s c o r e}^{(1)} (u)$ $= max A, B, s.t. A B = C, A \geq B 2 {\frac{1}{1 + A} + \frac{1}{1 - B}} - 1$

$= \frac{4}{1 - C} - 1 = \frac{4}{1 - e^{- (x_{3} - x_{1}) / h}} - 1$

which is attained when $A = B$ or $u = \frac{x_{1} + x_{3}}{2}$ .

This gives the following statement: For neighboring labeled points $x_{1} < x_{3}$ such that $y_{1} \neq y_{3}$ , we have

$max s_{1} \leq u \leq x_{3} {s c o r e}^{(1)} (u) = {s c o r e}^{(1)} (\frac{x_{1} + x_{3}}{2}) = \frac{4}{1 - e^{- (x_{3} - x_{1}) / h}} - 1 \geq 3 .$ (6)

Note that the above function is decreasing in $x_{3} - x_{1}$ .
Consider two neighboring points $x_{1} < x_{3}$ labeled $y = {+, +}$ . Let $u$ such that $x_{1} < u < x_{3}$ and $t = y_{3}$ . Define $A = e^{- (u - x_{1}) / h}$ and $B = e^{- (x_{3} - u) / h}$ .

$max x_{1} < u < x_{3} {s c o r e}^{(1)} (u)$ $= max A, B, s.t. A B = C 2 {\frac{1}{1 + A} + \frac{1}{1 + B}} - 1$

$= \frac{4}{1 + \sqrt{C}} - 1 = \frac{4}{1 + \sqrt{e^{- (x_{3} - x_{1}) / h}}} - 1$

This gives the following statement: For neighboring labeled points $x_{1} < x_{3}$ such that $y_{1} = y_{3}$ , we have

$max s_{1} \leq u \leq x_{3} {s c o r e}^{(1)} (u) = {s c o r e}^{(1)} (\frac{x_{1} + x_{3}}{2}) = \frac{4}{1 + \sqrt{e^{- (x_{3} - x_{1}) / h}}} - 1 < 3 .$ (7)

Note that the above function is increasing in $x_{3} - x_{1}$ .
Equations (6) and (7) give the second part of the proposition.

∎

Appendix B Max Min criteria Binary Search Corollary

Proof of Binary Search Corollary.

Figure 9: Uniform distribution of samples in unit interval, multiple thresholds between $\pm 1$ labels, and active learning using Laplace Kernel, Bandwidth $= .1$ . Probability of error of the interpolated function shown on right.

According to the last property in Proposition 3 the first sample selected will be at the midpoint of the unit interval and the second point will be at $1 / 4$ or $3 / 4$ . If the labels agree, then the next sample will be at the midpoint of the largest subinterval (e.g., at $3 / 4$ if $1 / 4$ was sampled first). Sampling at the midpoints of the largest subinterval between a consecutive pairs labeled points continues until a point with the opposite label is found. Once a point the with opposite label have been found, Proposition 3 implies that subsequent samples repeatedly bisect the subinterval between the closest pair of oppositely labeled points. This bisection process will terminate with two neighboring points having opposite labels, thus identifying one boundary/threshold of $g$ . The total number of labels collected by this bisection process is at most ${log}_{2} N$ . After this, the algorithm alternates between the two situations above. It performs bisection on the subinterval between the close pair of oppositely labeled points, if such an interval exists. If not, it samples at the midpoint of the largest subinterval between a consecutive pairs of labeled points. The stated label complexity bound follows from the assumptions that there are $K$ thresholds and the length of each piece (subinterval between thresholds) is on the order of $1 / K$ . ∎

The bisection process is illustrated experimentally in the Figure 9 below. ${s c o r e}^{(1)}$ uses the RKHS norm. For comparison, we also show the behavior of the algorithm using ${s c o r e}^{(2)}$ and the data-based norm discussed in Section 3.5. Data selection using either score drives the error to zero faster than random sampling (as shown on the left). We clearly see the bisection behavior of ${s c o r e}^{(1)}$ , locating one decision boundary/threshold and then another, as the proof corollary above suggests. Also, we see that the data-based norm does more exploration away from the decision boundaries. As a result, the data-based norm has a faster and more graceful error decay, as shown on the right of the figure. Similar behavior is observed in the multidimensional setting shown in Figure 4.

Appendix C One Dimensional Functions with Radial Basis Kernels

Proof of Proposition 4 on maximum score with radial basis kernels.

For the ease of notation, for fixed $p$ and $h$ , we define $a, b > 0$ as the normalized distance between samples such that $x_{2} - x_{1} = b h^{1 / p}$ and $x_{3} - x_{2} = a h^{1 / p}$ . For $x_{2} < u < x_{3}$ , we define $0 < c < a$ such that the distance between the point $u$ and $x_{2}$ is $u - x_{2} = c h^{1 / p}$ , as in Figure 10. The proposition is based on the assumption that for any pair of points, $| x - x^{'} | \geq Δ$ and $h$ is sufficiently small that $a, b, c, a - c \geq Δ h^{- 1 / p}$ .

We want to show that the max score happens at the zero crossing of function $f (u)$ . Since we normalized all pairwise distances by $h^{1 / p}$ , instead we will show that there exists a constant $D$ such that if $a, b, c, a - c \geq D$ , then the max score is achieved at the zero crossing.

Figure 10: The samples $x_{1}, x_{2}$ and $x_{3}$ are the labeled samples such that $y_{1} = y_{2} = + 1$ and $y_{3} = - 1$ .

Note that $∥ f_{+}^{(u)} ∥$ depends on the location of point $u$ , characterized by the normalized distance between $u$ and $x_{2}$ denoted by $c$ . We want to prove that for small enough bandwidth, $∥ f_{+}^{(u)} ∥$ is increasing in $c$ for $Δ h^{- 1 / p} < c < a - Δ h^{- 1 / p}$ . We can use similar argument to show that $∥ f_{-}^{(u)} ∥$ is decreasing in $c$ . This implies

maxΔh−1/p<c<a−Δh−1/pmint∈{\-,+}∥f(u)t∥=∥f(u∗)+∥=∥f(u∗)−∥

with $u^{*}$ defined to be the point in which $f (u^{*}) = 0$ .

To do so, we will show that $\frac{d}{d c} ∥ f_{+}^{(u)} ∥ > 0$ in this interval. In the proof of Proposition 2, we showed the following form for the function $∥ f_{+}^{(u)} ∥$ ,

∥ f_{+}^{u} (x) ∥ = \frac{[1 - f (u)]^{2}}{b - a_{u}^{T} K^{- 1} a_{u}}

where K is the kernel matrix for the points $x_{1}, x_{2}$ and $x_{3}$ . The vector $a_{v}$ is defined to be $a_{v} = [K (x_{1}, u), K (x_{2}, u), K (x_{3}, u)]^{T}$ . The term $f (u)$ is the minimum norm interpolating function based on the points $x_{1}, x_{2}$ and $x_{3}$ and their labels evaluated at $u$ . Equation 3 shows that

f (u) = y^{T} K^{- 1} a_{u} .

First, we look at K and its inverse in the setup explained above. Using the definition of Radial basis kernels in Equation (4),

K = ⎡ ⎢ ⎣ \begin{matrix} 1 & exp (- b^{p}) & exp [- (a + b)^{p}] exp (- b^{p}) & 1 & exp (- a^{p}) exp [- (a + b)^{p}] & exp (- a^{p}) & 1 \end{matrix} ⎤ ⎥ ⎦

(8)

Hence,

	$K^{- 1}$	$= \frac{1}{\| K \|} ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ \begin{matrix} 1 - e^{- 2 a^{p}} & - e^{- b^{p}} + e^{- a^{p} - (b + a)^{p}} & e^{- a^{p} - b^{p}} - e^{- (a + b)^{p}} - e^{- b^{p}} + e^{- a^{p} - (b + a)^{p}} & 1 - e^{- 2 (a + b)^{p}} & - e^{- a^{p}} + e^{- b^{p} - (a + b)^{p}} e^{- a^{p} - b^{p}} - e^{- (a + b)^{p}} & - e^{- a^{p}} + e^{- b^{p} - (a + b)^{p}} & 1 - e^{- 2 b^{p}} \end{matrix} ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦$
		$= \frac{1}{\| K \|} ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ \begin{matrix} 1 - e^{- 2 a^{p}} & - e^{- b^{p}} [1 - ϵ_{1}] & e^{- a^{p} - b^{p}} [1 - ϵ_{2}] - e^{- b^{p}} [1 - ϵ_{1}] & 1 - e^{- 2 (a + b)^{p}} & - e^{- a^{p}} [1 - ϵ_{3}] e^{- a^{p} - b^{p}} [1 - ϵ_{2}] & - e^{- a^{p}} [1 - ϵ_{3}] & 1 - e^{- 2 b^{p}} \end{matrix} ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦$

The determinant of matrix K is

| K |

= 1 + 2 exp [- a^{p} - b^{p} - (a + b)^{p}] - exp [- 2 (a + b)^{p}] - exp (- 2 a^{p}) - exp (- 2 b^{p})

= 1 - ϵ^{'}

where we defined

	$ϵ^{'}$	$= exp [- 2 (a + b)^{p}] + exp (- 2 a^{p}) + exp (- 2 b^{p}) - 2 exp [- a^{p} - b^{p} - (a + b)^{p}]$
	$ϵ_{1}$	$= exp (b^{p} - a^{p} - (b + a)^{p})$
	$ϵ_{2}$	$= exp (b^{p} + a^{p} - (b + a)^{p})$
	$ϵ_{3}$	$= exp (a^{p} - b^{p} - (b + a)^{p})$

Note that since for $p \geq 1$ , we have $(a + b)^{p} \geq a^{p} + b^{p}$ , then $ϵ^{'}, ϵ_{1}, ϵ_{2}, ϵ_{3} \geq 0$ . Also, there exists a constant $D$ such that if $a, b > D$ , then $| K | \geq 0.9$ .

The vector $a_{v}$ is

a_{v}

= [K (x_{1}, u), K (x_{2}, u), K (x_{3}, u)]^{T} = {[\begin{matrix} exp [- (c + b)^{p}] & exp (- c^{p}) & exp [- (a - c)^{p}] \end{matrix}]}^{T}

Next, we compute $f (u)$

	$f (u)$	$= y^{T} K^{- 1} a_{u}$
		$= \frac{1}{\| K \|} [\begin{matrix} 1 & 1 & - 1 \end{matrix}]$
		$⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ \begin{matrix} 1 - e^{- 2 a^{p}} & - e^{- b^{p}} [1 - ϵ_{1}] & e^{- a^{p} - b^{p}} [1 - ϵ_{2}] - e^{- b^{p}} [1 - ϵ_{1}] & 1 - e^{- 2 (a + b)^{p}} & - e^{- a^{p}} [1 - ϵ_{3}] e^{- a^{p} - b^{p}} [1 - ϵ_{2}] & - e \end{matrix}$

Active Learning in the Overparameterized and Interpolating Regime

Active Learning: Problem Settings and Recent Developments

One-Round Active Learning

Multi-class Active Learning: A Hybrid Informative and Representative Criterion Inspired Approach

Episode-Based Active Learning with Bayesian Neural Networks

Batch Active Learning via Coordinated Matching

Meta-Query-Net: Resolving Purity-Informativeness Dilemma in Open-set Active Learning

Bounded Memory Active Learning through Enriched Queries

1 Overparameterized Models and Active Learning

1.1 Review of Active Learning

2 A New Active Learning Criterion

3 Interpolating Active Learners in an RKHS

3.1 Kernel Methods

3.2 Properties of General Kernels for Active Learning

Proposition 1.

Proof.

Proposition 2.

3.3 Laplace Kernel in One Dimension

Proposition 3.

Corollary 1.

3.4 General Radial-Basis Kernels in One Dimension

Proposition 4 (One Dimensional Functions with Radial Basis Kernels).

3.5 Data-Based Norm

3.6 Multidimensional Setting

4 Interpolating Neural Network Active Learners

5 MNIST Experiment

References

Appendix A Max Min criteria in one dimension with Laplace Kernel

Proof of Proposition 1.

Appendix B Max Min criteria Binary Search Corollary

Proof of Binary Search Corollary.

Appendix C One Dimensional Functions with Radial Basis Kernels

Proof of Proposition 4 on maximum score with radial basis kernels.

	$∥ f_{t}^{u} (x) ∥^{2}$	$= t r a c e (K^{- 1}) + 2 t [(K^{- 1})_{2, 3} - (K^{- 1})_{1, 2}]$
		$= t r a c e (K^{- 1}) + 2 t [\frac{A}{1 - A^{2}} - \frac{B}{1 - B^{2}}]$

	$t (u)$	$= {a r g m i n}_{t} ∥ f_{t}^{u} (x) ∥^{2} = {\begin{matrix} + 1, if u > \frac{x_{1} + x_{3}}{2} or B > A - 1, if u \leq \frac{x_{1} + x_{3}}{2} or B \leq A \end{matrix}$
	${s c o r e}^{(1)} (u)$	$= min t ∥ f_{t}^{u} (x) ∥^{2} = t r a c e (K^{- 1}) - 2 ∣ ∣ \frac{A}{1 - A^{2}} - \frac{B}{1 - B^{2}} ∣ ∣$

	${s c o r e}^{(1)} (u)$	$= 2 {\frac{1}{1 - A^{2}} + \frac{1}{1 - B^{2}} - \frac{A}{1 - A^{2}} + \frac{B}{1 - B^{2}}} - 1$
		$= 2 {\frac{1}{1 + A} + \frac{1}{1 - B}} - 1$

	$max u \leq \frac{x_{1} + x_{3}}{2} {s c o r e}^{(1)} (u)$	$= max A, B, s.t. A B = C, A \geq B 2 {\frac{1}{1 + A} + \frac{1}{1 - B}} - 1$
		$= \frac{4}{1 - C} - 1 = \frac{4}{1 - e^{- (x_{3} - x_{1}) / h}} - 1$

	$max x_{1} < u < x_{3} {s c o r e}^{(1)} (u)$	$= max A, B, s.t. A B = C 2 {\frac{1}{1 + A} + \frac{1}{1 + B}} - 1$
		$= \frac{4}{1 + \sqrt{C}} - 1 = \frac{4}{1 + \sqrt{e^{- (x_{3} - x_{1}) / h}}} - 1$

Active Learning in the Overparameterized and Interpolating Regime

Related Research

Active Learning: Problem Settings and Recent Developments

One-Round Active Learning

Multi-class Active Learning: A Hybrid Informative and Representative Criterion Inspired Approach

Episode-Based Active Learning with Bayesian Neural Networks

Batch Active Learning via Coordinated Matching

Meta-Query-Net: Resolving Purity-Informativeness Dilemma in Open-set Active Learning

Bounded Memory Active Learning through Enriched Queries

1 Overparameterized Models and Active Learning

1.1 Review of Active Learning

2 A New Active Learning Criterion

3 Interpolating Active Learners in an RKHS

3.1 Kernel Methods

3.2 Properties of General Kernels for Active Learning

Proposition 1.

Proof.

Proposition 2.

3.3 Laplace Kernel in One Dimension

Proposition 3.

Corollary 1.

3.4 General Radial-Basis Kernels in One Dimension

Proposition 4 (One Dimensional Functions with Radial Basis Kernels).

3.5 Data-Based Norm

3.6 Multidimensional Setting

4 Interpolating Neural Network Active Learners

5 MNIST Experiment

References

Appendix A Max Min criteria in one dimension with Laplace Kernel

Proof of Proposition 1.

Appendix B Max Min criteria Binary Search Corollary

Proof of Binary Search Corollary.

Appendix C One Dimensional Functions with Radial Basis Kernels

Proof of Proposition 4 on maximum score with radial basis kernels.