Active Learning for Contextual Search with Binary Feedbacks

1 Introduction

Contextual search, which extends the classical binary search problem to high dimensions, finds a wide range of applications, such as auctions, dynamic pricing, and personalized medicine. In the contextual search problem, for each round $i = 1, 2, 3$ , an item (e.g., a customer or a patient) arrives sequentially, each with a contextual vector $x_{i} \in R^{d}$ accessible to the decision-maker. We assume that the context $x_{i}$ incurs an unknown stochastic value $u_{i} = v (x_{i}) + ξ_{i}$ , where $v (x_{i})$ is the mean value function of $x_{i}$ and $ξ_{i}$ is the stochastic noise. The decision-maker selects a query $b_{i} \in R$ and then observes the binary feedback, i.e., whether $u_{i} \geq b_{i}$ or vice versa. The true value $u_{i}$ will never be revealed. To better fit our motivating applications that illustrated below, the decision-maker is allowed to skip making a query to save her budget. Our goal is to learn the mean value function $v (x_{i})$ with a minimum number of queries. We now explain three motivating applications:

First-price Auction: In the Ads industry, many well-known Ad exchange platforms have recently shifted from the second-price auction to the first-price auction, such as ApNexus, Index Exchange, OpenX, and Google Ad Manager (Sluis 2017, Davies 2019). Compared to the second-price auction, the first-price auction has the advantages of enhanced transparency for bidders and potentially increased revenue for the seller. The first-price auction becomes increasingly popular in real-time bidding, which runs online auctions over a large number of demand side platforms.

For an agent to design her bidding strategy, it is critically important to learn the market price given a context of an ad, i.e., the highest bid from her competitors. This problem can be naturally cast as a contextual search problem. Each $x_{i}$ corresponds to a coming bid request¹¹1Each bid request usually contains the auction information, including user information (e.g., location, browser label), publisher information (e.g., web URL and ad slot size), and the ad content., and $u_{i} = v (x_{i}) + ξ_{i}$ corresponds to the unknown market price for this auction. The agent first decides whether she wants to participate in the auction and then posts her bid $b_{i}$ if she joins the auction. In many first-price auction settings in real-time bidding, the agent would only observe the binary feedback on whether she wins the auction but not the market price $u_{i}$ . This problem is referred to as the landscape forecasting problem in the literature (Zhang et al. 2014). Existing research mainly focuses on landscape forecasting under the second-price auction (Wang et al. 2016, Ren et al. 2019). However, the feedback structure of the first-price auction is fundamentally different from that of the second-price auction. In the second price auction, when $b_{i} > u_{i}$ , the winning agent will be able to observe the market price $u_{i}$ as $u_{i}$ is exactly the cost that she needs to pay. In contrast, the winning agent only receives the binary feedback in a standard first-price auction. Moreover, since an agent has the right to avoid making a bid, our contextual search problem setting allows the choice of not making a query to accommodate our motivating application.

Personalized Medicine Experiment: In personalized medicine, a common practice is to leverage a clinical trial experiment to determine the appropriate dosage of a drug for an individual. Bastani and Bayati (2020) adopted a linear bandit model (i.e., the linear form of $v (x)$ ) to investigate the relationship between the optimal dosage and the patients’ profile. We consider a clinical trial experiment for an expensive drug. The profile of each coming potential experimental unit is characterized by $x_{i}$ (e.g., her demographics, diagnosis, medications, and genetics). The algorithm will first decide whether the experimental unit will be given the drug, and if so, recommends a dosage $b_{i}$ . The appropriate level of the dosage is modeled by $u_{i} = v (x_{i}) + ξ_{i}$ . The algorithm receives binary feedback on whether the recommended dosage is above or below the appropriate level. Moreover, we assume that each clinical trial is costly. Thus, the goal is to use the minimum number of trials to learn the ideal personalized dosage level for the drug (i.e., the $v$ function).

Feature-based Pricing Experiment: For an online shopping platform that sells a large number of different products, it is important to understand customers’ valuation for each product. Feature-based pricing models the valuation as a linear function in the product’s feature vector (Qiang and Bayati 2016, Javanmard and Nazerzadeh 2019, Cohen et al. 2020). It would be very costly to conduct price experiments for all products. Thus, based on the feature vector for each product $i$ , the algorithm first decides whether the price experiment should be conducted for this product. If so, a price $b_{i}$ is suggested, and binary purchase decision will be observed subsequently.

Motivated by these applications, the goal of this paper is to propose an efficient algorithm to learn $v (x)$ . Following the existing literature on contextual search and feature-based pricing, we also adopt linear model of the mean valuation function, i.e., $v (x) = ⟨ x, w^{*} ⟩ - μ^{*}$ for some unknown coefficient vector $w^{*} \in R^{d}$ and the intercept $μ^{*} \in R$ . As compared to the existing literature, our contextual search problem has the following unique features, which calls for new algorithmic development:

First, the existing contextual search setup aims to minimize either the absolute loss $| b_{i} - v (x_{i}) |$ or the $ϵ$ -ball loss $I (| b_{i} - v (x_{i}) | > ϵ)$ for some pre-determined $ϵ$ over time. Here $I (\cdot)$ denotes the indicator function. In contrast, we consider a learning problem, where the goal is to learn $v (x)$
as accurately as possible. Therefore, we adopt a probably approximately correct (PAC) setting (see (
2) in Sec. 2) instead of regret minimization setting in existing literature (Lobel et al. 2017, Leme and Schneider 2018, Krishnamurthy et al. 2021). To facilitate the analysis of this learning problem, we assume the stochasticity of the contextual information $x_{i}$ .
Second, as we are motivated by experimental applications, the decision-maker should judge the benefit of a context $x_{i}$ to the learning problem. Therefore, compared to the existing contextual search, our problem has another layer of decision, i.e., whether to conduct a query or not, beyond the decision of the query point itself.

To address this problem, we adopt the active learning framework from machine learning

(Settles 2012). In particular, we adopt the margin-based active learning approach (Balcan et al. 2007). At a high level, let

ˆ v (\cdot)

be the current estimate of the underlying

v (\cdot)

function and

ˆ b

be the query point. For an arriving context

x

, the margin-based active learning will make a query if

| ˆ v (x) - ˆ b |

is sufficiently small, which indicates that it is difficult to determine the relationship between

ˆ b

and

ˆ v (x)

. Although it is an intuitive approach, existing margin-based active learning approach cannot be applied to address our problem due to the existence of the intercept

μ^{*}

. In fact, there is a famous negative result, which shows that active learning cannot significantly improve sample complexity over passive learning for linear binary classification models with intercepts (Dasgupta 2005b). Please refer to Figure 1 in Sec. 1.1 for details.

To address this challenge, we propose an active learning procedure consisting of three major stages:

The first stage of the algorithm is to use trisection search to locate two queries ${ˆ b}_{1}$ and ${ˆ b}_{2}$ that are close to the underlying intercept term $μ^{*}$ , without consuming too many labeled (queried) samples. In this first stage sample selection (i.e., determining whether a sample is to be labeled/queried or not) is not carried out, but the algorithm will actively explore different actions in order to obtain ${ˆ b}_{1}, {ˆ b}_{2}$ that are close to $μ^{*}$ ;
The second stage of the algorithm is to apply margin based active learning to learn the linear model $w^{*}$ and an intercept term depending on both $μ^{*}$ and ${ˆ b}_{1}, {ˆ b}_{2}$ . In this second stage sample selection will be carried out, as only those users with contextual vectors $x_{t}$
close to classification hyperplanes will be queried/labeled (see Algorithm
3 later for details). The actions taken in this stage (on selected samples) will be fixed to either ${ˆ b}_{1}$ or ${ˆ b}_{2}$ obtained in the first stage.

Note that, although this classification model still has non-zero intercept terms, the closeness of ${ˆ b}_{1}, {ˆ b}_{2}$ to $μ^{*}$ would imply that the obtained labels under actions ${ˆ b}_{1}$ or ${ˆ b}_{2}$ are balanced, circumventing the negative results in the work of Dasgupta (2005b) which specifically constructed counter-examples with unbalanced labels. In Figure 1 and the following related work section we give a detailed account of this negative example and how it presents challenges to active learning. Indeed, our theoretical analysis extends the arguments in Balcan et al. (2007) to this more general setting of linear classification with intercepts and balanced labels, with similar convergence rates derived.
The final stage of the algorithm is to reconstruct the mean utility model $ˆ v (\cdot)$ from the estimated linear model and intercepts. Because margin-based active learning can only estimate a linear model up to scales, we need model estimates at two different actions ${ˆ b}_{1}, {ˆ b}_{2}$ (corresponding to two different effective intercepts) in order to reconstruct $w^{*}$ and $μ^{*}$ in $v (\cdot)$ . Details of how this reconstruction is carried out are given in the last two lines of Algorithm 1.

We establish the sample complexity bound for the proposed margin-based active learning with a tri-section search scheme. We assume that with $˜ O (1 / ε^{3})$ total number of incoming contexts, the decision-maker only needs to make $˜ O (1 / ε^{2})$ queries to estimate the mean value function $v (x)$ within $ε$ -precision (with high probability). Here $˜ O$ here hides the dependence on $d$ and other logarithmic factors. We also show that in the passive setting, where the decision-maker is required to conduct queries for all arriving contexts as in the standard contextual search, the sample complexity would be at least $Ω (1 / ε^{4})$ (see Remark 3.3).

1.1 Related work

Our problem setting can be viewed as a variant of the contextual search problem, which is an extension of the classical binary search. In binary search, the decision-maker tries to guess a fixed constant $μ^{*}$ (i.e., the value $u_{i} \equiv μ^{*}$ for all $i$ in our problem) by iteratively making queries $b_{i}$ . In the PAC learning setting, the binary search algorithm only needs $O (log (1 / ε))$ queries to estimate $μ^{*}$ within $ε$ -precision. Due to the importance of applications such as personalized medicine and feature-based pricing, contextual search has received a lot of attention in recent years. The existing literature mainly adopts the linear model for the mean value function. For $ε$ -ball loss $\sum_{i} I (| b_{i} - v (x_{i}) | > ε)$ , Lobel et al. (2017) established the $Ω (d log (1 / ε \sqrt{d}))$ regret lower bound and proposed the project volume algorithm that achieves a near-optimal regret of $O (d log (d / ε))$ . For absolute loss $\sum_{i} | b_{i} - v (x_{i}) |$ , Leme and Schneider (2018) established the regret bound of $O (poly (d))$ . As we explained in the introduction, to fit the applications considered in our paper (e.g., landscape forecasting in the first-price auction), we adopt a PAC learning setting and equip the decision-maker with the ability to pass an incoming context (e.g., skipping an auction). While most contextual search settings in the literature consider adversarial contextual information, we assume the stochasticity of the contextual information as we study a learning problem.

In one of our motivating examples for the first-price auction, there are several recent research works devoted to optimizing the bidding strategy in repeated auctions (Balseiro et al. 2019, Han et al. 2020a, b). These works formulate the problem into contextual bandits with certain monotone structures. In particular, Balseiro et al. (2019) proposed a cross-learning contextual bandits formulation, where at each round, the decision-maker learns rewards associated with all possible contextual information after making a decision. In the context of the first-price auction, Balseiro et al. (2019) modeled the bidder’s own valuation as the context and thus, the bidder would know reward associated with other possible valuation after making the bid. In our setting, the contextual information is the bid request’s profile (e.g., the information of user, publisher, and ad content). Furthermore, the formulation and results of (Balseiro et al. 2019) allow for only a finite number of possible bids, following classical literature on contextual bandits. In contrast, our modeling and algorithm allow the placed bids to vary continuously on the real line to allow for more flexible bidding strategies. Han et al. (2020a, b) formulated the bidding problem into a contextual bandit framework with either the censored feedback structure (i.e., the bidder only knows the winner’s bid) or the full-information feedback structure (i.e., the bidder knows the highest competing bid). Our paper mainly focuses on the landscaping forecasting problem, which facilitates the design of the bidding strategy by providing an accurate estimation of the highest competing bid. Moreover, we consider a more limited (and thus challenging) feedback structure with only binary feedback.

Figure 1: Illustration of negative examples of problem instances constructed in (Dasgupta 2005b).

Active learning is an important research area in machine learning, originating from the seminal work of Cohn et al. (1994) dating back to the 1990s. The main idea behind active learning is to equip the learning algorithm with the ability to select samples or data points to be labeled, improving its sample complexity in applications where labels are expensive to obtain but unlabeled data are abundant. There have been many successful algorithms developed for active learning, such as bisection search for one-dimensional noiseless problems (Dasgupta 2005b), greedy method (Dasgupta 2005a), disagreement-based active learning (Hanneke 2007, Balcan et al. 2009, Zhang and Chaudhuri 2014), margin based active learning (Balcan et al. 2007, Balcan and Long 2013, Wang and Singh 2016)

and active learning based on surrogate loss functions

(Awasthi et al. 2017, Balcan and Zhang 2017). Due to the vast literature on active learning we cannot cite all related works here, and would like to refer interested readers to the excellent review of Hanneke et al. (2014) for an overview of this area.

Our approach in this paper resembles the margin-based active learning method (Balcan et al. 2007, Balcan and Long 2013, Wang and Singh 2016)

which is developed for linear classifiers and have been popular in the active learning literature due to its intuitive nature, tight sample complexity, and relative ease of implementation. However, while linear classifiers seem simple, non-homogeneous linear classifiers (i.e., linear classifiers with

an intercept term) present notorious challenges to active learning algorithms. More specifically, the work of Dasgupta (2005a) shows that when

d \geq 2

and non-homogeneous linear classifiers produce unbalanced samples, such as the example shown on the left panel of Figure 1. In this illustrative example, potential linear classifiers are within

O (ε)

distance to the domain boundary, and thus, active learning cannot asymptotically improve sample complexity over passive learning as it takes

O (1 / ε)

samples to hit the boundaries. Note that, it is easy to verify that, if a non-homogenous linear classifier is within

ϵ

distance to the boundary and the underlying distribution of unlabeled samples is relatively uniform, the probability of seeing a positive sample (as indicated in the region colored by blue in Figure 1) is also on the order of

O (ϵ)

. To overcome this counter-example, in this paper we exploit the special structure in the contextual search problem to “balance” the labels, as shown on the right panel of Figure 1. While the balanced model still possesses a non-zero intercept term, the classifier will be generally

Ω (1)

away from the boundary, which our theoretical analysis shows is sufficient of obtaining desired sample complexity results for active learning.

Active learning has been an important area in machine learning. However, it has not received a lot of attention in operations management. This paper takes a preliminary step on exploring the applications of active learning, and hopefully, it will inspire more research on active learning to address challenges arising from operations management.

1.2 Paper organization

The rest of the paper is organized as follows. Sec. 2 describes the problem formulation and necessary assumptions. Sec. 3 develops our margin-based active learning algorithm with the tri-section search and establishes the sample complexity bound. The technical proofs are provided in Sec. 4. We provide the numerical simulation studies in Sec. 5, followed by the conclusion in Sec. 6. Proofs of some technical lemmas are relegated to the appendix.

2 Problem Formulation and Assumptions

In our modeling, assuming the items (e.g., ads or experimental units) $i = 1, 2, 3, \dots$ arrive sequentially, each with a contextual or feature vector $x_{i} \in R^{d}$ accessible to the decision-maker. We assume that the contextual vectors ${x_{i}}_{i \geq 1}$ are independently and identically distributed with respect to an unknown underlying distribution $P_{X}$ . We also assume that $∥ x_{i} ∥_{2} \leq 1$ for the ease of illustration. Given the contextual vector $x_{i} \in R^{d}$ , the “valuation” of the item (e.g., the highest competing bid in a first-price auction or appropriate dosage in personlized medical treatment) follows a linear model:

u_{i} = v (x_{i}) + ξ_{i} = ⟨ x_{i}, w^{*} ⟩ - μ^{*} + ξ_{i},

(1)

where $v (\cdot) = ⟨ \cdot, w^{*} ⟩ - μ^{*}$ is an underlying linear model with a fixed but unknown coefficient vector $w^{*} \in R^{d}$ , the intercept $μ^{*} \in R$ , and the noise ${ξ_{i}}_{i \geq 1}$ , which are independently and identically distributed stochastic variations with respect to an unknown distribution $P_{ξ}$ .

After observing the contextual vector $x_{i} \in R^{d}$ , the decision-maker will make :

Let the item pass without taking any actions, and thereby without obtaining any feedback/information;
Make a query at $b_{i} \in R$ , and observe the binary feedback $y_{i} = 1$ if $u_{i} \geq b_{i}$ or $y_{i} = - 1$ if $u_{i} < b_{i}$ .

Since making a query (e.g., posting a bid to participate in an auction or admitting an experimental unit into a clinical trail program) incurs much higher implicit cost as compared to passing (i.e., taking no action), the main goal of the decision-maker is to use as fewer number of queries as possible to estimate the mean valuation function $v (\cdot)$ to a certain precision. More specifically, let $ε, δ \in (0, 1)$ be target accuracy and probability parameters. We use $n (ε, δ)$ to denote the number of queries an learning algorithm takes in order to produce an estimate $ˆ v (\cdot)$ that satisfies

sup ∥ x ∥_{2} \leq 1 ∣ ∣ ˆ v (x) - v (x) ∣ ∣ \leq ε, with probability \geq 1 - δ .

(2)

Clearly, the smaller $n (ε, δ)$ is the more efficient the designed learning algorithm is. The main objective of this paper is to design an active learning algorithm that minimizes $n (ε, δ)$ . Additionally, we use $m (ε, δ)$ to denote the number of total samples (i.e., the number of total incoming contexts) an algorithm requires to obtain an estimate $ˆ v$ satisfying Eq. (2). While those incoming contexts skipped by our algorithm usually do not incur extra cost, it is desirable that $m (ε, δ)$ is reasonable because the supply of experimental units might still be limited. In active learning literature, an $m (ε, δ)$ is reasonable if it is a polynomial function in terms of $1 / ε, log (1 / δ)$ and $d$ (Cohn 1996, Cohn et al. 1994, Balcan et al. 2007).

Throughout this paper we impose the following assumptions.

There exists a constant $B < \infty$ such that $∥ w^{*} ∥_{2} \leq B$ and $| μ^{*} | \leq B$ ;
The distribution $P_{X}$ satisfies the following condition: it is supported on the unit $ℓ_{2}$ ball $B_{2} (d) = {x \in R^{d} : ∥ x ∥_{2} \leq 1}$
; it admits a probability density function
$f_{x} (\cdot)$ ; there exist constants $0 < c_{x} \leq C_{x} < \infty$ such that $c_{x} f_{u} (x) \leq f_{x} (x) \leq C_{x} f_{u} (x)$ for all $x \in B_{2} (d)$ , where $f_{u}$
is the probability density function (PDF) of the uniform distribution on
$B_{2} (d)$ ;
The distribution $P_{ξ}$ satisfies the following condition: $Pr [ξ \leq 0] = Pr [ξ \geq 0] = 1 / 2$ ; it admits a probability density function $f_{ξ} (\cdot)$ ; there exist constants $0 < c_{ξ} \leq C_{ξ} < \infty$ such that ${sup}_{ξ \in R} f_{ξ} (ξ) \leq C_{ξ}$ and ${inf}_{| ξ | \leq 2} f_{ξ} (ξ) \geq c_{ξ}$ .

Assumption (A1) is a standard bounded assumption imposed on model parameters. Assumption (A2) assumes that the contextual vectors are independently and identically distributed, with respect to a bounded and non-degenerate distribution $P_{X}$ that is unknown. Similar “non-degenerate” or “covariate diversity” assumptions were also adopted in the contextual learning literature (Bastani and Bayati 2020, Bastani et al. 2021), and the assumption is actually weaker than some of the existing works on active learning (Balcan et al. 2007, Wang and Singh 2016), which requires $P_{X}$ to be the exact uniform distribution over $B_{2} (d)$ .

Assumption (A3) is a general condition imposed on the distribution $P_{ξ}$ of the noise variables. Essentially, it assumes that zero is the median of the noise distribution $P_{ξ}$ , which ensures that the linear classifier is the optimal Bayes classifier. The same assumption is common in the active learning literature (Balcan et al. 2007, Wang and Singh 2016). Note that we do not assume the noise distribution $P_{ξ}$ has any specific parametric forms (e.g., Logistic or Probit noises), making it generally applicable to a broad range of problems.

3 Margin-based Active Learning with Tri-section Search

1:Input: dimension

d

, accuracy parameters

ε, δ

, algorithm parameters

κ_{m}, κ_{n}, κ_{ε}, β_{0}

ˆb1,ˆb2←\textscTrisectionSearch(εs,δs)

with

ε_{s} = 0.1 / \sqrt{d - 1}

δ_{s} = δ / 3

;

3:Let

ε_{a} = κ_{ε} ε^{2} / {ln}^{2} (1 / ε)

δ_{a} = δ / 3

;

(ˆw1,ˆβ1)←\textscMarginBasedActiveLearning(ˆb1,εa,δa,κm,κn,√εa,β0)

;

(ˆw2,ˆβ2)←\textscMarginBasedActiveLearning(ˆb2,εa,δa,κm,κn,√εa,β0)

;

6:Let

ˆ α = ({ˆ b}_{2} - {ˆ b}_{1}) / ({ˆ β}_{2} - {ˆ β}_{1})

;

7:Output: utility function estimate

ˆ v (\cdot) = ⟨ \cdot, ˆ w ⟩ - ˆ μ

, where

ˆ w = ˆ α {ˆ w}_{1}

and

ˆ μ = ˆ α {ˆ β}_{1} - {ˆ b}_{1}

Algorithm 1 A meta-algorithm for actively learning contextual functions.

The main algorithm we proposed for actively learning contextual functions is given in Algorithm 1. The main idea of the proposed algorithm can be summarized as follows.

The first step is to find two actions or bidding prices ${ˆ b}_{1}, {ˆ b}_{2}$ that are reasonably close to the mean utility $μ^{*}$ . This is to ensure that when the actions are fixed at ${ˆ b}_{1}$ or ${ˆ b}_{2}$ , the labels received from user streams are relatively balanced, thereby circumventing the negative results in the work of Dasgupta (2005b). In Sec. 3.1 we show how ${ˆ b}_{1}, {ˆ b}_{2}$ can be found without using too much labeled samples, by using a trisection search idea.

After we obtained candidate bidding prices ${ˆ b}_{1}$ and ${ˆ b}_{2}$ , we use a margin-based active learning algorithm to estimate the linear model $w^{*}$ and mean utility $μ^{*}$ . The margin-based active learning algorithm is similar to the work of Balcan et al. (2007), with the difference being that in our setting the active learning algorithm needs to incorporate a (relatively small) intercept term, which complicates its design and analysis.

Finally, we use the estimates $({ˆ w}_{1}, {ˆ β}_{1})$ and $({ˆ w}_{2}, {ˆ β}_{2})$ obtained from the above-mentioned active learning procedure under two different fixed actions ${ˆ b}_{1}, {ˆ b}_{2}$ to reconstruct the linear utility parameters $w^{*}$ and $μ^{*}$ . The reason we need two fixed actions ${ˆ b}_{1}, {ˆ b}_{2}$ is because the active learning procedure solves a classification problem, for which we can only estimate the linear model and its intercept up to scalings because if one multiply both the linear model and its intercept by a constant the resulting classification problem is the same. Hence, we need two fixed actions ${ˆ b}_{1}, {ˆ b}_{2}$ to construct an approximate linear system of equations, the solution of which would give as consistent estimates of $w^{*}$ and $μ^{*}$ .

Below we briefly explain our intuition behind the construction of the utility function estimate $ˆ v (\cdot)$ in Algorithm 1. For simplicity we will omit the learning errors occurred in the two MarginBasedActiveLearing invocations. Because the margin based active learning algorithm learns linear classifiers up to normalization (see Algorithm 3), we have the following equivalence:

	$⟨ {ˆ w}_{1}, x ⟩ - {ˆ β}_{1} > 0$	$⟺$	$⟨ w^{}, x ⟩ - μ^{} > {ˆ b}_{1};$
	$⟨ {ˆ w}_{2}, x ⟩ - {ˆ β}_{2} > 0$	$⟺$	$⟨ w^{}, x ⟩ - μ^{} > {ˆ b}_{2},$

where $∥ {ˆ w}_{1} ∥_{2} = ∥ {ˆ w}_{2} ∥_{2} = 1$ due to the construction of Algorithm 3. Again, we emphasize that the above equivalence only holds approximately due to learning errors of ${ˆ w}_{1}, {ˆ β}_{1}, {ˆ w}_{2}, {ˆ β}_{2}$ , but we will omit these learning errors for the ease of explanation. Let $α = ∥ w^{*} ∥_{2}$ . We have ${ˆ β}_{1} = (μ^{*} + {ˆ b}_{1}) / α$ and ${ˆ β}_{2} = (μ^{*} + {ˆ b}_{2}) / α$ . Therefore, we set $ˆ α = ({ˆ b}_{2} - {ˆ b}_{1}) / ({ˆ μ}_{2} - ˆ μ_{1})$ as the estimate of $α$ , and $ˆ μ = ˆ α {ˆ β}_{1} - {ˆ b}_{1}$ as the estimate of $ˆ μ$ . Thus, we obtain the utility function estimate $ˆ v (\cdot)$ in Algorithm 1.

3.1 Tri-section search for accurate mean utility

1:function TrisectionSearch(

ε_{s}, δ_{s}

)

2: Initialize: sample counter

n = 0

, lower and upper bounds

{ˆ b}_{1} = - B

{ˆ b}_{2} = B

;

3: while

{ˆ b}_{2} - {ˆ b}_{1} > ε_{s}

{ˆ b}_{3} \leftarrow {ˆ b}_{1} + ({ˆ b}_{2} - {ˆ b}_{1}) / 3

{ˆ b}_{4} \leftarrow {ˆ b}_{2} - ({ˆ b}_{2} - {ˆ b}_{1}) / 3

ˆ n = {ˆ r}_{3} = {ˆ r}_{4} = 0

{p -}_{3} = {p -}_{4} = 0

{¯ ¯ ¯ p}_{3} = {¯ ¯ ¯ p}_{4} = 1

;

5: while

{p -}_{3} \leq 0.5 \leq {¯ ¯ ¯ p}_{3}

and

{p -}_{4} \leq 0.5 \leq {¯ ¯ ¯ p}_{4}

6: For an incoming user

x

, take action

{ˆ b}_{3}

and observe result

y \in {0, 1}

;

7: For another incoming user

x^{'}

, take action

{ˆ b}_{4}

and observe result

y^{'} \in {0, 1}

;

n \leftarrow n + 1

ˆ n \leftarrow ˆ n + 1

{ˆ r}_{3} = {ˆ r}_{3} + 1 {y_{3} = 1}

{ˆ r}_{4} = {ˆ r}_{4} + 1 {y_{4} = 1}

;

9: Update:

[{p -}_{3}, {¯ ¯ ¯ p}_{3}] \leftarrow \frac{{ˆ r}_{3}}{ˆ n} \pm \sqrt{\frac{ln (8 n^{2} / δ_{s})}{2 ˆ n}}]

and

[{p -}_{4}, {¯ ¯ ¯ p}_{4}] \leftarrow \frac{{ˆ r}_{4}}{ˆ n} \pm \sqrt{\frac{ln (8 n^{2} / δ_{s})}{2 ˆ n}}]

;

10: end while

11: Set

{ˆ b}_{1} \leftarrow {ˆ b}_{3}

{p -}_{3} > 0.5

{p -}_{4} > 0.5

and

{ˆ b}_{2} \leftarrow {ˆ b}_{4}

otherwise;

12: end while

13: return

({ˆ b}_{1}, {ˆ b}_{2})

14:end function

Algorithm 2 A tri-section search algorithm to roughly estimate the mean utility parameter

μ^{*}

Let $b^{*} \in R$ be the unique value such that ${Pr}_{x \sim P_{X}} [v (x) \geq b^{*}] = 1 / 2$ . Because $P_{X}$ and $P_{ξ}$ have PDFs, such a value of $b^{*}$ exists and is unique. Intuitively, if one commits to the fixed action $b^{*}$ then the labels received by the algorithm should be balanced. Algorithm 2 shows how to find actions ${ˆ b}_{1}, {ˆ b}_{2}$ that are reasonably close to $b^{*}$ , without consuming too many labeled samples.

Figure 2: Graphical illustration of the main idea behind Algorithm 2.

The main idea behind Algorithm 2 is a trisection search approach, motivated by the fact that the probability ${Pr}_{x \sim P_{X}} [v (x) \geq b]$ is a monotonically decreasing function of $b$ , and furthermore as $| b - b^{*} |$ increases the gap between ${Pr}_{x \sim P_{X}} [v (x) \geq b]$ and ${Pr}_{x \sim P_{X}} [v (x) \geq b^{*}] = 1 / 2$ will also increase (see, e.g., Lemma 4.1.2 in the proof). This allows us to use a trisection search procedure to localize the value of $b^{*}$ , by simply comparing an empirical estimate of ${Pr}_{x \in P_{X}} [v (x) \geq b]$ at the current value of $b$ . More specifically, at an iteration ${ˆ b}_{3}, {ˆ b}_{4}$ are the two midpoints and $[{p -}_{3}, {¯ ¯ ¯ p}_{3}]$ are lower and upper estimates of ${Pr}_{x \sim P_{X}} [v (x) \geq {ˆ b}_{3}]$ and similarly $[{p -}_{4}, {¯ ¯ ¯ p}_{4}]$ are lower and upper estimates for ${Pr}_{x \sim P_{X}} [v (x) \geq {ˆ b}_{4}]$ . With either probabilities being separated from $1 / 2$ , the algorithm could move ${ˆ b}_{1}$ or ${ˆ b}_{2}$ to ${ˆ b}_{3}$ or ${ˆ b}_{4}$ . The algorithm is guaranteed to maintain that $b^{*} \in [{ˆ p}_{1}, {ˆ p}_{2}]$ , thanks to the monotonicity of ${Pr}_{x \sim P_{X}} [v (x) \geq b]$ with respect to $b$ .

The following technical lemmas are the main results explaining the objective and guarantee of Algorithm 2, which are proved in Sec. 4.1. Suppose $d \geq 2$ and let $β^{*} = b^{*} + μ^{*}$ . Then $\frac{| β^{*} |}{∥ w^{*} ∥_{2}} \leq \sqrt{\frac{2 ln (100 C_{x} C_{ξ} / c_{x} c_{ξ})}{d - 1}} = O (1 / \sqrt{d})$ .

Suppose $d \geq 2$ and let $({ˆ b}_{1}, {ˆ b}_{2})$ be the values returned by $\textscBisectionSearch(εs,δs)$ . With probability $1 - δ_{s}$ the following hold: ${ˆ b}_{1} \leq b^{*} \leq {ˆ b}_{2}$ , and at most $O (ε_{s}^{- 2} log (1 / δ_{s} ε_{s})) = O (d log (d / δ))$ queried samples are consumed.

Intuitively, Lemma 2 establishes that the “balancing” intercept $b^{*}$ is $O (1 / \sqrt{d})$ close to the intercept $μ^{*}$ in the utility model, which is helpful for our later analysis. Lemma 2 further establishes that the returned two actions ${ˆ b}_{1}, {ˆ b}_{2}$ sandwich the “label-balancing” action $b^{*}$ , and also upper bound the total number of labeled (queried) samples consumed in the algorithmic procedure.

3.2 Margin-based Active Learning

In Algorithm 3 we provide the pseudocode description of the margin based active learning algorithm we use in this problem to actively learn a linear model with intercepts.

1:function MarginBasedActiveLearning(

b, ε_{a}, δ_{a}, κ_{m}, κ_{n}, ϵ_{0}, β_{0}

)

2: Collect

n_{0} = ⌈ κ_{n} / ϵ_{0}^{2} ⌉

samples with action

b

and let

D_{0} = {(x, y)} \subseteq B_{d} (2) \times {\pm 1}

| D_{0} | = n_{0}

be the queried samples;

3: Let

{ˆ w}_{0}, {ˆ β}_{0} \leftarrow arg {min}_{∥ w ∥_{2} = 1, | β | \leq β_{0}} \sum_{(x, y) \in D_{0}} 1 {y \neq s g n (⟨ x, w ⟩ - β)}

;

4: Let

k_{0} = min {k \in N : 2^{- k} ϵ_{0} \leq ε_{a}}

;

5: for

k = 1, 2, \dots, k_{0}

ϵ_{k} \leftarrow 2^{- k} ϵ_{0}

m_{k} \leftarrow κ_{m} \sqrt{ϵ_{k}}

n_{k} \leftarrow ⌈ κ_{n} d / ϵ_{k} ⌉

D_{k} = \emptyset

;

7: while

| D_{k} | < n_{k}

8: Observe context vector

x \in R^{d}

for the next object;

9: if

| ⟨ x, {ˆ w}_{k - 1} ⟩ - {ˆ β}_{k - 1} | \leq m_{k}

then

10: Invoke action

b

and let

y \in {\pm 1}

be the collected binary feedback;

11: Update

D_{k} \leftarrow D_{k} \cup {x, y}

;

12: end if

13: end while

14:

{ˆ w}_{k}, {ˆ β}_{k} \leftarrow arg {min}_{∥ w ∥_{2} = 1, | β | \leq β_{0}} \sum_{(x, y) \in D_{k}} 1 {y \neq s g n (⟨ x, w ⟩ - β)}

;

15: end for

16: return

{ˆ w}_{k_{0}}, {ˆ β}_{k_{0}}

17:end function

Algorithm 3 Margin-based Active Learning Non-homogeneous Linear Classifiers

Note that in Algorithm 3 the query point $b$ is fixed, with the algorithm only able to select which sample/contextual vector to act upon. Since the query point $b$ is fixed, we can consider linear models with intercepts as $ˆ v (\cdot) = ⟨ \cdot, ˆ w ⟩ - ˆ β$ . For such a model, we define the error of $ˆ v$ under the query point $b$ as

(3)

where $v (x) = ⟨ x, w^{*} ⟩ - μ^{*}$ . Note that for any $b \in R$ , the model $v_{b}^{*} (\cdot) := ⟨ \cdot, w^{*} ⟩ - μ^{*} - b$ has the smallest error defined in Eq. (3), This is because $v_{b}^{*} (\cdot)$ is the Bayes classifier; that is, $v_{b}^{*} (x) \geq 0$ if and only if $Pr [v (x) + ξ \geq b | x] \geq 1 / 2$ . Hence, we can also define the excess error of a model $ˆ v (\cdot)$ as

Δ {e r r}_{b} (ˆ v) := {e r r}_{b} (ˆ v) - {e r r}_{b} (v_{b}^{*}) .

(4)

Figure 3: Graphical illustration of the main idea of Algorithm 3.

Figure 3 illustrates the principles of Algorithm 3. The main idea of Algorithm 3

is simple: the algorithm first uses a “warm-up” epoch consisting of

n_{0}

queried samples to construct a preliminary model estimate

{ˆ w}_{0}

and

{ˆ β}_{0}

. There is no sample selection or active learning in this warm-up procedure, and the analysis of excess errors of

{ˆ w}_{0}, {ˆ β}_{0}

follow standard VC theory analyzing empirical risk minimizers of binary classifiers (see e.g. Lemma A in the proof and also Balcan et al. (2007), Vapnik and Chervonenkis (2015), Vapnik (2013)). Next, in each epoch the algorithm only take action

b

for those users with contextual vectors that are close to the current classification hyperplane (i.e., those users with small “margin”

| ⟨ x, {ˆ w}_{k - 1} ⟩ - {ˆ β}_{k - 1} |

). This concentrates our labeled/queried samples to the region that are close to the classification hyperplane, which helps reduce the number of queried samples as the queried samples are collected on regions that are the most uncertain from a binary classification perspective.

The following lemma is the main result of this section, which is proved in Sec. 4.2. Let $(ˆ w, ˆ β)$ be returned by Algorithm 3 with parameters satisfying $| μ^{*} - b | / ∥ w^{*} ∥_{2} \leq β_{0} = O (1 / \sqrt{d})$ , $κ_{m} = Ω (1)$ , $κ_{n} = Ω (d + log log (1 / ε_{a}) + log (1 / δ_{a}))$ and $ϵ_{0} = \sqrt{ε_{a}}$ . Let $ˆ v (\cdot) = ⟨ \cdot, ˆ w ⟩ - ˆ β$ . Then for sufficiently large $d$ and sufficiently small $ε_{a}$ , with probability $1 - δ_{a}$ the following hold:

$Δ e r r (ˆ v) \leq ε_{a}$ ;
Algorithm 3 consumes $O (κ_{n} d / ε_{a})$ queried samples and $O (κ_{n} \sqrt{d} e^{d ε_{a}} / ε_{a}^{3 / 2})$ total samples.

Essentially, Lemma 3.2 shows that the estimated linear model $ˆ v (\cdot)$ produced by Algorithm 3 has the target excess risk $ε_{a}$ with high probability. The lemma also upper bounds the number of queried and total samples consumed in the estimation procedure. As we can see, the number of labeled samples required is on the order of $O (1 / ε_{a})$ , which is an order of magnitude fewer than the total number of samples consumed (on the order of $O (1 / ε^{3 / 2})$ ). This shows that the active learning procedure is capable of drastically reducing the number of queried samples required to attain an accurate model estimate $ˆ v$ , by being selective in the user context vectors.

3.3 Sample complexity analysis of Algorithm 1

In this section we establish the following theorem, which analyzes the sample complexity (both samples that are queried on and samples that are passed) of Algorithm 1, and provides guidance on the selection of the algorithm input parameters.

Suppose Algorithm 1 is executed with $κ_{m} ≍ 1$ , $κ_{n} ≍ d + log log (1 / ε) + log (1 / δ)$ , $κ_{ε} ≍ 1 / d$ and $β_{0} ≍ 1 / \sqrt{d}$ . Then for sufficiently small $ε > 0$ and sufficiently large $d$ , with probability $1 - δ$ it holds that $| ˆ v (x) - v (x) | \leq ε$ for all $x \in B_{2} (d)$ . Furthermore, the algorithm makes $n (δ, ε)$ queries among $m (δ, ε)$ total samples/contexts, with

	$n (δ, ε)$	$=$	$O (\frac{d^{3} log (d log (d ε^{- 1}) / δ) {log}^{2} (1 / ε)}{ε^{2}}),$
	$m (δ, ε)$	$=$	$O (\frac{d^{3} log (d log (d ε^{- 1}) / δ) {log}^{3} (1 / ε)}{ε^{3}}) .$

If the decision-maker needs to make queries to all incoming contexts/samples (i.e., skipping uninformative samples is not allowed), then at least $Ω (d / ε^{4})$ samples are required. To see this, note that the standard classification theory establishes that $Ω (d / ε^{2})$ samples are needed to obtain a linear classifier $ˆ w$ such that $Pr [s g n ({ˆ w}^{⊤} x) \neq s g n ((w^{*})^{⊤} x)] \leq ε$ (see, e.g., Mammen and Tsybakov (1999), Ben-David and Urner (2014)). On the other hand, it can be shown via an integration argument as follows. Let $∠ (ˆ w, w^{*})$ denote the angle between $ˆ w$ and $w^{*}$ . If both $ˆ w, w^{*}$ are normalized (i.e., $∥ ˆ w ∥_{2} = ∥ w^{*} ∥_{2} = 1$ ) and $∥ ˆ w - w^{*} ∥_{2} \approx ∠ (ˆ w, w^{*}) \approx ε$ then ${Pr}_{x} [s g n ({ˆ w}^{⊤} x) \neq s g n ((w^{*})^{⊤} x)] \approx ε^{2}$ . This shows that in order to achieve $| ˆ v (\cdot) - v (\cdot) | \leq ε$ we must have ${Pr}_{x} [s g n ({ˆ w}^{⊤} x) \neq s g n ((w^{*})^{⊤} x)] ≲ ε^{2}$ , indicating a sample complexity lower bound of $Ω (d / (ε^{2})^{2}) = Ω (d / ε^{4})$ .

Theorem 3.3 shows that, by using more unlabeled/unqueried samples than those that are labeled (more specifically, $1 / ε^{3}$ total samples and $1 / ε^{2}$ labeled ones), the utility function estimate $ˆ v (\cdot)$ produced by our active learning algorithm is within $ε$ estimation error with high probability. In Sec. 5 of numerical studies, we will see that the availability of unlabeled samples will greatly improve the estimation accuracy of an active learning algorithm, compared to a passive learning baseline which cannot skip or select samples to query.

4 Technical Proofs

In this section we state the proofs of the main results in this paper. There are also some technical lemmas that either easy to prove, or cited/rephrased from existing works, which will be presented in the appendix. For simplicity, let $P_{U}$ be the uniform distribution on $B_{2} (d) = {x \in R^{d} : ∥ x ∥_{2} \leq 1}$ for all proofs in this section.

4.1 Proof of results in Sec. 3.1

4.1.1 Proof of Lemma 2.

First note that $⟨ w^{*}, x ⟩ - μ^{*} \geq b^{*}$ is equivalent to $⟨ w^{*}, x ⟩ - β^{*} \geq 0$ , with $β^{*} = μ^{*} + b^{*}$ . Note also that we may assume $∥ w^{*} ∥_{2} = 1$ because $| β^{*} | / ∥ w^{*} ∥_{2}$ is invariant to $∥ w^{*} ∥_{2}$ . In this proof we shall use the lower and upper bounds of $f_{x}$ by connecting it with the uniform distribution on $B_{2} (d)$ , $P_{U}$ . Because $P_{U}$ is isotropic, we may assume without loss of generality that $w^{*} = (1, 0, \dots, 0)$ and $β^{*} \geq 0$ . We will also abbreviate $η = η_{b^{*}}$ and $Δ = Δ_{b^{*}}$ since all margins in this proof are with respect to $b^{*}$ . Then for all $x \in B_{2} (d)$ with $x_{1} \geq β^{*}$ , $η (x) \geq 1 / 2$ and further more

η (x) - \frac{1}{2} = ϕ (x_{1} - β^{*}) = \int_{0}^{x_{1} - β^{*}} ϕ^{'} (u) d u \leq C_{ξ} (x_{1} - β^{*}) .

Subsequently, by Assumption (A2) and Lemma A, it holds that

	$\int_{x_{1} \geq β^{*}}$	$(η (x) - \frac{1}{2}) d P_{x} (x) \leq C_{x} \int_{x_{1} \geq β^{}} (η (x) - \frac{1}{2}) d P_{U} (x) \leq C_{x} C_{ξ} \int_{x_{1} \geq β^{}} (x_{1} - β^{*}) d P_{U} (x)$
		$\leq C_{x} C_{ξ} \int_{0}^{1} \sqrt{\frac{d + 1}{2 π}} e^{- (d - 1) (β^{} + γ)^{2} / 2} γ d γ \leq C_{x} C_{ξ} \sqrt{d} e^{- (d - 1) (β^{})^{2} / 2} \int_{0}^{1} γ e^{- (d - 1) γ^{2} / 2} d γ .$		(5)

With $γ \mapsto γ / \sqrt{d - 1}$ , we have $\int_{0}^{1} γ e^{- (d - 1) γ^{2} / 2} d γ \leq \sqrt{\frac{2 π}{} d - 1} E_{z \sim N (0, 1 / (d - 1))} [| z |] / 2 \leq 1 / (d - 1)$ . Noting that $\sqrt{d} \leq \sqrt{2 (d - 1)}$ for $d \geq 2$ , Eq. (5) can then be simplified to

\int_{x_{1} \geq β^{*}} (η (x) - \frac{1}{2}) d P_{x} (x) \leq \frac{\sqrt{2} C_{x} C_{ξ}}{\sqrt{d - 1}} e^{- (d - 1) (β^{*})^{2} / 2} .

(6)

On the other hand, for all $x \in B_{2} (d)$ with $x_{1} \leq β^{*}$ , $η (x) \leq 1 / 2$ and furthermore

\frac{1}{2} - η (x) = - ϕ (x_{1} - β^{*}) = \int_{0}^{β^{*} - x_{1}} ϕ^{'} (u) d u \geq c_{ξ} (β^{*} - x_{1}) .

Subsequently, by Assumption (A2) and Lemma A, it holds that

$\int_{x_{1} \leq β^{*}}$	$(\frac{1}{2} - η (x)) d P_{x} (x) \geq c_{x} \int_{x_{1} \leq 0} (\frac{1}{2} - η (x)) d P_{U} (x) \geq c_{x} c_{ξ} \int_{x_{1} \leq 0} (β^{*} - x_{1}) d P_{U} (x)$
	$\geq c_{x} c_{ξ} \int_{x_{1} \leq 0} - x_{1} d P_{U} (x) \geq c_{x} c_{ξ} \int_{0}^{1} \sqrt{\frac{d + 1}{16 π}} e^{- (d - 1) γ^{2} / 2} γ d γ \geq \frac{c_{x} c_{ξ} \sqrt{d}}{4 \sqrt{π}} \int_{1 / \sqrt{d - 1}}^{\sqrt{2} / \sqrt{d - 1}} e^{- (d - 1) γ^{2} / 2} γ d γ$
	$\geq \frac{c_{x} c_{ξ} \sqrt{d}}{4 \sqrt{π}} \times \frac{1}{e \sqrt{} d - 1} \times \frac{\sqrt{2} - 1}{\sqrt{d - 1}} \geq \frac{(\sqrt{2} - 1) c_{x} c_{ξ}}{4 e \sqrt{π (d - 1)}} .$	(7)

Combining Eqs. (5,7) we obtain

\frac{1}{2} = Pr x \sim P_{x} [y = 1 | b^{*}] \leq \frac{1}{2} + \frac{\sqrt{2} C_{x} C_{ξ}}{\sqrt{d - 1}} e^{- (d - 1) (β^{*})^{2} / 2} - \frac{(\sqrt{2} - 1) c_{x} c_{ξ}}{4 e \sqrt{π (d - 1)}} .

To satisfy the above inequality, $β^{*} \geq 0$ must satisfy

β^{*} \leq \sqrt{\frac{2 ln (100 C_{x} C_{ξ} / c_{x} c_{ξ})}{d - 1}} = O (1 / \sqrt{d}),

which proves Lemma 2.

4.1.2 Proof of Lemma 2.

For notational simplicity define $℘ (ˆ b) := {Pr}_{x \sim P_{x}} [v (x) \geq ˆ b]$ . Clearly, $℘ (b^{*}) = 1 / 2$ and $℘ (\cdot)$ is a monotonically decreasing function. By Hölder inequality, at sample $n$ we have $Pr [℘ ({ˆ b}_{3}) \in [{p -}_{3}, {¯ ¯ ¯ p}_{3}]] \geq 1 - 2 e^{- 2 ˆ n \times ln (8 n^{2} / δ_{s}) / (2 ˆ n)} \geq 1 - \frac{δ_{s}}{4 n^{2}}$ . The same inequality holds for $Pr [℘ ({ˆ b}_{4}) \in [{p -}_{4}, {¯ ¯ ¯ p}_{4}]]$ as well. By the union bound, the probability that $℘ ({ˆ b}_{3}) \in [{p -}_{3}, {¯ ¯ ¯ p}_{3}]$ and $℘ ({ˆ b}_{4}) \in [{p -}_{4}, {¯ ¯ ¯ p}_{4}]$ throughout the entire Algorithm 2 is lower bounded by

1 - \sum n \geq 1 2 \times \frac{δ_{s}}{4 n^{2}} = 1 - \frac{δ_{s}}{2} \sum n \geq 1 \frac{1}{n}

Active Learning for Contextual Search with Binary Feedbacks

Authors

Pricing Query Complexity of Revenue Maximization

The Power of Comparisons for Actively Learning Linear Classifiers

Learning Reserve Prices in Second-Price Auctions

Active Structure Learning of Bayesian Networks in an Observational Setting

The Geometry of Generalized Binary Search

HS^2: Active Learning over Hypergraphs

Near-Optimal Active Learning of Halfspaces via Query Synthesis in the Noisy Setting

1 Introduction

1.1 Related work

1.2 Paper organization

2 Problem Formulation and Assumptions

3 Margin-based Active Learning with Tri-section Search

3.1 Tri-section search for accurate mean utility

3.2 Margin-based Active Learning

3.3 Sample complexity analysis of Algorithm 1

4 Technical Proofs

4.1 Proof of results in Sec. 3.1

4.1.1 Proof of Lemma 2.

4.1.2 Proof of Lemma 2.

Active Learning for Contextual Search with Binary Feedbacks

Authors

Pricing Query Complexity of Revenue Maximization

The Power of Comparisons for Actively Learning Linear Classifiers

Learning Reserve Prices in Second-Price Auctions

Active Structure Learning of Bayesian Networks in an Observational Setting

The Geometry of Generalized Binary Search

HS^2: Active Learning over Hypergraphs

Near-Optimal Active Learning of Halfspaces via Query Synthesis in the Noisy Setting

This week in AI

1 Introduction

1.1 Related work

1.2 Paper organization

2 Problem Formulation and Assumptions

3 Margin-based Active Learning with Tri-section Search

3.1 Tri-section search for accurate mean utility

3.2 Margin-based Active Learning

3.3 Sample complexity analysis of Algorithm 1

4 Technical Proofs

4.1 Proof of results in Sec. 3.1

4.1.1 Proof of Lemma 2.

4.1.2 Proof of Lemma 2.