Top-k Multiclass SVM

1 Introduction

Figure 1: Images from SUN 397 [32] illustrating class ambiguity. Top: (left to right) Park, River, Pond. Bottom: Park, Campus, Picnic area.

As the number of classes increases, two important issues emerge: class overlap and multi-label nature of examples [10]

. This phenomenon asks for adjustments of both the evaluation metrics as well as the loss functions employed. When a predictor is allowed

k

guesses and is not penalized for

k - 1

mistakes, such an evaluation measure is known as top-

k

error. We argue that this is an important metric that will inevitably receive more attention in the future as the illustration in Figure 1 indicates.

How obvious is it that each row of Figure 1 shows examples of different

classes? Can we imagine a human to predict correctly on the first attempt? Does it even make sense to penalize a learning system for such “mistakes”? While the problem of class ambiguity is apparent in computer vision, similar problems arise in other domains when the number of classes becomes large.

We propose top- $k$ multiclass SVM as a generalization of the well-known multiclass SVM [6]. It is based on a tight convex upper bound of the top- $k$ zero-one loss which we call top- $k$ hinge loss. While it turns out to be similar to a top- $k$ version of the ranking based loss proposed by [30], we show that the top- $k$ hinge loss is a lower bound on their version and is thus a tighter bound on the top- $k$ zero-one loss. We propose an efficient implementation based on stochastic dual coordinate ascent (SDCA) [27]. A key ingredient in the optimization is the (biased) projection onto the top- $k$ simplex. This projection turns out to be a tricky generalization of the continuous quadratic knapsack problem, respectively the projection onto the standard simplex. The proposed algorithm for solving it has complexity $O (m log m)$ for $x∈\Rbm$ . Our implementation of the top- $k$ multiclass SVM scales to large datasets like Places 205 with about $2.5$ million examples and $205$ classes [33]. Finally, extensive experiments on several challenging computer vision problems show that top- $k$ multiclass SVM consistently improves in top- $k$ error over the multiclass SVM (equivalent to our top- $1$ multiclass SVM), one-vs-all SVM and other methods based on different ranking losses [12, 17].

2 Top- $k$ Loss in Multiclass Classification

In multiclass classification, one is given a set $S={(xi,yi)\giveni=1,…,n}$ of $n$ training examples $xi∈\Xc$ along with the corresponding labels $yi∈\Yc$ . Let $\Xc=\Rbd$ be the feature space and $\Yc={1,…,m}$ the set of labels. The task is to learn a set of $m$ linear predictors $wy∈\Rbd$ such that the risk of the classifier $\argmaxy∈\Yc\innerwy,x$ is minimized for a given loss function, which is usually chosen to be a convex upper bound of the zero-one loss. The generalization to nonlinear predictors using kernels is discussed below.

The classification problem becomes extremely challenging in the presence of a large number of ambiguous classes. It is natural in that case to extend the evaluation protocol to allow $k$ guesses, which leads to the popular top- $k$ error and top- $k$ accuracy performance measures. Formally, we consider a ranking of labels induced by the prediction scores $\innerwy,x$ . Let the bracket $[\cdot]$ denote a permutation of labels such that $[j]$ is the index of the $j$ -th largest score,

\innerw[1],x≥\innerw[2],x≥…≥\innerw[m],x.

The top- $k$ zero-one loss $\errk$ is defined as

\errk(f(x),y)=\ind\innernw[k],x>\innernwy,x,

where $f(x)=(\innerw1,x,…,\innerwm,x)⊤$ and $\indP=1$ if $P$ is true and $0$ otherwise. Note that the standard zero-one loss is recovered when $k = 1$ , and $\errk(f(x),y)$ is always $0$ for $k = m$ . Therefore, we are interested in the regime $1 \leq k < m$ .

2.1 Multiclass Support Vector Machine

In this section we review the multiclass SVM of Crammer and Singer [6] which will be extended to the top- $k$ multiclass SVM in the following. We mainly follow the notation of [27].

Given a training pair $(x_{i}, y_{i})$ , the multiclass SVM loss on example $x_{i}$ is defined as

maxy∈\Yc{\indy≠yi+\innerwy,xi−\innerwyi,xi}.

(1)

Since our optimization scheme is based on Fenchel duality, we also require a convex conjugate of the primal loss function (1). Let $c\bydef\ones−eyi$ , where $\ones$

is the all ones vector and

e_{j}

is the

j

-th standard basis vector in

\Rbm

, let

a∈\Rbm

be defined componentwise as

aj\bydef\innerwj,xi−\innerwyi,xi

, and let

Δ\bydef{x∈\Rbm\given\inner\ones,x≤1,0≤xi,i=1,…,m}.

[[27], § 5.1] A primal-conjugate pair for the multiclass SVM loss (1) is

ϕ (a)

= max {0, (a + c)_{[1]}},

ϕ^{*} (b)

={−\innerc,bif b∈Δ,+∞otherwise.

(2)

Note that thresholding with $0$ in $ϕ (a)$ is actually redundant as $(a + c)_{[1]} \geq (a + c)_{y_{i}} = 0$ and is only given to enhance similarity to the top- $k$ version defined later.

2.2 Top- $k$ Support Vector Machine

The main motivation for the top- $k$ loss is to relax the penalty for making an error in the top- $k$ predictions. Looking at $ϕ$ in (2), a direct extension to the top- $k$ setting would be a function

ψ_{k} (a) = max {0, (a + c)_{[k]}},

which incurs a loss iff $(a + c)_{[k]} > 0$ . Since the ground truth score $(a + c)_{[y_{i}]} = 0$ , we conclude that

ψk(a)>0⟺\innerw[1],xi≥…≥\innerw[k],xi>\innerwyi,xi−1,

which directly corresponds to the top- $k$ zero-one loss $\errk$ with margin $1$ . Note that the function $ψ_{k}$ ignores the values of the first $(k - 1)$ scores, which could be quite large if there are highly similar classes. That would be fine in this model as long as the correct prediction is within the first $k$ guesses. However, the function $ψ_{k}$ is unfortunately nonconvex since the function $f_{k} (x) = x_{[k]}$ returning the $k$ -th largest coordinate is nonconvex for $k \geq 2$ . Therefore, finding a globally optimal solution is computationally intractable.

Instead, we propose the following convex upper bound on $ψ_{k}$ , which we call the top- $k$ hinge loss,

ϕ_{k} (a) = max {0, \frac{1}{k} k \sum j = 1 (a + c)_{[j]}},

(3)

where the sum of the $k$ largest components is known to be convex [4]. We have that

ψ_{k} (a) \leq ϕ_{k} (a) \leq ϕ_{1} (a) = ϕ (a),

for any $k \geq 1$ and $a∈\Rbm$ . Moreover, $ϕ_{k} (a) < ϕ (a)$ unless all $k$ largest scores are the same. This extra slack can be used to increase the margin between the current and the $(m - k)$ remaining least similar classes, which should then lead to an improvement in the top- $k$ metric.

2.2.1 Top- $k$ Simplex and Convex Conjugate of the Top- $k$ Hinge Loss

In this section we derive the conjugate of the proposed loss (3). We begin with a well known result that is used later in the proof. All proofs can be found in the supplement. Let $[a]_{+} = max {0, a}$ .

[[19], Lemma 1] $\sum_{j = 1}^{k} h_{[j]} = {min}_{t} {k t + \sum_{j = 1}^{m} [h_{j} - t]_{+}} .$

Proof.

For a $t_{0} \in [h_{[k + 1]}, h_{[k]}]$ , we have

min t {k t + m \sum j = 1 [h_{j} - t]_{+}} \leq k t_{0} + m \sum j = 1 [h_{j} - t_{0}]_{+} = k t_{0} + k \sum j = 1 (h_{[j]} - t_{0}) = k \sum i = 1 h_{[i]} .

On the other hand, for any $t∈\Rb$ , we get

k \sum j = 1 h_{[j]} = k t + k \sum j = 1 (h_{[j]} - t) \leq k t + k \sum j = 1 {[h_{[j]} - t]}_{+} \leq k t + m \sum j = 1 {[h_{j} - t]}_{+} .

∎

Figure 2: Top- $k$ simplex $Δ_{k} (1)$ for $m = 3$ . Unlike the standard simplex, it has $(\frac{m}{k}) + 1$ vertices.

We also define a set $Δ_{k}$ which arises naturally as the effective domain¹¹1 A convex function $f:X→\Rb∪{±∞}$ has an effective domain $\domf={x∈X\givenf(x)<+∞}$ . of the conjugate of (3). By analogy, we call it the top- $k$ simplex as for $k = 1$ it reduces to the standard simplex with the inequality constraint ( $0 \in Δ_{k}$ ). Let $[m]\bydef1,…,m$ .

The top- $k$ simplex is a convex polytope defined as

Δk(r)\bydef{x\given\inner\ones,x≤r,0≤xi≤1k\inner\ones,x,i∈[m]},

where $r \geq 0$ is the bound on the sum $\inner\ones,x$ . We let $Δk\bydefΔk(1)$ . The crucial difference to the standard simplex is the upper bound on $x_{i}$ ’s, which limits their maximal contribution to the total sum $\inner\ones,x$ . See Figure 2 for an illustration.

The first technical contribution of this work is as follows. A primal-conjugate pair for the top- $k$ hinge loss (3) is given as follows:

ϕ_{k} (a)

= max {0, \frac{1}{k} k \sum j = 1 (a + c)_{[j]}},

ϕ_{k}^{*} (b)

={−\innerc,bif b∈Δk,+∞otherwise.

(4)

Moreover, $ϕk(a)=max{\innera+c,λ\givenλ∈Δk}$ .

Proof.

We use Lemma 2.2.1 to write

ϕk(a)=min{s\givens≥t+1km∑j=1ξj,s≥0,ξj≥aj+cj−t,ξj≥0}.

The Lagrangian is given as

\Lc(s,t,ξ,α,β,λ,μ)=s+α(t+1km∑j=1ξj−s)−βs+m∑j=1λj(aj+cj−t−ξj)−m∑j=1μjξj.

Minimizing over $(s, t, ξ)$ , we get $α + β = 1$ , $α = \sum_{j = 1}^{m} λ_{j}$ , $λ_{j} + μ_{j} = \frac{1}{k} α$ . As $β \geq 0$ and $μ_{j} \geq 0$ , it follows that $\inner\ones,λ≤1$ and $0≤λj≤1k\inner\ones,λ$ . Since the duality gap is zero, we get

ϕk(a)=max{\innera+c,λ\givenλ∈Δk}.

The conjugate $ϕ_{k}^{*} (b)$ can now be computed as

maxa{\innera,b−ϕk(a)}=maxaminλ∈Δk{\innera,b−\innera+c,λ}=minλ∈Δk{−\innerc,λ+maxa\innera,b−λ}.

Since $maxa\innera,b−λ=∞$ unless $b = λ$ , we get the formula for $ϕ_{k}^{*} (b)$ as in (4).

∎

Therefore, we see that the proposed formulation (3) naturally extends the multiclass SVM of Crammer and Singer [6], which is recovered when $k = 1$ . We have also obtained an interesting extension (or rather contraction, since $Δ_{k} \subset Δ$ ) of the standard simplex.

2.3 Relation of the Top- $k$ Hinge Loss to Ranking Based Losses

Usunier [30] have recently formulated a very general family of convex losses for ranking and multiclass classification. In their framework, the hinge loss on example $x_{i}$ can be written as

L_{β} (a) = m \sum y = 1 β_{y} max {0, (a + c)_{[y]}},

where $β_{1} \geq \dots \geq β_{m} \geq 0$ is a non-increasing sequence of non-negative numbers which act as weights for the ordered losses. The relation to the top- $k$ hinge loss becomes apparent if we choose $β_{j} = \frac{1}{k}$ if $j \leq k$ , and $0$ otherwise. In that case, we obtain another version of the top- $k$ hinge loss

{~ ϕ}_{k} (a) = \frac{1}{k} k \sum j = 1 max {0, (a + c)_{[j]}} .

(5)

It is straightforward to check that

ψ_{k} (a) \leq ϕ_{k} (a) \leq {~ ϕ}_{k} (a) \leq ϕ_{1} (a) = {~ ϕ}_{1} (a) = ϕ (a) .

The bound $ϕ_{k} (a) \leq {~ ϕ}_{k} (a)$ holds with equality if $(a + c)_{[1]} \leq 0$ or $(a + c)_{[k]} \geq 0$ . Otherwise, there is a gap and our top- $k$ loss is a strictly better upper bound on the actual top- $k$ zero-one loss. While [30] employed LaRank [1] and [10], [31] optimized an approximation of $L_{β} (a)$ , we show in § 5 how the loss function (5) can be optimized exactly and efficiently within the Prox-SDCA framework.

Multiclass to binary reduction. It is also possible to compare directly to ranking based methods that solve a binary problem using the following reduction. We employ it in our experiments to evaluate the ranking based methods ${S V M}^{P e r f}$ [12] and $T o p P u s h$ [17]. The trick is to augment the training set by embedding each $xi∈\Rbd$ into $\Rbmd$ using a feature map $Φ_{y}$ for each $y∈\Yc$ . The mapping $Φ_{y}$ places $x_{i}$ at the $y$ -th position in $\Rbmd$ and puts zeros everywhere else. The example $Φ_{y_{i}} (x_{i})$ is labeled $+ 1$ and all $Φ_{y} (x_{i})$ for $y \neq y_{i}$ are labeled $- 1$ . Therefore, we have a new training set with $m n$ examples and $m d$ dimensional (sparse) features. Moreover, $\innerw,Φy(xi)=\innerwy,xi$ which establishes the relation to the original multiclass problem.

Another approach to general performance measures is given in [12]. It turns out that using the above reduction, one can show that under certain constraints on the classifier, the recall@ $k$ is equivalent to the top- $k$ error. A convex upper bound on recall@ $k$ is then optimized in [12] via structured SVM. As their convex upper bound on the recall@ $k$ is not decomposable in an instance based loss, it is not directly comparable to our loss. While being theoretically very elegant, the approach of [12] does not scale to very large datasets.

3 Optimization Framework

We begin with a general $ℓ_{2}$ -regularized multiclass classification problem, where for notational convenience we keep the loss function unspecified. The multiclass SVM or the top- $k$ multiclass SVM are obtained by plugging in the corresponding loss function from § 2.

3.1 Fenchel Duality for $ℓ_{2}$ -Regularized Multiclass Classification Problems

Let $X∈\Rbd×n$ be the matrix of training examples $xi∈\Rbd$ , let $W∈\Rbd×m$ be the matrix of primal variables obtained by stacking the vectors $wy∈\Rbd$ , and $A∈\Rbm×n$ the matrix of dual variables.

Before we prove our main result of this section (Theorem 3.1), we first impose a technical constraint on a loss function to be compatible with the choice of the ground truth coordinate. The top- $k$ hinge loss from Section 2 satisfies this requirement as we show in Proposition 3.1. We also prove an auxiliary Lemma 3.1, which is then used in Theorem 3.1.

A convex function $ϕ$ is $j$ -compatible if for any $y∈\Rbm$ with $y_{j} = 0$ we have that

sup{\innery,x−ϕ(x)\givenxj=0}=ϕ∗(y).

This constraint is needed to prove equality in the following Lemma. Let $ϕ$ be $j$ -compatible, let $Hj=\Id−\ones\traej$ , and let $Φ (x) = ϕ (H_{j} x)$ , then

Φ∗(y)={ϕ∗(y−yjej)if \inner\ones,y=0,+∞otherwise.

Proof.

We have that $\KerHj={x\givenHjx=0}={t\ones\givent∈\Rb}$ and $\ort\KerHj={x\given\inner\ones,x=0}$ .

	$Φ^{*} (y)$	$=sup{\innery,x−Φ(x)\givenx∈\Rbm}$
		$=sup{\innerny,\parax+\innerny,\ortx−ϕ(Hj\ortx)\givenx=\parax+\ortx,\parax∈\KerHj,\ortx∈\ort\KerHj}.$

It follows that $Φ^{*} (y)$ can only be finite if $\innerny,\parax=0$ , which implies $y∈\ort\KerHj$ . Let $\pinvHj$ be the Moore-Penrose pseudoinverse of $H_{j}$ . For a $y∈\ort\KerHj$ , we can write

$Φ^{*} (y)$	$=sup{\innerny,\pinvHjHj\ortx−ϕ(Hj\ortx)\given\ortx∈\ort\KerHj}$	(6)
	$=sup{\innern\tra(\pinvHj)y,z−ϕ(z)\givenz∈\ImgHj}$
	$≤sup{\innern\tra(\pinvHj)y,z−ϕ(z)\givenz∈\Rbm}=ϕ∗(\tra(\pinvHj)y),$

where $\ImgHj={Hjx\givenx∈\Rbm}$ . Using rank- $1$ update of the Moore-Penrose pseudoinverse ([22], § 3.2.7), we can compute $\tra(\pinvHj)=\Id−ej\traej−1m(\ones−ej)\tra\ones$ . Since $y∈\ort\KerHj$ , the last term is zero and we have $\tra(\pinvHj)y=y−yjej$ . Finally, we use the fact that $ϕ$ is $j$ -compatible to prove that the inequality in (6) is satisfied with equality. We have that $\ImgHj={z\givenzj=0}$ and $(y - y_{j} e_{j})_{j} = 0$ . Therefore, when $\inner\ones,y=0$ , $Φ∗(y)=sup{\innerny−yjej,z−ϕ(z)\givenzj=0}=ϕ∗(y−yjej)$ . ∎

We can now use Lemma 3.1 to compute convex conjugates of the loss functions.

Let $ϕ_{i}$ be $y_{i}$ -compatible for each $i \in [n]$ , let $λ > 0$ be a regularization parameter, and let $K=\traXX$ be the Gram matrix. The primal and Fenchel dual objective functions are given as:

	$P (W)$	$=+1nn∑i=1ϕi(\traWxi−\innerwyi,xi\ones)+λ2\tr(\traWW),$
	$D (A)$	$=−1nn∑i=1ϕ∗i(−λn(ai−ayi,ieyi))−λ2\tr(AK\traA), if% \inner\ones,ai=0∀i,+∞ otherwise.$

Moreover, we have that $W=X\traA$ and $\traWxi=AKi$ , where $K_{i}$ is the $i$ -th column of $K$ .

Proof.

We use Fenchel duality (see [2], Theorem 3.3.5), to write $P(W)=g(\traXW)+f(W)$ , and $D(A)=−g∗(−\traA)−f∗(X\traA)$ , for the functions $g$ and $f$ defined as follows:

g(\traXW)

=1nn∑i=1Φi(\traWxi)=1nn∑i=1ϕi(Hyi\traWxi),

f (W)

=λ2\tr(\traWW)=λ2\normsWF,

where $Hyi=\Id−\ones\traeyi$ . One can easily verify that $g∗(−\traA)=1n∑ni=1Φ∗i(−nai)$ and $f∗(X\traA)=λ2\norms1λX\traAF$ . From Lemma 3.1, we have that $Φ_{i}^{*} (- n a_{i}) = ϕ^{*} (- n (a_{i} - a_{y_{i}, i} e_{y_{i}})),$ if $\inner\ones,−nai=0$ , and $+ \infty$ otherwise. To complete the proof, we redefine $A \leftarrow \frac{1}{λ} A$ for convenience, and use the first order optimality condition ([2], Ex. 9.f in § 3) for the $W=X\traA$ formula. ∎

Finally, we show that Theorem 3.1 applies to the loss functions that we consider.

The top- $k$ hinge loss function from Section 2 is $y_{i}$ -compatible.

Proof.

Let $c=\ones−eyi$ and consider the loss $ϕ_{k}$ . As in Proposition 2.2.1, we have

maxa,ayi=0{\innera,b−ϕk(a)}=minλ∈Δk{−\innerc,λ+maxa,ayi=0\innera,b−λ}=ϕ∗k(b),

where we used that $c_{y_{i}} = 0$ and $b_{y_{i}} = 0$ (Definition 3.1), the $y_{i}$ -th coordinate has no influence. ∎

We have repeated the derivation from Section 5.7 in [27] as there is a typo in the optimization problem (20) leading to the conclusion that $a_{y_{i}, i}$ must be $0$ at the optimum. Lemma 3.1 fixes this by making the requirement $a_{y_{i}, i} = - \sum_{j \neq y_{i}} a_{j, i}$ explicit. Note that this modification is already mentioned in their pseudo-code for Prox-SDCA.

3.2 Optimization of Top- $k$ Multiclass SVM via Prox-SDCA

1: Input: training data

{(x_{i}, y_{i})_{i = 1}^{n}}

, parameters

k

(loss),

λ

(regularization),

ϵ

(stopping cond.)

2: Output:

W∈\Rbd×m

A∈\Rbm×n

3: Initialize:

W \leftarrow 0

A \leftarrow 0

4: repeat

5: randomly permute training data

6: for

i = 1

n

si←\traWxi

{prediction scores}

a_{i}^{o l d} \leftarrow a_{i}

{cache previous values}

ai←update(k,λ,\normsxi,yi,si,ai)

{see § 3.2.1 for details}

10:

W←W+xi\tra(ai−aoldi)

{rank-

1

update}

11: end for

12: until relative duality gap is below

ϵ

Algorithm 1 Top-

k

Multiclass SVM

As an optimization scheme, we employ the proximal stochastic dual coordinate ascent (Prox-SDCA) framework of Shalev-Shwartz and Zhang [27], which has strong convergence guarantees and is easy to adapt to our problem. In particular, we iteratively update a batch $ai∈\Rbm$ of dual variables corresponding to the training pair $(x_{i}, y_{i})$ , so as to maximize the dual objective $D (A)$ from Theorem 3.1. We also maintain the primal variables $W=X\traA$ and stop when the relative duality gap is below $ϵ$ . This procedure is summarized in Algorithm 1.

Let us make a few comments on the advantages of the proposed method. First, apart from the update step which we discuss below, all main operations can be computed using a BLAS library, which makes the overall implementation efficient. Second, the update step in Line 9 is optimal in the sense that it yields maximal dual objective increase jointly over $m$ variables. This is opposed to SGD updates with data-independent step sizes, as well as to maximal but scalar updates in other SDCA variants. Finally, we have a well-defined stopping criterion as we can compute the duality gap (see discussion in [3]). The latter is especially attractive if there is a time budget for learning. The algorithm can also be easily kernelized since $\traWxi=AKi$ (Theorem 3.1).

3.2.1 Dual Variables Update

For the proposed top- $k$ hinge loss from Section 2, optimization of the dual objective $D (A)$ over $ai∈\Rbm$ given other variables fixed is an instance of a regularized (biased) projection problem onto the top- $k$ simplex $Δ_{k} (\frac{1}{λ n})$ . Let $\woaj$ be obtained by removing the $j$ -th coordinate from vector $a$ .

The following two problems are equivalent with $\woaiyi=−x$ and $ayi,i=\inner\ones,x$

maxai{D(A)\given\inner\ones,ai=0}≡minx{\normsb−x+ρ\inner\ones,x2\givenx∈Δk(1λn)},

where $b=1\innerxi,xi(\woqyi+(1−qyi)\ones)$ , $q=\traWxi−\innerxi,xiai$ and $ρ = 1$ .

Proof.

Using Proposition 2.2.1 and Theorem 3.1, we write

max a_{i} {

−1nϕ∗i(−λn(ai−ayi,ieyi))−λ2\tr(AK\traA)\given\inner\ones,ai=0}.

For the loss function, we get

- \frac{1}{n} ϕ_{i}^{*} (- λ n (a_{i} - a_{y_{i}, i} e_{y_{i}})) = λ a_{y_{i}, i},

with $- λ n (a_{i} - a_{y_{i}, i} e_{y_{i}}) \in Δ_{k}$ . One can verify that the latter constraint is equivalent to $−\woaiyi∈Δk(1λn)$ , $ayi,i=\innern\ones,−\woaiyi$ . Similarly, we write for the regularization term

\tr(AK\traA)=Kii\innerai,ai+2∑j≠iKij\innerai,aj+const,

where the $c o n s t$ does not depend on $a_{i}$ . Note that $\sum_{j \neq i} K_{i j} a_{j} = A K_{i} - K_{i i} a_{i} = q$ and can be computed using the “old” $a_{i}$ . Let $x\bydef−\woaiyi$ , we have

\innerai,ai

=\inner\ones,x2+\innerx,x,

\innerq,ai

=qyi\inner\ones,x−\innern\woqyi,x.

Plugging everything together and multiplying with $- 2 / λ$ , we obtain

Collecting the corresponding terms finishes the proof. ∎

We discuss in the following section how to project onto the set $Δ_{k} (\frac{1}{λ n})$ efficiently.

4 Efficient Projection onto the Top- $k$ Simplex

One of our main technical results is an algorithm for efficiently computing projections onto $Δ_{k} (r)$ , respectively the biased projection introduced in Proposition 3.2.1. The optimization problem in Proposition 3.2.1 reduces to the Euclidean projection onto $Δ_{k} (r)$ for $ρ = 0$ , and for $ρ > 0$ it biases the solution to be orthogonal to $\ones$ . Let us highlight that $Δ_{k} (r)$ is substantially different from the standard simplex and none of the existing methods can be used as we discuss below.

4.1 Continuous Quadratic Knapsack Problem

Finding the Euclidean projection onto the simplex is an instance of the general optimization problem $minx{\normsa−x2\given\innerb,x≤r,l≤xi≤u}$ known as the continuous quadratic knapsack problem (CQKP). For example, to project onto the simplex we set $b=\ones$ , $l = 0$ and $r = u = 1$ . This is a well examined problem and several highly efficient algorithms are available (see the surveys [20, 21]). The first main difference to our set is the upper bound on the $x_{i}$ ’s. All existing algorithms expect that $u$ is fixed, which allows them to consider decompositions $minxi{(ai−xi)2\givenl≤xi≤u}$ which can be solved in closed-form. In our case, the upper bound $1k\inner\ones,x$ introduces coupling across all variables, which makes the existing algorithms not applicable. A second main difference is the bias term $ρ\inner\ones,x2$ added to the objective. The additional difficulty introduced by this term is relatively minor. Thus we solve the problem for general $ρ$ (including $ρ = 0$ for the Euclidean projection onto $Δ_{k} (r)$ ) even though we need only $ρ = 1$ in Proposition 3.2.1. The only case when our problem reduces to CQKP is when the constraint $\inner\ones,x≤r$ is satisfied with equality. In that case we can let $u = r / k$ and use any algorithm for the knapsack problem. We choose [14] since it is easy to implement, does not require sorting, and scales linearly in practice. The bias in the projection problem reduces to a constant $ρ r^{2}$ in this case and has, therefore, no effect.

4.2 Projection onto the Top- $k$ Cone

When the constraint $\inner\ones,x≤r$ is not satisfied with equality at the optimum, it has essentially no influence on the projection problem and can be removed. In that case we are left with the problem of the (biased) projection onto the top- $k$ cone which we address with the following lemma.

Let $x∗∈\Rbd$ be the solution to the following optimization problem

minx{\normsa−x+ρ\inner\ones,x2\given0≤xi≤1k\inner\ones,x,i∈[d]},

and let $U\bydef{i\givenx∗i=1k\inner\ones,x∗}$ , $M\bydef{i\given0<x∗i<1k\inner\ones,x∗}$ , $L\bydef{i\givenx∗i=0}$ .

If $U = \emptyset$ and $M = \emptyset$ , then $x^{*} = 0$ .
If $U \neq \emptyset$ and $M = \emptyset$ , then $U = {[1], \dots, [k]}$ , $x_{i}^{*} = \frac{1}{k + ρ k^{2}} \sum_{i = 1}^{k} a_{[i]}$ for $i \in U$ , where $[i]$ is the index of the $i$ -th largest component in $a$ .
Otherwise ( $M \neq \emptyset$ ), the following system of linear equations holds

$⎧⎪⎨⎪⎩u=(\absM∑i∈Uai+(k−\absU)∑i∈Mai)/D,t′=(\absU(1+ρk)∑i∈Mai−(k−\absU+ρk\absM)∑i∈Uai)/D,D=(k−\absU)2+(\absU+ρk2)\absM,$ (7)

together with the feasibility constraints on $t\bydeft′+ρuk$

$max i \in L a_{i}$ $\leq t \leq min i \in M a_{i},$ $max i \in M a_{i}$ $\leq t + u \leq min i \in U a_{i},$ (8)

and we have $x^{*} = min {max {0, a - t}, u}$ .

Proof.

We consider an equivalent problem

minx,s{12\normsa−x+12ρs2\given\inner\ones,x=s,0≤xi≤sk,i∈[d]}.

Let $t$ , $μ_{i} \geq 0$ , $ν_{i} \geq 0$ be the dual variables, and let $\Lc$ be the Lagrangian:

\Lc(x,s,t,μ,ν)=12\normsa−x+12ρs2+t(\inner\ones,x−s)−\innerμ,x+\innerν,x−sk\ones.

From the KKT conditions, we have that

∂x\Lc

=x−a+t\ones−μ+ν=0,

∂s\Lc

=ρs−t−1k\inner\ones,ν=0,

μ_{i} x_{i}

= 0,

ν_{i} (x_{i} - \frac{s}{k})

= 0.

We have that $x_{i} = min {max {0, a_{i} - t}, \frac{s}{k}}$ , $ν_{i} = max {0, a_{i} - t - \frac{s}{k}}$ , and $s=1ρ(t+1k\inner\ones,ν)$ . Let $p\bydef\inner\ones,ν$ . We have $t = ρ s - \frac{p}{k}$ . Using the definition of the sets $U$ and $M$ , we get

	$s$	$=∑i∈Usk+∑i∈M(ai−t)=∑i∈Mai−\absM(ρs−pk)+\absUsk,$
	$p$	$=∑i∈U(ai−t−sk)=∑i∈Uai−\absU(ρs−pk)−\absUsk.$

In the case $U \neq \emptyset$ and $M = \emptyset$ we get the simplified equations

	$s$	$= \sum i \in U \frac{s}{k} = \| U \| \frac{s}{k} ⟹ \| U \| = k,$
	$p$	$= \sum i \in U a_{i} - k ρ s + p - s ⟹ x_{i} = \frac{s}{k} = \frac{1}{k + ρ k^{2}} \sum i \in U a_{i}, i \in U .$

In the remaining case solving this system for $u\bydefsk$ and $t′\bydef−pk$ , we get exactly the system in (7). The constraints (8) follow from the definition of the sets $U$ , $M$ , $L$ , and ensure that the computed thresholds $(t, u)$ are compatible with the corresponding partitioning of the index set. ∎

We now show how to check if the (biased) projection is $0$ . For the standard simplex, where the cone is the positive orthant $\Rbd+$ , the projection is $0$ when all $a_{i} \leq 0$ . It is slightly more involved for $Δ_{k}$ .

The biased projection $x^{*}$ onto the top- $k$ cone is zero if $\sum_{i = 1}^{k} a_{[i]} \leq 0$ (sufficient condition). If $ρ = 0$ this is also necessary.

Proof.

Let $K\bydef{x\given0≤xi≤1k\inner\ones,x}$ be the top- $k$ cone. It is known that the Euclidean projection of $a$ onto $K$ is $0$ if and only if $a∈NK(0)\bydef{y\given∀x∈K,\innery,x≤0}$ , $a$ is in the normal cone to $K$ at $0$ . Therefore, we obtain as an equivalent condition that $maxx∈K\innera,x≤0$ . Take any $x \in K$ and let $s=\inner\ones,x$ . If $s > 0$ , we have that at least $k$ components in $x$ must be positive. To maximize $\innera,x$ , we would have exactly $k$ positive $x_{i} = \frac{s}{k}$ corresponding to the $k$ largest components in $a$ . That would result in $\innera,x=sk∑ki=1a[i]$ , which is non-positive if and only if $\sum_{i = 1}^{k} a_{[i]} \leq 0$ .

For $ρ > 0$ , the objective function has an additional term $ρ\inner\ones,x2$ that vanishes at $x = 0$ . Therefore, if $x = 0$ is optimal for the Euclidean projection, it must also be optimal for the biased projection. ∎

Projection. Lemmas 4.2 and 4.2 suggest a simple algorithm for the (biased) projection onto the top- $k$ cone. First, we check if the projection is constant (cases $1$ and $2$ in Lemma 4.2). In case $2$ , we compute $x$ and check if it is compatible with the corresponding sets $U$ , $M$ , $L$ . In the general case $3$ , we suggest a simple exhaustive search strategy. We sort $a$ and loop over the feasible partitions $U$ , $M$ , $L$ until we find a solution to (7) that satisfies (8). Since we know that $0≤\absU<k$ and $k≤\absU+\absM≤d$ , we can limit the search to $(k - 1) (d - k + 1)$ iterations in the worst case, where each iteration requires a constant number of operations. For the biased projection, we leave $x = 0$ as the fallback case as Lemma 4.2 gives only a sufficient condition. This yields a runtime complexity of $O (d log (d) + k d)$ , which is comparable to simplex projection algorithms based on sorting.

4.3 Projection onto the Top- $k$ Simplex

As we argued in § 4.1, the (biased) projection onto the top- $k$ simplex becomes either the knapsack problem or the (biased) projection onto the top- $k$ cone depending on the constraint $\inner\ones,x≤r$ at the optimum. The following Lemma provides a way to check which of the two cases apply.

Let $x∗∈\Rbd$ be the solution to the following optimization problem

minx{\normsa−x+ρ\inner\ones,x2\given\inner

Top-k Multiclass SVM

Nonlinear classifiers for ranking problems based on kernelized SVM

Nonparallel Hyperplane Classifiers for Multi-category Classification

A Parallel Way to Select the Parameters of SVM Based on the Ant Optimization Algorithm

Massively Scaling Heteroscedastic Classifiers

Research on Parallel SVM Algorithm Based on Cascade SVM

Multiclass Universum SVM

Tighter bounds lead to improved classifiers

Code Repositories

libsdca

1 Introduction

2 Top- $k$ Loss in Multiclass Classification

2.1 Multiclass Support Vector Machine

2.2 Top- $k$ Support Vector Machine

2.2.1 Top- $k$ Simplex and Convex Conjugate of the Top- $k$ Hinge Loss

Proof.

Proof.

2.3 Relation of the Top- $k$ Hinge Loss to Ranking Based Losses

3 Optimization Framework

3.1 Fenchel Duality for $ℓ_{2}$ -Regularized Multiclass Classification Problems

Proof.

Proof.

Proof.

3.2 Optimization of Top- $k$ Multiclass SVM via Prox-SDCA

3.2.1 Dual Variables Update

Proof.

4 Efficient Projection onto the Top- $k$ Simplex

4.1 Continuous Quadratic Knapsack Problem

4.2 Projection onto the Top- $k$ Cone

Proof.

Proof.

4.3 Projection onto the Top- $k$ Simplex

Top-k Multiclass SVM

Related Research

Nonlinear classifiers for ranking problems based on kernelized SVM

Nonparallel Hyperplane Classifiers for Multi-category Classification

A Parallel Way to Select the Parameters of SVM Based on the Ant Optimization Algorithm

Massively Scaling Heteroscedastic Classifiers

Research on Parallel SVM Algorithm Based on Cascade SVM

Multiclass Universum SVM

Tighter bounds lead to improved classifiers

Code Repositories

libsdca

1 Introduction

2 Top-k Loss in Multiclass Classification

2.1 Multiclass Support Vector Machine

2.2 Top-k Support Vector Machine

2.2.1 Top-k Simplex and Convex Conjugate of the Top-k Hinge Loss

Proof.

Proof.

2.3 Relation of the Top-k Hinge Loss to Ranking Based Losses

3 Optimization Framework

3.1 Fenchel Duality for ℓ2-Regularized Multiclass Classification Problems

Proof.

Proof.

Proof.

3.2 Optimization of Top-k Multiclass SVM via Prox-SDCA

3.2.1 Dual Variables Update

Proof.

4 Efficient Projection onto the Top-k Simplex

4.1 Continuous Quadratic Knapsack Problem

4.2 Projection onto the Top-k Cone

Proof.

Proof.

4.3 Projection onto the Top-k Simplex

2 Top- $k$ Loss in Multiclass Classification

2.2 Top- $k$ Support Vector Machine

2.2.1 Top- $k$ Simplex and Convex Conjugate of the Top- $k$ Hinge Loss

2.3 Relation of the Top- $k$ Hinge Loss to Ranking Based Losses

3.1 Fenchel Duality for $ℓ_{2}$ -Regularized Multiclass Classification Problems

3.2 Optimization of Top- $k$ Multiclass SVM via Prox-SDCA

4 Efficient Projection onto the Top- $k$ Simplex

4.2 Projection onto the Top- $k$ Cone

4.3 Projection onto the Top- $k$ Simplex