NIPS2016
This project collects the different accepted papers and their link to Arxiv or Gitxiv
view repo
In supervised binary hashing, one wants to learn a function that maps a highdimensional feature vector to a vector of binary codes, for application to fast image retrieval. This typically results in a difficult optimization problem, nonconvex and nonsmooth, because of the discrete variables involved. Much work has simply relaxed the problem during training, solving a continuous optimization, and truncating the codes a posteriori. This gives reasonable results but is quite suboptimal. Recent work has tried to optimize the objective directly over the binary codes and achieved better results, but the hash function was still learned a posteriori, which remains suboptimal. We propose a general framework for learning hash functions using affinitybased loss functions that uses auxiliary coordinates. This closes the loop and optimizes jointly over the hash functions and the binary codes so that they gradually match each other. The resulting algorithm can be seen as a corrected, iterated version of the procedure of optimizing first over the codes and then learning the hash function. Compared to this, our optimization is guaranteed to obtain better hash functions while being not much slower, as demonstrated experimentally in various supervised datasets. In addition, our framework facilitates the design of optimization algorithms for arbitrary types of loss and hash functions.
READ FULL TEXT VIEW PDF
Representing images by compact hash codes is an attractive approach for
...
read it
Binary hashing is a wellknown approach for fast approximate nearestnei...
read it
An attractive approach for fast search in image databases is binary hash...
read it
Recently, it has been observed that 0,1,1ternary codes which are simpl...
read it
Embedding image features into a binary Hamming space can improve both th...
read it
In this work, we firstly propose deep network models and learning algori...
read it
In the era of big data, methods for improving memory and computational
e...
read it
This project collects the different accepted papers and their link to Arxiv or Gitxiv
Implementation of our ICCV 2017 paper: MIHash  Hashing with Mutual Information
Information retrieval arises in several applications, most obviously web search. For example, in image retrieval, a user is interested in finding similar images to a query image. Computationally, this essentially involves defining a highdimensional feature space where each relevant image is represented by a vector, and then finding the closest points (nearest neighbors) to the vector for the query image, according to a suitable distance (Shakhnarovich et al., 2006). For example, one can use features such as SIFT (Lowe, 2004) or GIST (Oliva and Torralba, 2001) and the Euclidean distance for this purpose. Finding nearest neighbors in a dataset of images (where can be millions), each a vector of dimension (typically in the hundreds) is slow, since exact algorithms run essentially in time and space (to store the image dataset). In practice, this is approximated, and a successful way to do this is binary hashing (Grauman and Fergus, 2013). Here, given a highdimensional vector , the hash function maps it to a bit vector , and the nearest neighbor search is then done in the binary space. This now costs time and space, which is orders of magnitude faster because typically and, crucially, (1) operations with binary vectors (such as computing Hamming distances) are very fast because of hardware support, and (2) the entire dataset can fit in (fast) memory rather than slow memory or disk.
The disadvantage is that the results are inexact, since the neighbors in the binary space will not be identical to the neighbors in the original space. However, the approximation error can be controlled by using sufficiently many bits and by learning a good hash function
. This has been the topic of much work in recent years. The general approach consists of defining a supervised objective that has a small value for good hash functions and minimizing it. Ideally, such an objective function should be minimal when the neighbors of any given image are the same in both original and binary spaces. Practically in information retrieval, this is often evaluated using precision and recall. However, this ideal objective cannot be easily optimized over hash functions, and one uses approximate objectives instead. Many such objectives have been proposed in the literature. We focus here on
affinitybased loss functions, which directly try to preserve the original similarities in the binary space. Specifically, we consider objective functions of the form(1) 
where is the highdimensional dataset of feature vectors, means minimizing over the parameters of the hash function (e.g. over the weights of a linear SVM), and is a loss function that compares the codes for two images (often through their Hamming distance ) with the groundtruth value that measures the affinity in the original space between the two images and (distance, similarity or other measure of neighborhood; Grauman and Fergus, 2013). The sum is often restricted to a subset of image pairs (for example, within the nearest neighbors of each other in the original space), to keep the runtime low. Examples of these objective functions (described below) include models developed for dimension reduction, be they spectral such as Laplacian Eigenmaps (Belkin and Niyogi, 2003) and Locally Linear Embedding (Roweis and Saul, 2000), or nonlinear such as the Elastic Embedding (CarreiraPerpiñán, 2010) or SNE (van der Maaten and Hinton, 2008); as well as objective functions designed specifically for binary hashing, such as Supervised Hashing with Kernels (KSH) (Liu et al., 2012), Binary Reconstructive Embeddings (BRE) (Kulis and Darrell, 2009) or Semisupervised sequential Projection Learning Hashing (SPLH) (Wang et al., 2012).
If the hash function was a continuous function of its input
and its parameters, one could simply apply the chain rule to compute derivatives over the parameters of
of the objective function (1) and then apply a nonlinear optimization method such as gradient descent. This would be guaranteed to converge to an optimum under mild conditions (for example, Wolfe conditions on the line search), which would be global if the objective is convex and local otherwise (Nocedal and Wright, 2006). Hence, optimally learning the function would be in principle doable (up to local optima), although it would still be slow because the objective can be quite nonlinear and involve many terms.In binary hashing, the optimization is much more difficult, because in addition to the previous issues, the hash function must output binary values, hence the problem is not just generally nonconvex, but also nonsmooth. In view of this, much work has sidestepped the issue and settled on a simple but suboptimal solution. First, one defines the objective function (1) directly on the dimensional codes of each image (rather than on the hash function parameters) and optimizes it assuming continuous codes (in
). Then, one binarizes the codes for each image. Finally, one learns a hash function given the codes. Optimizing the affinitybased loss function (
1) can be done using spectral methods or nonlinear optimization as described above. Binarizing the codes has been done in different ways, from simply rounding them to using zero as threshold (Weiss et al., 2009; Zhang et al., 2010; Liu et al., 2011, 2012), to optimally finding a threshold (Liu et al., 2011; Strecha et al., 2012), to rotating the continuous codes so that thresholding introduces less error (Yu and Shi, 2003; Gong et al., 2013). Finally, learning the hash function for each of theoutput bits can be considered as a binary classification problem, where the resulting classifiers collectively give the desired hash function, and can be solved using various machine learning techniques. Several works (e.g.
Zhang et al., 2010; Lin et al., 2013, 2014) have used this approach, which does produce reasonable hash functions (in terms of retrieval measures such as precision and recall).In order to do better, one needs to take into account during the optimization (rather than after the optimization) the fact that the codes are constrained to be binary. This implies attempting directly the discrete optimization of the affinitybased loss function over binary codes. This is a daunting task, since this is usually an NPcomplete problem with binary variables altogether, and practical applications could make this number as large as millions or beyond. Recent works have applied alternating optimization (with various refinements) to this, where one optimizes over a usually small subset of binary variables given fixed values for the remaining ones (Lin et al., 2013, 2014), and this did result in very competitive precision/recall compared with the stateoftheart. This is still slow and future work will likely improve it, but as of now it provides an option to learn better binary codes.
Of the threestep suboptimal approach mentioned (learn continuous codes, binarize them, learn hash function), these works manage to join the first two steps and hence learn binary codes. Then, one learns the hash function given these binary codes. Can we do better? Indeed, in this paper we show that all elements of the problem (binary codes and hash function) can be incorporated in a single algorithm that optimizes jointly over them. Hence, by initializing it from binary codes from the previous approach, this algorithm is guaranteed to achieve a lower error and learn better hash functions. In fact, our framework can be seen as an iterated, corrected version of the twostep approach: learn binary codes given the current hash function, learn hash functions given codes, iterate (note the emphasis). The key to achieve this in a principled way is to use a recently proposed method of auxiliary coordinates (MAC) for optimizing “nested” systems, i.e., consisting of the composition of two or more functions or processing stages. MAC introduces new variables and constraints that cause decoupling between the stages, resulting in the mentioned alternation between learning the hash function and learning the binary codes. Section 2 reviews affinitybased loss functions, section 3 describes our MACbased proposed framework, section 4 evaluates it in several supervised datasets, using linear and nonlinear hash functions, and section 5 discusses implications of this work.
Although one can construct hash functions without training data (Andoni and Indyk, 2008; Kulis and Grauman, 2012)
, we focus on methods that learn the hash function given a training set, since they perform better, and our emphasis is in optimization. The learning can be unsupervised, which attempts to preserve distances in the original space, or supervised, which in addition attempts to preserve label similarity. Many objective functions have been proposed to achieve this and we focus on affinitybased ones. These create an affinity matrix for a subset of training points based on their distances (unsupervised) or labels (supervised) and combine it with a loss function
(Liu et al., 2012; Kulis and Darrell, 2009; Norouzi and Fleet, 2011; Lin et al., 2013, 2014). Some methods optimize this directly over the hash function. For example, Binary Reconstructive Embeddings (Kulis and Darrell, 2009) use alternating optimization over the weights of the hash functions. Supervised Hashing with Kernels (Liu et al., 2012) learns hash functions sequentially by considering the difference between the inner product of the codes and the corresponding element of the affinity matrix. Although many approaches exist, a common theme is to apply a greedy approach where one first finds codes using an affinitybased loss function, and then fits the hash functions to them (usually by training a classifier). The codes can be found by relaxing the problem and binarizing its solution (Weiss et al., 2009; Zhang et al., 2010; Liu et al., 2011), or by approximately solving for the binary codes using some form of alternating optimization (possibly combined with GraphCut), as in twostep hashing (Lin et al., 2013, 2014; Ge et al., 2014), or by using relaxation in other ways (Liu et al., 2012; Norouzi and Fleet, 2011).The dimensionality reduction literature has developed a number of objective functions of the form (1) (often called “embeddings”) where the lowdimensional projection
of each highdimensional data point
is a free, realvalued parameter. The neighborhood information is encoded in the values (using labels in supervised problems, or distancebased affinities in unsupervised problems). A representative example is the elastic embedding (CarreiraPerpiñán, 2010), where has the form:(2) 
where the first term tries to project true neighbors (having ) close together, while the second repels all nonneighbors’ projections (having ) from each other. Laplacian Eigenmaps (Belkin and Niyogi, 2003) and Locally Linear Embedding (Roweis and Saul, 2000) result from replacing the second term above with a constraint that fixes the scale of , which results in an eigenproblem rather than a nonlinear optimization, but also produces more distorted embeddings. Other objectives exist, such as SNE (van der Maaten and Hinton, 2008), that do not separate into functions of pairs of points. Optimizing nonlinear embeddings is quite challenging, but much progress has been done recently (CarreiraPerpiñán, 2010; Vladymyrov and CarreiraPerpiñán, 2012; van der Maaten, 2013; Yang et al., 2013; Vladymyrov and CarreiraPerpiñán, 2014). Although these models were developed to produce continuous projections, they have been successfully used for binary hashing too by truncating their codes (Weiss et al., 2009; Zhang et al., 2010) or using the twostep approach of (Lin et al., 2013, 2014).
Other loss functions have been developed specifically for hashing, where now is a bit vector (where binary values are in ). For example (see a longer list in Lin et al., 2013), for Supervised Hashing with Kernels (KSH) has the form
(3) 
where is if , are similar and if they are dissimilar. Binary Reconstructive Embeddings (Kulis and Darrell, 2009) uses where . The exponential variant of SPLH (Wang et al., 2012) proposed by Lin et al. (2013) (which we call eSPLH) uses . Our approach can be applied to any of these loss functions, though we will mostly focus on the KSH loss for simplicity. When the variables are binary, we will call these optimization problems binary embeddings, in analogy to the more traditional continuous embeddings for dimension reduction.
The optimization of the loss in eq. (1) is difficult because of the thresholded hash function, which appears as the argument of the loss function . We use the recently proposed method of auxiliary coordinates (MAC) (CarreiraPerpiñán and Wang, 2012, 2014), which is a metaalgorithm to construct optimization algorithms for nested functions. This proceeds in 3 stages. First, we introduce new variables (the “auxiliary coordinates”) as equality constraints into the problem, with the goal of unnesting the function. We can achieve this by introducing one binary vector for each point. This transforms the original, unconstrained problem into the following, constrained problem:
(4) 
which is seen to be equivalent to (1) by eliminating . We recognize as the objective function the “embedding” form of the loss function, except that the “free” parameters are in fact constrained to be the deterministic outputs of the hash function .
Second, we solve the constrained problem using a penalty method, such as the quadraticpenalty or augmented Lagrangian (Nocedal and Wright, 2006). We discuss here the former for simplicity. We solve the following minimization problem (unconstrained again, but dependent on ) while progressively increasing , so the constraints are eventually satisfied:
(5) 
The quadratic penalty is proportional to the Hamming distance between the binary vectors and .
Third, we apply alternating optimization over the binary codes and the hash function parameters . This results in iterating the following two steps (described in detail later):
Optimize the binary codes given (hence, given the output binary codes for each of the images). This can be seen as a regularized binary embedding, because the projections are encouraged to be close to the hash function outputs . Here, we try two different approaches (Lin et al., 2013, 2014) with some modifications.
Optimize the hash function given binary codes . This reduces to training binary classifiers using as inputs and as targets.
This is very similar to the twostep (TSH) approach of Lin et al. (2013), except that the latter learns the codes in isolation, rather than given the current hash function, so iterating the twostep approach would change nothing, and it does not optimize the loss . More precisely, TSH corresponds to optimizing for . In practice, we start from a very small value of (hence, initialize MAC from the result of TSH), and increase slowly while optimizing , until the equality constraints are satisfied, i.e., for .
Fig. 1 gives the overall MAC algorithm to learn a hash function by optimizing an affinitybased loss function. We now describe the steps over and , and the path followed by the iterates as a function of .
It is possible to prove that once after a step (regardless of the value of ), the MAC algorithm will make no further changes to or , since then the constraints are satisfied. This gives us a reliable stopping criterion that is easy to check, and the MAC algorithm will stop after a finite number of iterations (see below).
It is also possible to prove that the path of minimizers of over the continuous penalty parameter is in fact discrete, with changes to happening only at a finite number of values . Based on this and on our practical experience, we have found that the following approach leads to good schedules for with little effort. We use exponential schedules, of the form for , so the user has to set only two parameters: the initial and the multiplier . We choose exponential schedules because typically the algorithm makes most progress at the beginning, and it is important to track a good minimum there. The upper value past which no changes occur will be reached by our exponential schedule in a finite number of iterations, and our stopping criterion will detect that. We set the multiplier to a value that is as small as computationally convenient. If is too small, the algorithm will take many iterations, some of which may not even change or (because the path of minima is discrete). If is too big, the algorithm will reach too quickly a stopping point, without having had time to find a better minimum. As for the initial
, we estimate it by trying values (exponentially spaced) until we find a
for which changes to from its initial value (for ) start to occur. (It is also possible to find lower and upper bounds for and , respectively, for a particular loss function, such as KSH, eSPH or EE.) Overall, the computational time required to estimate and is comparable to running a few extra iterations of the MAC algorithm.Finally, in practice we use a form of early stopping in order to improve generalization. We use a small validation set to evaluate the precision achieved by the hash function along the MAC optimization. If the precision decreases over that of the previous step, we ignore the step and skip to the next value of . Besides helping to avoid overfitting, this saves computation, by avoid such extra optimization steps. Since the validation set is small, it provides a noisy estimate of the generalization ability at the current iterate, and this occasionally leads to skipping a valid value. This is not a problem because the next value, which is close to the one we skipped, will likely work. At some point during the MAC optimization, we do reach an overfitting region and the precision stops increasing, so the algorithm will skip all remaining values until it stops. In summary, using this validation procedure guarantees that the precision (in the validation set) is greater or equal than that of the initial , thus resulting in a better hash function.
Given the binary codes , since does not appear in the first term of , this simply involves finding a hash function that minimizes
where is the th bit of the binary vector . Hence, we can find onebit hash functions in parallel and concatenate them into the bit hash function. Each of these is a binary classification problem using the number of misclassified patterns as loss. This allows us to use a regular classifier for , and even to use a simpler surrogate loss (such as the hinge loss), since this will also enforce the constraints eventually (as increases). For example, we can fit an SVM by optimizing the margin plus the slack and using a high penalty for misclassified patterns. We discuss other classifiers in the experiments.
Although the MAC technique has significantly simplified the original problem, the step over is still complex. This involves finding the binary codes given the hash function , and it is an NPcomplete problem in binary variables. Fortunately, some recent works have proposed practical approaches for this problem based on alternating optimization: a quadratic surrogate method (Lin et al., 2013), and a GraphCut method (Lin et al., 2014). In both cases, this would correspond to the first step in the twostep hashing of Lin et al. (2013).
In both the quadratic surrogate and the GraphCut method, the starting point is to apply alternating optimization over the th bit of all points given the remaining bits are fixed for all points (for ), and to solve the optimization over the th bit approximately. We describe this next for each method. We start by describing each method in their original form (which applies to the loss function over binary codes, i.e., the first term in ), and then we give our modification to make it work with our step objective (the regularized loss function over binary codes, i.e., the complete ).
This is based on the fact that any loss function that depends on the Hamming distance of two binary variables can be equivalently written as a quadratic function of those two binary variables (Lin et al., 2013). Since this is the case for every term (because only the th bit in each of and is free), we can write the first term in as a binary quadratic problem. We now consider the second term (on ) as well. (We use a similar notation as that of Lin et al., 2013.) The optimization for the th bit can be written as:
(6) 
where is the loss function defined on the th bit, is the th bit of the th point, is a vector containing the binary codes of the th point except the th bit, and is the th bit of the binary code of the th point generated by the hash function . Lin et al. (2013) show that can be replaced by a binary quadratic function
(7) 
as long as and , where . Equation (7) helps us to rewrite the optimization (6) as the following:
(8) 
By defining as the element of a matrix and ignoring the coefficients, we have the following optimization problem:
where is a vector of length (one bit per data point). Both terms in the above minimization are quadratic on binary variables. This is still an NPcomplete problem (except in special cases), and we approximate it by relaxing it to a continuous quadratic program (QP) over and binarizing its solution. In general, the matrix is not positive definite and the relaxed QP is not convex, so we need an initialization. (However, the term on adds to , so even if is not positive definite, will be positive definite for large enough , and the QP will be convex.) We construct an initialization by converting the binary QP into a binary eigenproblem:
(9) 
To solve this problem we use spectral relaxation, where the constraints and are relaxed to
. The solution to this problem is the eigenvector corresponding to the smallest eigenvalue of
. We use the truncated eigenvector as the initialization for minimizing the relaxed, boundconstrained QP:which we solve using LBFGSB (Zhu et al., 1997).
As noted above, the step is an NPcomplete problem in general, so we cannot expect to find the global optimum. It is even possible that the approximate solution could increase the objective over the previous iteration’s (this is likely to happen as the overall MAC algorithm converges). If that occurs, we simply skip the update, in order to guarantee that we decrease monotonically on , and avoid oscillating around a minimum.
To optimize over the th bit (given all the other bits are fixed), we have to minimize eq. (8). In general, this is an NPcomplete problem over bits (the th bit for each image), with the form of a quadratic function on binary variables. We can apply the GraphCut algorithm (Boykov and Kolmogorov, 2003, 2004; Kolmogorov and Zabih, 2003), as proposed by the FastHash algorithm of Lin et al. (2014). This proceeds as follows. First, we assign all the data points to different, possibly overlapping groups (blocks). Then, we minimize the objective function over the binary codes of the same block, while all the other binary codes are fixed, then proceed with the next block, etc. (that is, we do alternating optimization of the bits over the blocks). Specifically, to optimize over the bits in block , we define and, ignoring the constants, we can rewrite equation (8) as:
We then rewrite this equation in the standard form for the GraphCut algorithm:
where , . To minimize the objective function using the GraphCut algorithm, the blocks have to define a submodular function. For the objective functions that we explained in the paper, this can be easily achieved by putting points with the same label in one block (Lin et al., 2014 give a simple proof of this).
Unlike in the quadratic surrogate method, using the GraphCut algorithm with alternating optimization on blocks defining submodular functions is guaranteed to find a that has a lower or equal objective value that the initial one, and therefore to decrease monotonically .
We have tested our framework with several combinations of loss function, hash function, number of bits, datasets, and comparing with several stateoftheart hashing methods (appendix A contains additional experiments). We report a representative subset to show the flexibility of the approach. We use the KSH (3) (Liu et al., 2012) and eSPLH (Wang et al., 2012) loss functions. We test quadratic surrogate and GraphCut methods for the step in MAC. As hash functions (for each bit), we use linear SVMs (trained with LIBLINEAR; Fan et al., 2008) and kernel SVMs^{1}^{1}1To train a kernel SVM, we use radial basis functions with centers given by a random subset of the training points, and apply a linear SVM to their output. Computationally, this is fast because we can use a constant Gram matrix. Using as hash function a kernel SVM trained with LIBSVM gave similar results, but is much slower because the support vectors change when the labels change. We set the RBF bandwidth to the average Euclidean distance of the first points.
We use the following labeled datasets (all using the Euclidean distance in feature space): (1) CIFAR (Krizhevsky, 2009) contains images in classes. We use GIST features (Oliva and Torralba, 2001) from each image. We use images for training and for test. (2) Infinite MNIST (Loosli et al., 2007). We generated, using elastic deformations of the original MNIST handwritten digit dataset, images for training and for test, in 10 classes. We represent each image by a vector of raw pixels. Because of the computational cost of affinitybased methods, previous work has used training sets limited to a few thousand points (Kulis and Darrell, 2009; Norouzi and Fleet, 2011; Liu et al., 2012; Lin et al., 2013). We train the hash functions in a subset of points of the training set, and report precision and recall by searching for a test query on the entire dataset (the base set).
We report precision and precision/recall for the test set queries using as ground truth (set of true neighbors in original space) all the training points with the same label. In precision curves, the retrieved set contains the nearest neighbors of the query point in the Hamming space. We report precision for different values of to test the robustness of different algorithms. In precision/recall curves, the retrieved set contains the points inside Hamming distance of the query point. These curves show the precision and recall at different Hamming distances to . We report zero precision when there is no neighbor inside Hamming distance of a query. This happens most of the time when is large and is small. In most of our precision/recall curves, the precision drops significantly for very small and very large values of . For small values of , this happens because most of the query points do not retrieve any neighbor. For large values of , this happens because the number of retrieved points becomes very large.
The main comparison point are the quadratic surrogate and GraphCut methods of Lin et al. (2013, 2014), which we denote in this section as quad and cut, respectively, regardless of the hash function that fits the resulting codes. Correspondingly, we denote the MAC version of these as MACquad and MACcut, respectively. We use the following schedule for the penalty parameter in the MAC algorithm (regardless of the hash function type or dataset). We initialize with , i.e., the result of quad or cut. Starting from (MACcut) or (MACquad), we multiply by after each iteration ( and step).
Our experiments show that the MAC algorithm indeed finds hash functions with a significantly and consistently lower objective function value than rounding or twostep approaches (in particular, cut and quad); and that it outperforms other stateoftheart algorithms on different datasets, with MACcut beating MACquad most of the time. The improvement in precision makes using MAC well worth the relatively small extra runtime and minimal additional implementation effort it requires. In all our plots, the vertical arrows indicate the improvement of MACcut over cut and of MACquad over quad.
The goal of this paper is not to introduce a new affinitybased loss or hash function, but to describe a generic framework to construct algorithms that optimize a given combination thereof. We illustrate its effectiveness here with the CIFAR dataset, with different sizes of retrieved neighbor sets, and using to bits. We optimize two affinitybased loss functions (KSH from eq. (3) and eSPLH), and two hash functions (linear and kernel SVM). In all cases, the MAC algorithm achieves a better hash function both in terms of the loss and of the precision/recall. We compare 4 ways of optimizing the loss function: quad (Lin et al., 2013), cut (Lin et al., 2014), MACquad and MACcut.
For each point in the training set, we use positive (similar) and negative (dissimilar) neighbors, chosen at random to have the same or a different label as , respectively. Fig. 2(top panel) shows the KSH loss function for all the methods (including the original KSH method in Liu et al., 2012) over iterations of the MAC algorithm (KSH, quad and cut do not iterate), as well as precision and recall. It is clear that MACcut (red lines) and MACquad (magenta lines) reduce the loss function more than cut (blue lines) and quad (black lines), respectively, as well as the original KSH algorithm (cyan), in all cases: type of hash function (linear: dashed lines, kernel: solid lines) and number of bits to . Hence, applying MAC is always beneficial. Reducing the loss nearly always translates into better precision and recall (with a larger gain for linear than for kernel hash functions, usually). The gain of MACcut/MACquad over cut/quad is significant, often comparable to the gain obtained by changing from the linear to the kernel hash function within the same algorithm.
We usually find cut outperforms quad (in agreement with Lin et al., 2014), and correspondingly MACcut outperforms MACquad. Interestingly, MACquad and MACcut end up being very similar even though they started very differently. This suggests it is not crucial which of the two methods to use in the MAC step, although we still prefer cut, because it usually produces somewhat better optima. Finally, fig. 2(bottom panel) shows the MACcut results using the eSPLH loss. All settings are as in the first KSH experiment. As before, MACcut outperforms cut in both loss function and precision/recall using either a linear or a kernel SVM.
In both the twostep and MAC approaches, the starting point are the “free” binary codes obtained by minimizing the loss over the codes without them being the output of a particular hash function. That is, minimizing (4) without the “” constraints:
(10) 
The resulting free codes try to achieve good precision/recall independently of whether a hash function can actually produce such codes. Constraining the codes to be realizable by a specific family of hash functions (say, linear), means the loss will be larger than for free codes. How difficult is it for a hash function to produce the free codes? Fig. 3 plots the loss function for the free codes, the twostep codes from cut, and the codes from MACcut, for both linear and kernel hash functions in the same experiment as in fig. 2. It is clear that the free codes have a very low loss , which is far from what a kernel function can produce, and even farther from what a linear function can produce. Both of these are relatively smooth functions that cannot represent the presumably complex structure of the free codes. This could be improved by using a very flexible hash function (e.g. using a kernel function with many centers), which could better approximate the free codes, but 1) a very flexible function would likely not generalize well, and 2) we require fast hash functions for fast retrieval anyway. Given our linear or kernel hash functions, what the twostep cut optimization does is fit the hash function directly to the free codes. This is not guaranteed to find the best hash function in terms of the original problem (1), and indeed it produces a pretty suboptimal function. In contrast, MAC gradually optimizes both the codes and the hash function so they eventually match, and finds a better hash function for the original problem (although it is still not guaranteed to find the globally optimal function of problem (1), which is NPcomplete).
Fig. 4 illustrates this conceptually. It shows the space of all possible binary codes, the contours of (green) and the set of codes that can be produced by (say) linear hash functions (gray), which is the feasible set . The twostep codes “project” the free codes onto the feasible set, but these are not the codes for the optimal hash function .
01[l][l]
free[][]

The runtime per iteration for our point training sets with bits and and neighbors in a laptop is 2’ for both MACcut and MACquad. They stop after 10–20 iterations. Each iteration is comparable to a single cut or quad run, since the step dominates the computation. The iterations after the first one are faster because they are warmstarted.
Fig. 5 shows results on CIFAR and Infinite MNIST. We create affinities for all methods using the dataset labels as before, with similar neighbors and dissimilar neighbors. We compare MACquad and MACcut with TwoStep Hashing (quad) (Lin et al., 2013), FastHash (cut) (Lin et al., 2014), Hashing with Kernels (KSH) (Liu et al., 2012), Iterative Quantization (ITQ) (Gong et al., 2013), Binary Reconstructive Embeddings (BRE) (Kulis and Darrell, 2009) and SelfTaught Hashing (STH) (Zhang et al., 2010). MACquad, MACcut, quad and cut all use the KSH loss function (3). The results show that MACcut (and MACquad) generally outperform all other methods, often by a large margin, in nearly all situations (dataset, number of bits, size of retrieved set). In particular, MACcut and MACquad are the only ones to beat ITQ, as long as one uses sufficiently many bits.
The twostep approach of TwoStep Hashing (Lin et al., 2013) and FastHash (Lin et al., 2014) is a significant advance in finding good codes for binary hashing, but it also causes a maladjustment between the codes and the hash function, since the codes were learned without knowledge of what hash function would use them. Ignoring the interaction between the loss and the hash function limits the quality of the results. For example, a linear hash function will have a harder time than a nonlinear one at learning such codes. In our algorithm, this tradeoff is enforced gradually (as increases) in the step as a regularization term (eq. (5)): it finds the best codes according to the loss function, but makes sure they are close to being realizable by the current hash function. Our experiments demonstrate that significant, consistent gains are achieved in both the loss function value and the precision/recall in image retrieval over the twostep approach.
A similar, wellknown situation arises in feature selection for classification
(Kohavi and John, 1998). The best combination of classifier and features will result from jointly minimizing the classification error with respect to both classifier and features (the “wrapper” approach), rather than first selecting features according to some criterion and then using them to learn a particular classifier (the “filter” approach). From this point of view, the twostep approaches of (Lin et al., 2013, 2014) are filter approaches that first optimize the loss function over the codes (equivalently, optimize with ) and then fit the hash function to those codes. Any such filter approach is then equivalent to optimizing over for .The method of auxiliary coordinates algorithmically decouples (within each iteration) the two elements that make up a binary hashing model: the hash function and the loss function. Both elements act in combination to produce a function that maps input patterns to binary codes so that they represent neighborhood in input space, but they play distinct roles. The hash function role is to map input patterns to binary codes. The loss function role is to assign binary codes to input patterns in order to preserve neighborhood relations, regardless of how easy it is for a mapping to produce such binary codes. By itself, the loss function would produce a nonparametric hash function for the training set with the form of a table of (image,code) pairs. However, the hash function and the loss function cannot act independently, because the objective function depends on both. The optimal combination of hash and loss is difficult to obtain, because of the nonlinear and discrete nature of the objective. Several previous optimization attempts for binary hashing first find codes that optimize the loss, and then fit a hash function to them, thus imposing a strict, suboptimal separation between loss function and hash function. In MAC, both elements are decoupled within each iteration, while still optimizing the correct objective: the step over the hash function does not involve the loss, and the step over the codes does not involve the hash function, but both are iterated. The connection between both steps occurs through the auxiliary coordinates, which are the binary codes themselves. The penalty regularizes the loss so that its optimal codes are progressively closer to what a hash function from the given class (e.g. linear) can achieve.
What is the best type of hash function to use? The answer to this is not unique, as it depends on applicationspecific factors: quality of the codes produced (to retrieve the correct images), time to compute the codes on highdimensional data (since, after all, the reason to use binary hashing is to speed up retrieval), ease of implementation within a given hardware architecture and software libraries, etc. Our MAC framework facilitates considerably this choice, because training different types of hash functions simply involves reusing an existing classification algorithm within the step, with no changes to the step.
In terms of runtime, the resulting MAC algorithm is not much slower than the twostep approach; it is comparable to iterating the latter a few times. Besides, since all iterations except the first are warmstarted, the average cost of one iteration is lower than for the twostep approach.
Finally, note that the method of auxiliary coordinates can be used also to learn an outofsample mapping for a continuous embedding (CarreiraPerpiñán and Vladymyrov, 2015), such as the elastic embedding (CarreiraPerpiñán, 2010) or SNE (van der Maaten and Hinton, 2008)—rather than to learn hash functions for a discrete embedding, as is our case in binary hashing. The resulting MAC algorithm optimizes over the outofsample mapping and the auxiliary coordinates (which are the data points’ lowdimensional projections), by alternating two steps. One step optimizes the outofsample mapping that projects highdimensional points to the continuous, latent space, given the auxiliary coordinates . This is a regression problem, while in binary hashing this is a classification problem (per hash function). The other step optimizes the auxiliary coordinates given the mapping and is a regularized continuous embedding problem. Both steps can be solved using existing algorithms. In particular, solving the step can be done efficiently with large datasets by using body methods and efficient optimization techniques (CarreiraPerpiñán, 2010; Vladymyrov and CarreiraPerpiñán, 2012; van der Maaten, 2013; Yang et al., 2013; Vladymyrov and CarreiraPerpiñán, 2014). In binary hashing the
step is a combinatorial optimization and, at present, far more challenging to solve. However, with continuous embeddings one must drive the penalty parameter
to infinity for the constraints to be satisfied and so the solution follows a continuous path over , while with binary hashing the solution follows a discretized, piecewise path which terminates at a finite value of .The method of auxiliary coordinates has also been applied in the context of binary hashing to a different objective function, the
binary autoencoder (BA)
(CarreiraPerpiñán and Raziperchikolaei, 2015):(11) 
where is the hash function, or encoder (which outputs binary values), and is a decoder. (ITQ, Gong et al., 2013, can be seen as a suboptimal way to optimize this.) As with the affinitybased loss function, the MAC algorithm alternates between fitting the hash function (and the decoder) given the codes, and optimizing over the codes. However, in the binary autoencoder the optimization over the codes decouples over every data point (since the objective function involves one term per data point). This has an important computational advantage in the step: rather than having to solve one large optimization problem over binary variables, it has to solve small optimization problems each over variables, which is much faster and easier to solve (since is relatively small in practice), and to parallelize. Also, the BA objective does not require any neighborhood information (e.g. the affinity between pairs of neighbors) and scales linearly with the dataset. Computing these affinity values, or even finding pairs of neighbors in the first place, is computationally costly. For these reasons, the BA can scale to training on larger datasets than affinitybased loss functions.
The BA objective function does have the disadvantage of being less directly related to the goals that are desirable from an information retrieval point of view, such as precision and recall. Neighborhood relations are only indirectly preserved by autoencoders (CarreiraPerpiñán and Raziperchikolaei, 2015), whose direct aim is to reconstruct its inputs and thus to learn the data manifold (imperfectly, because of the binary projection layer). Affinitybased loss functions of the form (1) allow the user to specify more complex neighborhood relations, for example based on class labels, which may significantly differ from the actual distances in image feature space. Still, finding more efficient and scalable optimization methods for binary embeddings (in the step of the MAC algorithm), that are able to handle larger numbers of training and neighbor points, would improve the quality of the loss function. This is an important topic of future research.
We have proposed a general framework for optimizing binary hashing using affinitybased loss functions. It improves over previous, twostep approaches based on learning binary codes first and then learning the hash function. Instead, it optimizes jointly over the binary codes and the hash function in alternation, so that the binary codes eventually match the hash function, resulting in a better local optimum of the affinitybased loss. This was possible by introducing auxiliary variables that conditionally decouple the codes from the hash function, and gradually enforcing the corresponding constraints. Our framework makes it easy to design an optimization algorithm for a new choice of loss function or hash function: one simply reuses existing software that optimizes each in isolation. The resulting algorithm is not much slower than the suboptimal twostep approach—it is comparable to iterating the latter a few times—and well worth the improvement in precision/recall.
The step over the hash function is essentially a solved problem if using a classifier, since this can be learned in an accurate and scalable way using machine learning techniques. The most difficult and timeconsuming part in our approach is the optimization over the binary codes, which is NPcomplete and involves many binary variables and terms in the objective. Although some techniques exist (Lin et al., 2013, 2014) that produce practical results, designing algorithms that reliably find good local optima and scale to large training sets is an important topic of future research.
Another direction for future work involves learning more sophisticated hash functions that go beyond mapping image features onto output binary codes using simple classifiers such as SVMs. This is possible because the optimization over the hash function parameters is confined to the
step and takes the form of a supervised classification problem, so we can apply an array of techniques from machine learning and computer vision. For example, it may be possible to learn image features that work better with hashing than standard features such as SIFT, or to learn transformations of the input to which the binary codes should be invariant, such as translation, rotation or alignment.
Although affinitybased hashing is intended to work with supervised datasets, it can also be used with unsupervised ones, and our MAC approach applies just as well. We use the SIFT1M dataset (Jégou et al., 2011), which contains training highresolution color images and test images, each represented by SIFT features. The experiments and conclusions are generally the same as with supervised datasets, with small differences in the settings of the experiments. In order to construct an affinitybased objective function, we define neighbors as follows. For each point in the training set we use the nearest neighbors as positive (similar) neighbors, and points chosen randomly among the remaining points as negative (dissimilar) neighbors. We report precision and precision/recall for the test set queries using as ground truth (set of true neighbors in original space) the nearest neighbors in unsupervised datasets, and all the training points with the same label in supervised datasets.
Fig. 7 shows results using KSH and eSLPH loss functions, respectively, with different sizes of retrieved neighbor sets and using to bits. As with the supervised datasets, it is clear that the MAC algorithm finds better optima and that MACcut is generally better than MACquad. Fig. 8 shows one case, using , and (1% of the base set), where quad outperforms cut and correspondingly MACquad outperforms MACcut, although both MAC results are very close, particularly in precision and recall.
Fig. 10 shows results comparing with binary hashing methods. All methods are trained on a subset of points. We consider two types of methods. In the first type, we create pseudolabels for each point and then apply supervised methods as in CIFAR (in particular, cut/quad and MACcut/MACquad, using the KSH loss function). The pseudolabels for each training point are obtained by declaring as similar points its true nearest neighbors and as dissimilar points a random subset of points among the remaining points. In the second type, we use purely unsupervised methods (not based on similar/dissimilar affinities): thresholded PCA (tPCA), Iterative Quantization (ITQ) (Gong et al., 2013), Binary Autoencoder (BA) (CarreiraPerpiñán and Raziperchikolaei, 2015), Spectral Hashing (SH) (Weiss et al., 2009), AnchorGraph Hashing (AGH) (Liu et al., 2011), and Spherical Hashing (SPH) (Heo et al., 2012). The results are again in general agreement with the conclusions in the main paper.
Fig. 12 shows the results (for all methods on SIFT1M) in effective number of bits . This is a measure of code utilization of a hash function introduced by CarreiraPerpiñán and Raziperchikolaei (2015), defined as the entropy of the code distribution. That is, given the codes for the training set, we consider them as samples of a distribution over the possible codes. The entropy of this distribution, measured in bits, is between (when all codes are equal) and (when all codes are distributed as uniformly as possible). We do the same for the test set. Although code utilization correlates to some extent with precision/recall when ranking different methods, a large does not guarantee a good hash function, and indeed, tPCA (which usually achieves a low precision compared to the stateoftheart) typically achieves the largest ; see the discussion in CarreiraPerpiñán and Raziperchikolaei (2015). However, a large does indicate a better use of the available codes (and fewer collisions if ), and has the advantage over precision/recall that it does not depend on any user parameters (such as ground truth size or retrieved set size), so we can compare all binary hashing methods with a single number (for a given number of bits ). It is particularly useful to compare methods that are optimizing the same objective function. With this in mind, we can compare MACcut with cut and MACquad with quad because these pairs of methods optimize the same objective function.
Work supported by NSF award IIS–1423515. We thank MingHsuan Yang, YiHsuan Tsai and Mehdi Alizadeh (UC Merced) for useful discussions.
Proc. of the 2015 IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR’15)
, pages 557–566, Boston, MA, June 7–12 2015.Proc. of the 17th Int. Conf. Artificial Intelligence and Statistics (AISTATS 2014)
, pages 10–19, Reykjavik, Iceland, Apr. 22–25 2014.Fast supervised hashing with decision trees for highdimensional data.
In Proc. of the 2014 IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR’14), pages 1971–1978, Columbus, OH, June 23–28 2014.Training invariant support vector machines using selective sampling.
In L. Bottou, O. Chapelle, D. DeCoste, and J. Weston, editors, Large Scale Kernel Machines, Neural Information Processing Series, pages 301–320. MIT Press, 2007.Multiclass spectral clustering.
In Proc. 9th Int. Conf. Computer Vision (ICCV’03), pages 313–319, Nice, France, Oct. 14–17 2003.
Comments
There are no comments yet.