Recently, similarity-preserving hashing methods have been extensively studied for large-scale image retrieval. Compared with unsupervised hashing, supervised hashing methods for labeled data have usually better performance by utilizing semantic label information. Intuitively, for unlabeled data, it will improve the performance of unsupervised hashing methods if we can first mine some supervised semantic ’label information’ from unlabeled data and then incorporate the ’label information’ into the training process. Thus, in this paper, we propose a novel Object Detection based Deep Unsupervised Hashing method (ODDUH). Specifically, a pre-trained object detection model is utilized to mining supervised ’label information’, which is used to guide the learning process to generate high-quality hash codes.Extensive experiments on two public datasets demonstrate that the proposed method outperforms the state-of-the-art unsupervised hashing methods in the image retrieval task.
With the rapid growth of image data, approximate nearest neighbour (ANN) search have attracted more and more attention from researchers in the large scale image search area. Among the existing ANN search techniques, similarity-preserving hashing methods are advantageous due to their high retrieval efficiency and low storage cost. The main idea of hashing methods are to transform high dimensional data points into a set of compact binary codes, meanwhile, maintain similarity of the original data points. Since the original data points are represented by binary codes instead of real valued features, the time and memory cost of searching can be dramatically reduced.
In general, data-dependent hashing can be divided into unsupervised [8, 10, 5, 18] and supervised [33, 21, 32] methods. The unsupervised hashing methods mainly utilize the features of images to generate similarity-preserving binary codes without any supervised imformation. Compared with unsupervised hashing, supervised hashing methods incorporate sematic labels of training data into training process, thus they can perform more remarkably in generating similarity-preserving binary code. However, in many real applications, there are no semantic labels of images that can be used as supervised information. Hence, we just can use the unsupervised hashing methods to tackle the large scale image retrieval task in these case.
Intuitively, if we can detect the objects in images and use their classes as the pseudo-labels of the images, then we can use the pseudo-labels as ’supervised information’ to guide hash codes learning to obtain batter performance. An illustrative example is shown in Figure 1, the block (a) is the procedure of existing unsupervised hashing methods. They use the hand-crafted or learnt features as inputs. And they directly use the Euclidean distance between images or the similarity between one image and its rotated image to guide the hash training, which will make the existing unsupervised hashing models to judge that the images and are dissimilar with high possibility. Actually, the images and are similar, that the two images are belong to the class ’
’. On the contrary, in the block (b), if we use an object detection model to get the pseudo-labels of the two images by detecting objects inside an image and classify each object into one of many different classes. And by utilizing the pseudo-labels that both classes of images are ’’, we can construct pair-wise similarity to train hashing models and make the hashing models to judge that the two images and are similar with high possibility.
Inspired by this idea, we propose a novel Object Detection based Deep Unsupervised Hashing model, called ODDUH. In particular, an object detection model is first pre-trained on a large database which contains all the tags belonging to the hashing dataset. Then, we utilize the object detection model to mine latent semantic ’label information’( i.e., pseudo-labels) from images. And by taking use of pseudo-labels learned from the pre-trained object detection model, we define a novel similarity criterion called pair-wise percentage similarity inspired by ISDH  . Moreover, a shared CNN is introduced to capture the feature representations of images. Finally, we combine the pair-wise percentage similarity and the learnt feature representations of images to learn hash functions and generate high-quality similarity-preserving hash codes.
Extensive experiments on two real-world public datasets illustrate that our method outperforms the state-of-the-art unsupervised hash methods in image retrieval tasks. Our main contributions are outlined as follows:
We propose a novel unsupervised hashing architecture by introducing an pre-trained object detection model to mining semantic ’labels information’ from images.
Pair-wise percentage similarity is the first used in unsupervised hashing methods. With the guiding of pair-wise percentage similarity, we can greatly take use of the power of deep models to learn high-quality similarity-preserving binary codes. Moreover, the binary codes can preserve ranking imformation.
Experiments have shown that the proposed method can perform batter than the existing unsupervised hashing methods in large-scale image retrieval tasks.
Generally, existing hashing methods can be divided into data-independent hashing and data-dependent hashing. For data-independent hashing methods, the hashing functions are typically randomly generated without any training data. The representative data-independent methods include Locality Sensitivity Hashing (LSH)  and its variants [2, 15]. For data-dependent hashing methods, they can achieve better accuracy with shorter codes by learning hash functions from training data. Futhermore, data-dependent can be further classified into two categories: supervised [28, 20, 26] and unsupervised [6, 12, 5] methods. The supervised methods can achieve remarkable performance by utilizing labeled data to learn hashing functions. And the label information in supervised hashing methods can be used in the following three ways: point-wise label, pair-wise labels and triplet. Representative point-wise label based methods include: supervised discrete hashing (SDH) . Representative pair-wise labels based deep hashing methods include: Deep Supervised Hashing with Pairwise (DPSH) , Deep Supervised Discrete Hashing (DSDH) , Supervised Hierarchical Deep Hashing (SHDH) . Representative triplet based deep hashing methods include: Deep Semantic-preserving and Ranking-based Hashing (DSRH) , Deep Semantic Hashing with GANs (DSH-GANs) .
The unsupervised hashing can be divided into traditional unsupervised hashing methods and deep unsupervised hashing methods. The traditional unsupervised hashing methods used hand-crafted features and shallow hash functions to obtain binary hash code. Lots of algorithms in this category have been proposed, including Spectral Hashing (SH) , Iterative Quantization (ITQ) . However, limited by the hand-crafted features and shallow hash functions, it is hard for them to deal with complex and high dimensional real-world data and keep the semantic similarity between original data in the binary hash codes. The deep unsupervised hashing methods utilize deep architecture to learn hash code. Among the deep unsupervised hashing methods, Deepbit  get rotation invariant and balanced binary hash codes by defined a quantization loss. Unsupervised triplet hashing (UTH)  employs an unsupervised triplet loss to get balanced hash codes. HashGAN  generate compact hash codes by a generative adversarial hashing network.
However, few existing unsupervised hashing methods take a good use of the latent sematic ’label information’ in the images. Thus, in this paper, we propose a novel deep unsupervised hashing model based on object detection to generate high-quality hash codes by mining the latent semantic ’label information’ contained in images and incorporating the ’label information’ into the training process.
Object detection has been studied widely for locating an object inside the image and classifying the object into one of many different categories. And object detection can be simple categorized into two categories: classical models and deep learning models. In the classical models, one of the most popular is Viola-Jones framework, and it works by generating different (possibly thousands) simple binary classifiers using Haar features. However, with the growing success of deep learning, deep learning models are now state of the art in object detection, and many studies have been published about deep object detection. In this category, some models are based on region proposal which can solve the sliding windows problem, such as R-CNN ; and some models are based on regression which can have a fast detection speed, e.g., YOLO , SSD .
Most of the above approaches can have a good detection effect, and can mine the latent semantic ’label information’ in images, which is exactly what our hashing architecture need. Thus, we can chose one of state-of-the-art object detection methods such as YOLOv2  as a part of our hashing architecture.
Object Detection based Deep Unsupervised Hashing Network
In this section, we will present the proposed Object Detection based Deep Unsupervised Hashing Network (ODDUH) in detail.
Suppose a dataset has n images , and the image is . The goal of similarity-preserving hashing is to learn a mapping , where is the length of hashing codes, such that an input image will be encoded into a k-bit binary code .
The Architecture of ODDUH
As shown in Figure 2, our architecture consists of three parts: mining latent semantic ’label information’, feature learning and hash function learning.
In the mining latent semantic ’label information’ part, the ODDUH uses a pre-trained object detection model named YOLOv2  to mining the latent semantic ’label information’ in images. Note that other state-of-the-art object detection models can also be used such as SSD  and Mask R-CNN 
The feature learning part includes a convolutional neural network (CNN) component which has five convolutional layers and two fully-connected layers. What’s more, all the seven layers are the same as those of CNN-F network in Alexnet.Note that other CNN architectures can also be used here, such as VGG  and GoogLeNet .
The hash function learning part is a hashing layer which has units. Furthermore, is the length of hash code, and the hashing functions are learnt by the hashing layer. Eventually, we use element-wise sign function , which returns if the element is positive and returns otherwise, to process the outputs of the hashing layer and get the binary code b.
In ODDUH, we use an object detection model to mine the objects in images and get their classes (i.e., pseudo-labels). And for many images, more than one pesudo-labels will be mined. Thus, the unlabeled training dataset will become a ’mutil-label’ dataset. In oder to take a good use of the mined semantic ’lable information’, inspired by ISDH , the pair-wise percentage similarity is defined as:
where calculate the inner product and
is the pseudo-label vector of, where c is the total number of classes that pseudo-labels belong to. If image has the pseudo-label, then , else .
By incorporating the pair-wise percentage similarity into the training process, the learnt binary codes can preserve the similarity in . More specifically, if , the binary codes and should have large Hamming distance; if , the binary codes and should have a small Hamming distance; otherwise, the binary codes and should have a suitable Hamming distance complying with the similarity .
Given the binary codes for all the images, we can define the likelihood of the pair-wise percentage similarity as:
where , and . When , we take the negative log-likelihood of the observed pair-wise labels in S to measure the pair-wise similarity loss , and when , we take mean square error to measure the pair-wise similarity loss. Thus, the pair-wise similarity loss function can be defined as:
where is a hyper-parameter. is used to denote two conditions, when the pseudo-labels of image and the pseudo-labels of image are completely similar or dissimilar, i.e., , and when the pseudo-labels of image and the pseudo-labels of image are partly similar, i.e., .
By minimizing Eq. (3), we can make the hamming distance between two completely similar points as small as possible, and simultaneously make the hamming distance between two dissimilar points as large as possible. Meanwhile, we can make the partly similar image and image have the suitable hamming distance complying with the similarity .
where . is the outputs of hashing layer: , where the mapping is parameterized by and represents the parameters of the seven layers of network in the feature learning part. is the weight matrix to be learnt at the hashing layer, is the bias. Due to the is not the binary codes, we used a quantization loss to make to be close to binary codes. The quantization loss is defined as:
Then, by connecting the pseudo-label pair-wise similarity loss and quantization loss, the final objective function can be defined as:
where is a hyper-parameter.
In our method, the parameters containing need to be learnt, during the training phase. A mini-batch gradient descent method is used for learning. Moreover, we design an alternating method for learning. More specifically, we optimize B with fixed and optimize with B fixed.
The can be directly optimized as follows:
For the other parameters , standard back-propagation algorithm is used for learning. Especially, we are able to compute the derivatives of the loss function about as follows:
Then, we can use the standard back-propagation algorithm to update and v with Eq. (8):
The outline of the proposed method is described in Algorithm 1.
Dataset and Baseline
We conduct experiments on two public benchmark datasets: Pascal VOC 2007 111http://host.robots.ox.ac.uk/pascal/VOC/voc2007/  and BMVC 2009 222http://pascal.inrialpes.fr/data2/flickr-bmvc2009/ . Pascal VOC 2007 consists of 9,963 multi-label images. There are 20 object classes in this dataset. On average, each image is annotated with 1.5 labels. BMVC 2009 contains 96,378 images collected from Flickr. Each image in the dataset is associated with one or multiple labels in 20 semantic concepts.
Our proposed method is an unsupervised method, thus we compare our method with eight calssical and state-of-the-art unsupervised hashing methods including: LSH , ITQ , SH , PCAH , SGH , UH_BDNN , UTH , HashGAN , where LSH, SH, ITQ, PCAH and SGH are traditional unsupervised methods and the other three are deep unsupervised methods. Note that the five traditional unsupervised hashing methods use hand-crafted features as inputs. And each image in Pascal VOC 2007 and BMVC 2009 is represented by a 512-dimensional GIST vector. For the deep unsupervised hashing method UH_BDNN, it use the outputs of fc7 layer in AlexNet as image representation. And for the other two deep unsupervised hashing methods and our proposed method, we resize all the images to be pixels and then directly use the raw image pixels as input. When carry out experiments on the two datasets respectively, we randomly select 2,000 images as test set and the left images as training dataset. Moreover, we also conduct the experiments by using the outputs of fc7 layer in AlexNet  as image representation in the five traditional hashing approaches and denote them as LSH+CNN, SH+CNN, ITQ+CNN and PCAH+CNN, respectively.
For the object detection component, we choose YOLOv2 . And it is pre-trained in COCO 2014 dataset which contains 81 object classes. Please note that all the object classes contained in Pascal VOC 2007 and BMVC 2009 are subdet of the 81 object classes. For the hash code learning model formed by the feature learning part and hash function learning part, all the weights and bias are learned via back-propagation algorithm. Furthermore, the weights and bias in the feature learning part are initialized as the values pre-trained in Alexnet . We adopt SGD with a mini-batch size of 128 as our optimization algorithm. The learning rate is initialized as 0.01. The hyper-parameters in ODDUH are empirically set as 2 and 100, respectively. We will discuss the effect of
in the followed subsection. And the learning rate is adjusted to one tenth of the current learning rate every one third of epoches.
To verify the effectiveness of hash code, we evaluate the image retrieval quality for different methods by Average Cumulative Gains (ACG), Normalized Discounted Cumulative Gains (NDCG), Mean Average Precision (MAP) and Weighted Mean Average Precision (W-MAP).
ACG@n represents the average of similarities between the query image and the top n retrieved images, which can be calculated as:
where is defined as the number of shared labels between the query image and the retrieved image.
NDCG is a widely used evaluation metric in information retrieval. Given a query image, the DCG score of top n retrieved images is defined as:
Then, the normalized DCG (NDCG) score at the position n can be calculated by, where is the maximum value of DCG@n, constraining the value of NDCG in the range [0,1].
MAP, a standard evaluation metric for information retrieval, is the mean of average precision for each query. It is defined as:
where , R(j) represents the number of relevant images within the top j images. is a indicator function, if the image at the position j shares at least one label with the query image, is 1; otherwise is 0. N represents the total number of relevant images, i.e., it shares at least one label with the query image.
The definition of W-MAP is similar with MAP, defined as:
Furthermore, the three evaluation criterions WMAP, NDCG and ACG are usually used to measure the ranking quality of hashing models. Beacuse the greater value of WMAP or NDCG means that the related items in the retrieved result list have higher ranks. And the larger value of ACG indicates that the images in the retrieved result list are more similar to the query image.
Table 2 summarizes the comparative results of different hashing methods over Pascal VOC 2007. And table 2 shows the performance comparison of different hashing methods over BMVC 2009. In general, It can be found that our proposed method substantially outperforms the other unsupervised hashing methods for different length of hash code. In particular, on Pascal VOC 2007, comparing with the best traditional competitor ITQ+CNN on 48-bits, the results of ODDUH have a relative increase of 25.3% on MAP, 18.9% on WMAP, 20.0% on NDCG, 20.8% on ACG, and comparing with the best deep unsupervised competitor UH_BDNN on 48-bits, the results of ODDUH have a relative increase of 25.9% on MAP, 22.7% on WMAP, 25.9% on NDCG, 28.5% on ACG. Moreover, on BMVC 2009, comparing with the competitor UH_BDNN on 48-bits, the results of ODDUH have a relative increase of 8.9% on MAP, 8.4% on WMAP, 15.0% on NDCG, 18.0% on ACG. And as the growth of the three evaluation criterions WMAP, NDCG, ACG illustrate that the ODDUH can preserve more ranking information in the binary codes than the baselines. Furthermore,the results obviously indicates that pseudo-labels mined from images are more advantageous than the Euclidean distance between images or the similarity between one image and its rotated images to generate similarity-preserving hashing codes. In addition, most traditional unsupervised methods with image representations extracted from deep CNN architecture perform better than these methods with GIST features, which proves that the learnt representations by deep network from raw images are more superior than hand-crafted features in hash learning procedure.
The precision@n ( the precision value is calculated based on the top n returned neighbors) curves at different length of hash code are also showed in Figure 3. Figure 3(a) and Figure 3(b) exhibit the precision@n curves over Pascal VOC 2007 and BMVC 2009, respectively. And both Figure 3(a) and Figure 3(b) show that our ODDUH model perform batter than baselines. Also, the precisions for most of the baselines drop when the length of hash codes increases. Contrarily, the precision of ODDUH is still growing, which means ODDUH is more stable.
Sensitivity to Hyper-Parameters
Figure 4(a) shows the effect of the hyper-parameter on 48 bits over Pascal VOC 2007 and BMVC 2009. It can be found that ODDUH is not sensitive to on both datasets. For instance, ODDUH can achieve good performance on both datasets with . Figure 4(b) shows the effect of the hyper-parameter on 48 bits over Pascal VOC 2007 and BMVC 2009. Also, ODDUH is not sensitive to in a large range. For example, ODDUH can achieve good performance on both datasets with . And we can also obtain similar conclusion on other length of hash codes for both hyper-Parameters and .
In this paper, we have proposed a novel Object Detection based Deep Unsupervised Hashing method, called, for unlabeled data. To the best of our knowledge, ODDUH is the first method which utilize object detection model to mine semantic ’label information’ from images. By incorporating the semantic ’label information’ into the training process, the learnt hashing functions can generat high-quality similarity-preserving hash codes. Extensive experiments on two real-world public datasets have shown that the proposed ODDUH method can outperform other methods to achieve the state-of-the-art performance in image retrieval applications.
-  Moray Allan and Jakob Verbeek. Ranking user-annotated images for multiple query terms. In BMVC 2009-British Machine Vision Conference, pages 20–1. BMVA Press, 2009.
-  Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab S Mirrokni. Locality-sensitive hashing scheme based on p-stable distributions. In Proceedings of the twentieth annual symposium on Computational geometry, pages 253–262. ACM, 2004.
Thanh-Toan Do, Anh-Dzung Doan, and Ngai-Man Cheung.
Learning to hash with binary deep neural network.
European Conference on Computer Vision, pages 219–234. Springer, 2016.
-  Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303–338, 2010.
Kamran Ghasedi Dizaji, Feng Zheng, Najmeh Sadoughi, Yanhua Yang, Cheng Deng,
and Heng Huang.
Unsupervised deep generative adversarial hashing network.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3664–3673, 2018.
-  Aristides Gionis, Piotr Indyk, Rajeev Motwani, et al. Similarity search in high dimensions via hashing. In Vldb, volume 99, pages 518–529, 1999.
-  Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014.
-  Yunchao Gong and Svetlana Lazebnik. Iterative quantization: A procrustean approach to learning binary codes. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 817–824. IEEE, 2011.
-  Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Computer Vision (ICCV), 2017 IEEE International Conference on, pages 2980–2988. IEEE, 2017.
-  Shanshan Huang, Yichao Xiong, Ya Zhang, and Jia Wang. Unsupervised triplet hashing for fast image retrieval. In Proceedings of the on Thematic Workshops of ACM Multimedia 2017, pages 84–92. ACM, 2017.
-  Qing-Yuan Jiang and Wu-Jun Li. Scalable graph hashing with feature transformation. In IJCAI, pages 2248–2254, 2015.
-  Zhongming Jin, Cheng Li, Yue Lin, and Deng Cai. Density sensitive hashing. IEEE Trans. Cybernetics, 44(8):1362–1371, 2014.
-  Michael J Jones and Paul Viola. Robust real-time object detection. In Workshop on statistical and computational theories of vision, volume 266, page 56, 2001.
-  Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012.
-  Brian Kulis, Prateek Jain, and Kristen Grauman. Fast similarity search for learned metrics. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(12):2143–2157, 2009.
-  Qi Li, Zhenan Sun, Ran He, and Tieniu Tan. Deep supervised discrete hashing. In Advances in Neural Information Processing Systems, pages 2482–2491, 2017.
-  Wu-Jun Li, Sheng Wang, and Wang-Cheng Kang. Feature learning based deep supervised hashing with pairwise labels. arXiv preprint arXiv:1511.03855, 2015.
-  Kevin Lin, Jiwen Lu, Chu-Song Chen, and Jie Zhou. Learning compact binary descriptors with unsupervised deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1183–1192, 2016.
-  Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pages 21–37. Springer, 2016.
-  Wei Liu, Jun Wang, Rongrong Ji, Yu-Gang Jiang, and Shih-Fu Chang. Supervised hashing with kernels. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 2074–2081. IEEE, 2012.
-  Zhaofan Qiu, Yingwei Pan, Ting Yao, and Tao Mei. Deep semantic hashing with generative adversarial networks. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 225–234. ACM, 2017.
-  Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.
-  Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. arXiv preprint, 2017.
-  Fumin Shen, Chunhua Shen, Wei Liu, and Heng Tao Shen. Supervised discrete hashing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 37–45, 2015.
-  Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
-  Dongjin Song, Wei Liu, Rongrong Ji, David A Meyer, and John R Smith. Top rank supervised binary coding for visual search. In Proceedings of the IEEE International Conference on Computer Vision, pages 1922–1930, 2015.
-  Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015.
-  Dan Wang, Heyan Huang, Chi Lu, Bo-Si Feng, Liqiang Nie, Guihua Wen, and Xian-Ling Mao. Supervised deep hashing for hierarchical labeled data. pages 7388–7395, 2018.
-  Jun Wang, Sanjiv Kumar, and Shih-Fu Chang. Semi-supervised hashing for scalable image retrieval. 2010.
-  Yair Weiss, Antonio Torralba, and Rob Fergus. Spectral hashing. In Advances in neural information processing systems, pages 1753–1760, 2009.
-  Ting Yao, Fuchen Long, Tao Mei, and Yong Rui. Deep semantic-preserving and ranking-based hashing for image retrieval. In IJCAI, pages 3931–3937, 2016.
-  Zheng Zhang, Qin Zou, Qian Wang, Yuewei Lin, and Qingquan Li. Instance similarity deep hashing for multi-label image retrieval. arXiv preprint arXiv:1803.02987, 2018.
-  Han Zhu, Mingsheng Long, Jianmin Wang, and Yue Cao. Deep hashing network for efficient similarity retrieval. In AAAI, pages 2415–2421, 2016.