Generalisation and Sharing in Triplet Convnets for Sketch based Visual Search

We propose and evaluate several triplet CNN architectures for measuring the similarity between sketches and photographs, within the context of the sketch based image retrieval (SBIR) task. In contrast to recent fine-grained SBIR work, we study the ability of our networks to generalise across diverse object categories from limited training data, and explore in detail strategies for weight sharing, pre-processing, data augmentation and dimensionality reduction. We exceed the performance of pre-existing techniques on both the Flickr15k category level SBIR benchmark by 18%, and the TU-Berlin SBIR benchmark by ∼10 T_b, when trained on the 250 category TU-Berlin classification dataset augmented with 25k corresponding photographs harvested from the Internet.


page 3

page 4

page 8


Fine-Grained Instance-Level Sketch-Based Video Retrieval

Existing sketch-analysis work studies sketches depicting static objects ...

Sketch-Specific Data Augmentation for Freehand Sketch Recognition

Sketch recognition remains a significant challenge due to the limited tr...

Instance-level Sketch-based Retrieval by Deep Triplet Classification Siamese Network

Sketch has been employed as an effective communicative tool to express t...

Sketch-QNet: A Quadruplet ConvNet for Color Sketch-based Image Retrieval

Architectures based on siamese networks with triplet loss have shown out...

Adaptive Fine-Grained Sketch-Based Image Retrieval

The recent focus on Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) ...

SketchParse : Towards Rich Descriptions for Poorly Drawn Sketches using Multi-Task Hierarchical Deep Networks

The ability to semantically interpret hand-drawn line sketches, although...

Universal Perceptual Grouping

In this work we aim to develop a universal sketch grouper. That is, a gr...

1 Introduction

Sketches are an intuitive and concise modality for communicating everyday concepts; abstract visual depictions pre-date the written language of our early ancestors, and drawings are one of the earliest communication mediums employed by children. With the advent of modern touch-screen based devices, gestural forms of interaction are of increasing interest as a means for navigating visual media. This paper addresses the particular problem of sketch based image retrieval (SBIR); searching a collection of photographs (images) for a particular visual concept using a free-hand sketched query.

The principal contribution of this paper is to explore SBIR from the perspective of a cross-domain modelling problem, in which a low dimensional embedding is learned between the space of sketches and photographs. Historically, SBIR has been addressed using sparse feature extraction and dictionary learning approaches, following the successful application of the same to object recognition and the measurement of visual similarity in natural images

[14, 9, 15]

. Deep convolutional neural networks (convnets/CNNs) have since gained traction as a powerful and flexible tool for tackling a diverse range of machine perception problems

[18], and very recently have also been explored for SBIR within the context of fine-grain retrieval, e.g. to find a specific shoe within a dataset of only shoes [30, 24]. Our work is aligned with the larger body of SBIR work addressing the problem of category-level retrieval, in which a user sketches a kind of object with particular attributes (e.g. a specific furniture form, a spotted dog, or a building with particular structure), seeking images that conform to that structure rather than a specific single image [10, 9, 15, 23, 22]

. Acknowledging the complementarity and value in both perspectives, this paper for the first time explores deep learning for category-level SBIR; compatible with the inherent ambiguity of sketch and the common use case of sketching an unseen prototypical object ‘from the mind’s eye’ (

e.g. web search), rather than to recall a previous observation of a specific object to the last detail. Specifically this paper explores appropriate convnet architectures, weight sharing schemes and training methodologies to learn a low-dimensional embedding for the representation of both sketches and photographs — in practical terms, a space amenable to fast approximate nearest neighbour (ANN) search (e.g. norm) for high performance SBIR. We explore several important questions around effective deep learning of such representations.

1. Generalisation: Given the diversity of visual concepts in the wild ( categories) and the challenges of annotating large sketch datasets (current best, categories [9]) how well can a CNN generalise beyond its training concepts to represent unseen sketched object categories? Are class diversity and volume of exemplars equally important?

2. Input Modality: SBIR and the related task of sketched image classification variously employ edge extraction as a pre-processing step to align the statistics of sketch and photo distributions. Is this a beneficial strategy when learning a SBIR feature embedding?

3. Architecture: Recent exploration of SBIR has indicated triplet loss CNNs as a promising archetype for SBIR embedding, however what kind of loss objective should be considered and where, and which weight sharing strategies are most effective? What is the best way to enforce a low dimensional embedding for efficient SBIR indexing?

2 Related Work

Sketch based Image Retrieval (SBIR) began to gain momentum in the early nineties with colour-blob based query systems such as Flickner et al.’s QBIC [12] that matched coarse attributes of colour, shape and texture using region adjacency graphs. Several global image descriptors for matching blob based queries were subsequently proposed, using spectral signatures derived from Haar Wavelets [16]

and the Short-Time Fourier Transform

[25]. This early wave of SBIR systems was complemented in the late nineties by algorithms accepting line-art sketches, more closely resembling the free-hand sketches casually generated by lay users in the act of sketching a throw-away query [7]. Such systems are characterised by their optimisation based matching approach; fitting the sketch under a deformable model to measure the support for sketched structure within each photograph in the database [3, 6]. Despite good accuracy, such approaches are slow and scale at best linearly. It was not until comparatively recently that global image descriptors were derived from line-art sketches, enabling more scalable indexing solutions.

2.1 SBIR with shallow features

Mirroring the success of gradient domain features and dictionary learning methods to photo retrieval, both Hu et al. [14] and Eitz et al. [11] extended the Bag of Visual Words (BoVW) pipeline to SBIR, notably also proposing the Flickr15k dataset which became a defacto benchmark for category-level SBIR [15]

. Sparse features including the Structure Tensor

[10], SHoG [11], Gradient Field Histogram of Oriented Gradients (GF-HOG) [15] and its extended version [4] are extracted from images pre-processed via Canny edge detection. Mid-level features were also explored through the HELO and key-shapes schemes of Saavedra and Bebustos [23, 22], which although not indexable via BoVW could be matched via Hungarian algorithm. Mid-level structures were also explored in the Mindfinder system of Cao et al. [5] who were the first to propose inverse index structure for scalable SBIR. Such systems score around 10-15% on the Flickr15k dataset often failing in the presence of edge clutter. Recently, Qi et al. employed an alternative edge detection pre-process raising the state of the art to 18% over Flickr15k. Their Perceptual Edge [20] filter simplifies the edge map prior to HoG/BoVW delivering a performance gain in cluttered scenes.

2.2 SBIR with deep networks

The use of deep networks to obtain data-driven representations can potentially overcome the challenges of learning from different domains. An example of single network model is the SketchANet [31], a smaller version of AlexNet optimised for sketch recognition only. However, while learning from a single domain is straightforward — provided enough data — mapping between a sketch and a photograph often requires multi-branch networks for cross-domain mapping. In this context, it is possible to learn representations for each domain independently, or perform a multi-domain learning by sharing knowledge across the different – but related – domains. Triplet convnets are especially interesting in this scenario since they employ three branches [28]

: an anchor branch, which models the reference object, one branch representing positive examples (which should be similar to the anchor) and another modeling negative examples (which should be as different as possible to the anchor). The triplet loss function is responsible to guide the training stage considering the relationship between the three models. This method was used for photographic queries by Wang

et al[28] and for sketched queries in order to refine search within a specific object class (e.g. a dataset of shoes) [30]. Similarly, a fine-grained approach to SBIR was adopted by the recent Sketchy system of Sangkloy et al. [24] in which careful reproduction of stroke detail is invited for instance-level search.

In Yu et al. [30], the authors train one model for each target category, the sketch is matched against the edgemap extracted from a well-behaved image (without clutter, often with constant background). They report that using a fully-shared network was better than use two networks without weight sharing. However, in category-level problem it may be beneficial to avoid sharing all layers to encourage generalisation (c.f. in Sec. 4.4). Wang et al[29] used a two-branch network with contrastive loss for sketch-based 3D shape retrieval without sharing weights, indicating that when two domains are very different sharing knowledge may not be ideal. Therefore, questions remain around training methodology including architecture, weight-sharing strategies, and loss functions, as well as the generalisation capability of such models. Our work explores these questions, and broadens the investigation of deep learning to SBIR beyond intra-class or instance level search to retrieval across multiple object categories. To avoid confusion we hereafter refer as no-share or Heterogeneous those multi-branch networks for which there are no shared weights between layers [29]; as full-share or Siamese those for which all layers have shared weights in all layers [30, 28]; and half-share or Hybrid those for which only a subset of layers are shared.

3 Methodology

We propose and investigate several triplet network designs, comparing the Siamese architecture with the Heterogeneous and a Hybrid design. Triplet networks are commonly used to learn joint embeddings from data distributions, and have been recently applied to photographic visual search [13, 21]. In an image retrieval context, a query image is presented to the anchor branch, and relevant/irrelevant images are presented to the positive and negative branches respectively. The positive and negative branches share their weights because they represent the same domain (e.g. photos). The anchor branch represents a different domain (sketch) and differences in design arise from the degree to which layers within the anchor branch share weights with the other branches. One or several fully-connected (FC) layers unifying the branches may be installed, as may loss functions pre- and post- unification. In this section we explore several permutations of this design, evaluating their performance in Sec. 4 with the aim of testing the generalisation capability of the network across categories, and identifying the best performing architecture (CNN architecture, loss) and training strategy to optimise retrieval accuracy.

3.1 Network architecture

Whilst some SBIR techniques perform feature extraction directly from image pixels, several pre-process images into edge-maps [30, 27]. We explore both strategies through two base architectures in our experiments.

Sketch-edgemap matching

: we utilise the SketchANet architecture [31] for both sketch and edgemap (positive/negative) branches, adopting similar branch design on the assumption that a photo’s edgemap is statistically closer to a sketch. The SketchANet network is similar to AlexNet [18] but 8 times smaller and optimised for sketch, which is ideal for applications that require fast processing time like SBIR. Fig. 1 depicts three weight sharing variants explored in this work. The full-share network (Siamese) has three identical branches. The no-share network (Heterogeneous) allows the anchor branch to be learned independently. The half-share network (Hybrid) has part of the anchor branch shared with the two others. Since the top layers of a CNN network learn high-level concepts we shared the top 4 layers among the 3 branches.

Figure 1: Three triplet network designs evaluated for sketch-edgemap matching and the corresponding visualisation of its first convolutional layer. Arrows/same colour indicates sharing of weights. For simplicity we omit the negative branch. Note the effect of integrating classification loss into the Heterogeneous network (middle), resulting in a learned convolution layer contrasting with the Siamese (left) and half-share (right) networks.

Sketch-photo matching

: we utilise the AlexNet CNN for the photo branches, and a hybrid CNN combining AlexNet and SketchANet for the sketch branch that shares layers 6-7 between the sketch and photo branches (see Fig. 2). Specifically, layers 1-3 have SketchANet architecture, layers 6-7 mirror AlexNet while the middle layers 4-5 we have modified from SketchANet as a hybridisation of the two designs. We configure fewer shareable layers than sketch-edgemap’s because the domain gap between sketch and photo is larger. Nevertheless, layers 6 and 7 alone account for more than 90% of the parameters of both networks. To the best of our knowledge, this is the first triplet network presented for visual search that has different architecture for the anchor and the positive/negative branches.

Figure 2: Half-share triplet network for sketch-photo direct mapping hybridised from the AlexNet and SketchANet designs. Layers with the same colour indicate weight sharing.The dimensionality reduction layer

(red) is not associated with any activation function. The 4 FC layers can alternatively be configured unshared for a Heterogeneous network. When training this network the softmax loss layer is only activated during the training phase 2.

3.2 Dimensionality reduction

It is desirable in visual search to have a compact representation of images. Dimensionality reduction is generally within a CNN by adding an intermediate fully-connected (FC) layer with a lower number of nodes. In a classification network, this number is constrained by number of categories i.e. intermediate FC layer should be larger than the final -way FC layer where D is class number. An advantage of the triplet network is that the dimensionality of the output embedding can be freely set, since the triplet loss function only compares the output features of the three branches, and does not take into account labels of the input. In all experiments we fix the embedding dimension , regardless of network design or number of categories in the training set. This allows us to encode the whole Flickr15K dataset using just 7MB memory.

Even when classification loss must be integrated in the network during certain training phases (see Sec. 3.3), we propose an unique way to overcome the problem as illustrated in Fig. 2. Here we add an embedding layer () between layer FC7 () and FC8 (

) without being followed by an activation (ReLU) layer. This prevents the embedding layer becoming a bottleneck in the network, since from the perspective of the softmax-loss layer the connection from FC7 to FC8 is linear. We empirically verify that during training the performance of the classification layer is not affected whether

is integrated or not.

3.3 Training procedure

We now describe the different training strategies for different network designs. Practically we observed that difficulties in achieving convergence increase from full-share, half-share to no-share, and that due to the domain gap, learning a sketch-photo mapping is more challenging that the sketch-edgemap case. In all experiments, we steer away from using classification loss to prevent affecting the generalisation capability of the network. Nevertheless we found that integrating softmax-loss, even if only at an early phase, is unavoidable for the training to converge in some cases.

Full-share sketch-edgemap: this configuration was trainable using standard triplet loss in a single step.
Half-share sketch-edgemap: this configuration encounters the so called “gradient vanishing problem”. We proposed a new loss function to overcome it (Sec. 3.4).
No-share sketch-edgemap: we conduct a two-step training process: (i) pre-train the sketch and edgemap branches separately using softmax-loss; (ii) remove softmax-loss and train the network further with the triplet-loss.

Given the instantiation of existing networks with branches of the triplet CNN, an opportunity exists to exploit  a prior trained models. Although we leverage fine-tuning in our experiments training sketch-photo networks, (subsec. 4.4) we train all networks from scratch during our generalisation experiments (subsec. 4.3).

Sketch-photo i.e. cross-domain learning requires more effort to achieve good convergence. We use the following multi-step training strategy:
Phase 1: Classification train the sketch and photo branches independently using a softmax-loss at FC8.
Phase 2: Classification + regression for the half-share network only. We form a double-branch network, freeze the unshared layers which were already learnt during phase 1. Next, we use contrastive loss together with softmax-loss to train the sharing layers. The use of softmax-loss in such double-branch network helps the sharing layers to learn discriminative features from both sketch and edgemap domains, whilst contrastive-loss provides an early step of regression to bring the two domains together.
Phase 3: Regression Remove softmax-loss and its associated fully-connected layer, unfreeze all frozen layers and train the network with triplet loss.
Phase 4 (optional) Any auxiliary sketch-photo datasets available can be used to further refine the model.

In the above training procedure, classification loss is used to guide the training during the early phases. Without it the training would not converge. The latter phases purely use regression to deliver the embedding for SBIR.

Data augmentation

plays an important role in preventing overfitting, especially when training data is limited. In all experiments we apply the following augmentation techniques for both sketch/edgemap and photo: random crop (sketch/edgemap crop size 225x225 for SketchANet, photo crop size 224x224 for Alexnet), random rotation in range degrees, random scaling in range and random horizontal flip.

We also propose an augmentation method applicable for sketches only. For sketches with at least strokes ( in our experiments) we divide them into four equal groups of strokes in drawing order. The first group contains the most important strokes — related to the more coarse structure of the object — and it is always kept. A new sketch is created by randomly discarding some of the other groups. This technique is inspired by Yu et al. [31, 30] who observe that people tend to draw sketches in stages at distinct levels of abstraction. We observed a mAP improvement across the board using this random stroke removal augmentation method on the Flickr15k benchmark.

3.4 The gradient vanishing problem

The gradient vanishing problem [28]

manifests itself during the training of several of the above network designs where the vectors output from the three network branches fall very close to each other. In this scenario, instead of pushing the negative feature point away while bringing the positive point closer to the anchor point, the training ends up equalising distance between all three points (Fig. 


-left). Consequently the standard triplet loss flattens at half of the margin and stops learning (its derivative w.r.t each input becomes zero). We hypothesise that the training probably falls to a “saddle” space and unable to move down the either way (illustrated in Fig. 

3-middle). We adopt a modified loss function to prevent “saddle” from being created at the outset of training:


where , and are output features from the anchor, positive and negative branches respectively; is the margin and is size of the training mini-batch.

Figure 3: The gradient vanishing problem: (left) a training failure case where loss flattens at half of the margin; (middle top) a saddle created in the loss space when anchor, positive and negative points are very close to each other; (middle bottom) modified loss slope ideal for gradient descent when anchor point is far from positive and negative points; (right) successful training of the half-share network with new loss function 1. Note: the validation mAP is the retrieval score when querying the validation sketch set against the training photos.

Under this loss, forcing sketch feature to be relatively half of the edgemap feature will break that balance, thus creating a “slope” for gradient descent to converge (Fig. 3-right). At query time, the sketch (query) feature is scaled by 2.0 to match the scale of features extracted from the photos.

4 Experiments

We now evaluate the variants of our proposed triplet architecture and weight sharing schemes to determine best performance embedding for SBIR. In particular we evaluate the ability of the network to generalise beyond the categories to which it is exposed during training. This is important for SBIR in the wild, where one cannot reasonably train with a sufficiently diverse sample of potential query images. We also investigate the impact of volume of sketch data used to train the network, and the impact of using photos or their edge-maps during training (in addition to the various weight sharing variants).

4.1 Datasets

We train and evaluate our networks using four sketch datasets: the TU-Berlin datasets for classification [9] and SBIR [11], the Flickr15K benchmark dataset for SBIR [15], and the Sketchy [24] dataset:

TU-Berlin-Class [9] (used here for training) for sketch classification comprising 250 categories of sketches, 80 per category, crowd-sourced from 1350 different non-expert participants with diverse drawing styles;

TU-Berlin-Retr [11] (used here for testing) takes into account not only the category of the retrieved images but also the relative order of the relevant images. The dataset consists of 31 sketches and 40 ranked images for each sketch (1240 total images), mixed with a set of 100,000 distracting Flickr photos. The authors propose a Kendal score as the evaluation method;

Sketchy [24] (used here for model fine-tuning) is a recent fine-grained dataset in which each photo image has instance-level matching sketches drawn by different subjects. The dataset has only 125 categories but over 500 sketches per category;

Flickr15K [15] (used for testing) has labels for 33 categories of sketches, 10 sketches per category. It also has a different number of photo images per category totalling 15,024 images. The Flickr15K has only four categories that are similar to the ones in TU-Berlin-Class dataset: “swan”, “flowers”, “bicycle” and “airplane”. Those two datasets also differ in the concept of some categories, for example TU-Berlin has a general “bridge” category while Flickr15K distinguishes “London bridge”, “Oxford bridge” and “Sydney bridge”. Also, their sketch depiction is different, motivating a need for good generalisation beyond training.

As TU-Berlin-Class comprises only sketches, in order to obtain our training triplets we automatically generate per-category photograph sets by querying the 250 category names on Creative Commons image repositories. The Flickr API was used to download images from 184 categories. Google and Bing engines were used for the remaining 66 categories which are mainly human body parts (e.g. brain, tooth, skeleton) and fictional objects (e.g. UFO, mermaid, dragon) where Flickr content was sparse. We manually selected the 100 most relevant photos per category to form our training data.

4.2 Experimental settings

We use the TU-Berlin-Class as the training sketch set. Sketches are skeletonised to be consistent with the photo’s edgemap. For the sketch-edgemap experiments, we extracted the edgemaps of the photo dataset using a state-of-the-art non-deep learning detector, gPb [1], after [20]. Hysteresis thresholding is applied on the resulting gPb soft-edge as per Canny edge detection to remove weak edges as well as isolated edge pixels.

As test set, we use the category-level SBIR Flicrk15K benchmark [15]. Additionally, we also tested our models on the TU-Berlin-Retr [11]. Note that this dataset is completely different from TU-Berlin-Class [9] (subsec. 4.1). While the performance metric for the Flickr15K benchmark is retrieval Mean Average Precision (mAP), the TU-Berlin-Retr benchmark supports labels for the order of relevance of the returned photos so the Kendall’s rank correlation coefficient was employed.

A series of experiments was carried out, starting with a subset of 20 random training categories and 20 sketches per category, up to the whole training dataset. As the TU-Berlin-Class has 80 sketches per category, the remaining sketches of the chosen categories are used for validation. We use Caffe the deep-learning library 

[17] for the training tasks. When training the triplet network, positive and negative photo samples are selected randomly.

4.3 Generalisation and weight sharing

We first report the results of generalisation capability of our sketch-edgemap triplet network with varying amount of training data. Fig. 4 (top) shows that the performance is benefited by increasing the number of training categories. All three network designs achieve near-linear improvement of retrieval performance against Flickr15k benchmark (discarding the four intersecting categories with the training set) with exposure to more diverse category set during training. The mAP of all models jumps from to - when raising training data from 20 to 250 categories.

Figure 4: Top: performance of the 3 different weight sharing (sketch-edgemap) strategies on Flickr15K, with the half-sharing scheme outperforming the others. Number of training sketches per category: 20. Bottom: generalisation with respect to number of training sketches per category using 250 training categories.

Fig. 4 (top) also indicates the superior performance of the half-share triplet architecture against the others in sketch-edgemap matching. Although all three perform the same with 20-categories training data, the half-share model outperforms the alternatives by mAP when more categories are available. Additionally, the no-share network, despite being pretrained (via step 1 of our training process, c.f. subsec 3.1) with softmax loss, outperforms the Siamese configuration by mAP. This implies that (i) sketch and photo’s edgemap, although assumed statistically similar, should be treated as two different domains; (ii) sharing the top layers of the two branches can deliver a significant improvement when the two input domains are similar.

Next, we test the performance with different number of sketches per category during training. To do so we fix the number of training categories at 250, and vary the number of training sketches per category from 20 to 80 samples while keeping the other settings. All models benefit from using more sketches, with a more notable boost for half and no-share models. In Fig. 4 (bottom) we observe an improvement of 2.5% on the half-share model, which is significant given that with 250 training categories this model had already achieved a high mAP.

4.4 Modality: Sketch-photo vs. Sketch-edgemap

We tested two network designs as depicted in Fig. 2: one sharing all FC layers (half-share), and another without sharing any layers (no-share). The training procedure for the two networks differs only during the second phase as described in Sec. 3.3. Table 1 reports the mAP scores when compared with the best score of the sketch-edgemap network and other baselines on the Flickr15K benchmark. Observe that: (i) All the deep-learning methods outperform the traditional methods using shallow-features by a large margin. (ii) There is also a dramatic improvement in performance of the sketch-photo networks over sketch-edgemap configurations. (iii) In the sketch-photo case, the no-share network now beats the half-share by mAP. Our third observation contrasts with the results of the sketch-edgemap evaluations (Fig. 4) where the half-share network performs more strongly. One explanation may be the significant difference between sketch and photo domains; the high-level concepts represented by the higher layers might not coincide (e.g. a sketched cat is different from a cat photo) due to abstraction, or caricaturing in the sketch itself.

Method mAP (%)
Triplet (fine-tuned final model) 36.17
Triplet (sketch-photo, no-share) 32.87
Triplet (sketch-photo, half-share) 31.38
Triplet (sketch-edgem, half-share) 24.45
Perceptual Edge [20] 18.37
Extended GF-HoG [4] 18.20
GF-HoG [15] 12.22
SHoG [11] 10.93
SSIM [26] 9.57
SIFT [19] 9.11
Shape Context [2] 8.14
Structure Tensor [10] 7.98
Table 1: SBIR comparison results (mAP) on the Flickr15K benchmark. The final model is achieved by fine-tuning the best model (sketch-photo, no-share) over the Sketchy dataset.

Fine-tuned final model

: our learned models so far were trained through the first 3 phases outlined in subsec. 3.3

, in which the training sketches and photos from TU-Berlin-Class have only class labels. In phase 4, we create the final model by fine-tuning our best trained model (i.e. sketch-photo, no-share triplet) using the recently released Sketchy dataset. Note that we do not require alignment between the category sets of Sketchy and TU-Berlin-Class since we perform regression using the triplet (not classification) loss only at this stage. This 4th stage adds fine-grained level of regression: each epoch is a pass-through of the training photos, for each photo image entering the positive branch we choose a random instance-level matching sketch entering the anchor branch and a random photo of the same category for the negative branch. The use of the Sketchy data delivers a final boost to our best case model of 3% mAP on the standard Flickr15K benchmark (Fig. 


4.5 Performance over SBIR benchmarks

It appears that the fine-grain labels of the Sketchy dataset (in contrast to the class-level labels already provided by the training set) allow our model to learn deeper representation of photos and sketches as well as their cross-domain mapping. The improvement can be seen clearer in Fig. 6, where we plot the Precision/Recall (PR) curves of the sketch-photo no-share triplet model (snapshot after phase 3 training, denoted as ), the final model ( finetuned with the Sketchy dataset, denoted as ) and one of the state-of-the-art non-deeplearning methods whose implementation code publicly available: extended GF-HoG [4] or . While the triplet sketch-photo no-share has higher mAP (blue line, mAP 32.87%) than the extended GF-HoG (green line, mAP 18.20%), the precision score of the first few retrieval results is actually lower for . After finetuning with the Sketchy dataset, our final model is able to surpass the precision score of the at every recall point.

Figure 5: Representative SBIR results on Flickr15K using (left) sketches and (right) images as queries. For each query, two sets of results are returned, one for intra-domain and the other for cross domain search. Non-relevant retrieved objects are outlined in red.
Figure 6: PR curve of the proposed triplet CNN (sketch-photo, c.f. Table 1) compared with a state-of-the-art non-deep learning method [15].
Triplet (fine-tuned final model) 37.4
Triplet (sketch-photo) 33.3
SHoG [11] 27.7
Triplet (sketch-edgemap) 22.3
HoG (global) [11] 22.3
Structure Tensor [10] 22.3
Spark [11] 21.7
HoG (local) [8] 17.5
Shape Context [2] 16.1
Table 2: SBIR comparison results (using Kendal’s rank correlation coefficient, ) on TU-Berlin-Retr dataset [11]. The triplet sketch-edgemap model has much lower score, likely due to poor gPb edge extraction (dataset is more cluttered than Flickr15K).

We also evaluated over TU-Berlin-Retr, using the metric (Table 2) as proposed by [11]. Interestingly the sketch-edgemap underperforms their proposed method SHoG, according to the new metric. We believe this is because TU-Berlin-Retr images are noisier than the ones on Flickr15K, which may have negative effects on the gPb edge extraction. Nevertheless, the final sketch-photo models outperform the baseline method SHoG by a significant margin.

5 Conclusion

We described the first deep learning algorithm for category level SBIR, comprising a triplet convnet trained using a query sketch anchor accompanied by positive and negative training photos harvested from the web. We presented comprehensive experiments exploring variants of our triplet convnet, contrasting appropriate strategies for weight sharing, dimensionality reduction, and training data pre-processing and reporting on the generalisation capabilities of the network. We reported the half-sharing triplet performs the best when learning matching between two close domains such as sketch-edgemap, while the no-share network is more suitable for sketch-photo learning. Our best performing variant yielded a performance of 36.2% mAP over the Flickr15k dataset and on the TU-Berlin dataset; the twin international benchmarks for category level SBIR [11, 15]. These scores exceed the state of the art by mAP and respectively. Training sketches were derived from the two largest available sketch datasets: the TU-Berlin dataset of Eitz et al. and the Sketchy dataset of Sangkloy et al. [24]. Further work might build upon this significant performance gain exploring multi-domain learning, for example sketch-photo-3D models mapping or multi-style work-art retrieval.

5.1 Limitations and Future Work

The limited class diversity and volume of sketch databases prevents us from exploring our generalisation test further than the most diverse i.e. 250 category dataset available (TU-Berlin-Class). Also, because of the unbalanced sketch and photo sets the sketch branch seems to be less discriminative than the photo branch. Fig. 5 depicts several intra- and cross-domain visual search examples where false positives often occur on sketch side. Unfortunately it is difficult to control the generalisation, discrimination and overfitting of an individual branch during training of the triplet network during regression of our SBIR embedding.

Finally, although deep learning method proves far better than the methods using shallow features, it comes with its own cost. It is a supervised learning method which require expensive labelled training data, as opposed to a free shallow-feature extractor. A deep network that can learn cross-domain mapping with minimal or no supervision is highly desirable, especially given the content diversity faced by SBIR “in the wild”.


  • [1] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5):898–916, 2011.
  • [2] S. Belongie, J. Malik, and J. Puzicha. Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(4):509–522, Apr 2002.
  • [3] A. D. Bimbo and P. Pala. Visual image retrieval by elastic matching of user sketches. IEEE Trans. Pattern Analysis and Machine Intelligence, 19(2):121–132, 1997.
  • [4] T. Bui and J. Collomosse. Scalable sketch-based image retrieval using color gradient features. In

    Proceedings of the IEEE International Conference on Computer Vision Workshops

    , pages 1–8, 2015.
  • [5] Y. Cao, C. Wang, L. Zhang, and L. Zhang. Edgel index for large-scale sketch-based image search. In Proc. CVPR, pages 761–768. IEEE, 2011.
  • [6] J. P. Collomosse, G. McNeill, and Y. Qian. Storyboard sketches for content based video retrieval. In Proc. ICCV, pages 245–252, 2009.
  • [7] J. P. Collomosse, G. McNeill, and L. Watts. Free-hand sketch grouping for video retrieval. In

    Intl. Conf on Pattern Recognition (ICPR)

    , 2008.
  • [8] N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In Proc. CVPR, volume 1, pages 886–893, 2005.
  • [9] M. Eitz, J. Hays, and M. Alexa. How do humans sketch objects? ACM Trans. Graph. (Proc. SIGGRAPH), 31(4):44:1–44:10, 2012.
  • [10] M. Eitz, K. Hildebrand, T. Boubekeur, and M. Alexa. A descriptor for large scale image retrieval based on sketched feature lines. In Proc. SBIM, pages 29–36, 2009.
  • [11] M. Eitz, K. Hildebrand, T. Boubekeur, and M. Alexa. Sketch-based image retrieval: Benchmark and bag-of-features descriptors. IEEE Trans. Visualization and Computer Graphics, 17(11):1624–1636, 2011.
  • [12] M. Flickner, H. Sawhney, W. Niblack, J. Ashley, Q. Huang, B. Dom, M. Gorkani, J. Hafner, D. Lee, and D. Petkovic. Query by image and video content: The qbic system. Computer, 28(9):23–32, 1995.
  • [13] A. Gordo, J. Almazan, J. Revaud, and D. Larlus. Deep image retrieval: Learning global representations for image search. arXiv preprint arXiv:1604.01325, 2016.
  • [14] R. Hu, M. Barnard, and J. P. Collomosse. Gradient field descriptor for sketch based retrieval and localization. In Proc. ICIP, volume 10, pages 1025–1028, 2010.
  • [15] R. Hu and J. Collomosse. A performance evaluation of gradient field HOG descriptor for sketch based image retrieval. Computer Vision and Image Understanding, 117(7):790–806, 2013.
  • [16] C. E. Jacobs, A. Finkelstein, and D. H. Salesin. Fast multiresolution image querying. In Proc. ACM SIGGRAPH, pages 277–286, 1995.
  • [17] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.
  • [18] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In Proc. NIPS, 2012.
  • [19] D. G. Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2):91–110, 2004.
  • [20] Y. Qi, Y.-Z. Song, T. Xiang, H. Zhang, T. Hospedales, Y. Li, and J. Guo. Making better use of edges via perceptual grouping. In Proc. CVPR, 2015.
  • [21] F. Radenović, G. Tolias, and O. Chum. Cnn image retrieval learns from bow: Unsupervised fine-tuning with hard examples. arXiv preprint arXiv:1604.02426, 2016.
  • [22] J. M. Saavedra and J. M. Barrios. Sketch based image retrieval using learned keyshapes. In Proc. BMVC, 2015.
  • [23] J. M. Saavedra and B. Bustos. Sketch-based image retrieval using keyshapes. Multimedia Tools and Applications, 73(3):2033–2062, 2014.
  • [24] P. Sangkloy, N. Burnell, C. Ham, and J. Hays. The sketchy database: learning to retrieve badly drawn bunnies. ACM Transactions on Graphics (TOG), 35(4):119, 2016.
  • [25] E. D. Sciascio, G. Mingolla, and M. Mongiello. Content-based image retrieval over the web using query by sketch and relevance feedback. In Visual Information and Information Systems, pages 123–130. Springer, 1999.
  • [26] E. Shechtman and M. Irani. Matching local self-similarities across images and videos. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8, June 2007.
  • [27] J. Song, Y.-Z. Song, T. Xiang, T. Hospedales, and X. Ruan. Deep multi-task attribute-driven ranking for fine-grained sketch-based image retrieval. In British Machine Vision Conference, 2016.
  • [28] J. Wang, Y. Song, T. Leung, C. Rosenberg, J. Wang, J. Philbin, B. Chen, and Y. Wu. Learning fine-grained image similarity with deep ranking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1386–1393, 2014.
  • [29] X. Wang and A. Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE International Conference on Computer Vision, pages 2794–2802, 2015.
  • [30] Q. Yu, F. Liu, Y.-Z. Song, T. Xiang, T. M. Hospedales, and C. C. Loy. Sketch me that shoe. In Proc. CVPR. IEEE, 2016.
  • [31] Q. Yu, Y. Yang, Y.-Z. Song, T. Xiang, and T. M. Hospedales. Sketch -a-net that beats humans. In Proc. BMVC. IEEE, 2015.