Multi-granularity Association Learning Framework for on-the-fly Fine-Grained Sketch-based Image Retrieval

by   Dawei Dai, et al.

Fine-grained sketch-based image retrieval (FG-SBIR) addresses the problem of retrieving a particular photo in a given query sketch. However, its widespread applicability is limited by the fact that it is difficult to draw a complete sketch for most people, and the drawing process often takes time. In this study, we aim to retrieve the target photo with the least number of strokes possible (incomplete sketch), named on-the-fly FG-SBIR (Bhunia et al. 2020), which starts retrieving at each stroke as soon as the drawing begins. We consider that there is a significant correlation among these incomplete sketches in the sketch drawing episode of each photo. To learn more efficient joint embedding space shared between the photo and its incomplete sketches, we propose a multi-granularity association learning framework that further optimizes the embedding space of all incomplete sketches. Specifically, based on the integrity of the sketch, we can divide a complete sketch episode into several stages, each of which corresponds to a simple linear mapping layer. Moreover, our framework guides the vector space representation of the current sketch to approximate that of its later sketches to realize the retrieval performance of the sketch with fewer strokes to approach that of the sketch with more strokes. In the experiments, we proposed more realistic challenges, and our method achieved superior early retrieval efficiency over the state-of-the-art methods and alternative baselines on two publicly available fine-grained sketch retrieval datasets.



There are no comments yet.


page 4


Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval

Fine-grained sketch-based image retrieval (FG-SBIR) addresses the proble...

Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval

Sketch as an image search query is an ideal alternative to text in captu...

Deep Reinforced Attention Regression for Partial Sketch Based Image Retrieval

Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) aims at finding a sp...

Generic Sketch-Based Retrieval Learned without Drawing a Single Sketch

We cast the sketch-based retrieval as edge-map matching. A shared convol...

Learning to Sketch with Shortcut Cycle Consistency

To see is to sketch -- free-hand sketching naturally builds ties between...

Deep Factorised Inverse-Sketching

Modelling human free-hand sketches has become topical recently, driven b...

Compact and Effective Representations for Sketch-based Image Retrieval

Sketch-based image retrieval (SBIR) has undergone an increasing interest...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.