Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval

03/28/2022
by   Ayan Kumar Bhunia, et al.
12

Sketching enables many exciting applications, notably, image retrieval. The fear-to-sketch problem (i.e., "I can't sketch") has however proven to be fatal for its widespread adoption. This paper tackles this "fear" head on, and for the first time, proposes an auxiliary module for existing retrieval models that predominantly lets the users sketch without having to worry. We first conducted a pilot study that revealed the secret lies in the existence of noisy strokes, but not so much of the "I can't sketch". We consequently design a stroke subset selector that detects noisy strokes, leaving only those which make a positive contribution towards successful retrieval. Our Reinforcement Learning based formulation quantifies the importance of each stroke present in a given subset, based on the extent to which that stroke contributes to retrieval. When combined with pre-trained retrieval models as a pre-processing module, we achieve a significant gain of 8 new state-of-the-art performance. Last but not least, we demonstrate the selector once trained, can also be used in a plug-and-play manner to empower various sketch applications in ways that were not previously possible.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/22/2021

Sketch-QNet: A Quadruplet ConvNet for Color Sketch-based Image Retrieval

Architectures based on siamese networks with triplet loss have shown out...
research
02/24/2020

Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval

Fine-grained sketch-based image retrieval (FG-SBIR) addresses the proble...
research
04/18/2019

Generative Model for Zero-Shot Sketch-Based Image Retrieval

We present a probabilistic model for Sketch-Based Image Retrieval (SBIR)...
research
11/21/2021

Deep Reinforced Attention Regression for Partial Sketch Based Image Retrieval

Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) aims at finding a sp...
research
12/10/2020

Sketchy With a Chance of Adoption: Can Sketch-Based Telemetry Be Ready for Prime Time?

Sketching algorithms or sketches have emerged as a promising alternative...
research
04/20/2021

Compact and Effective Representations for Sketch-based Image Retrieval

Sketch-based image retrieval (SBIR) has undergone an increasing interest...
research
02/24/2020

Sketchformer: Transformer-based Representation for Sketched Structure

Sketchformer is a novel transformer-based representation for encoding fr...

Please sign up or login with your details

Forgot password? Click here to reset