Partially Does It: Towards Scene-Level FG-SBIR with Partial Input

03/28/2022
by   Pinaki Nath Chowdhury, et al.
0

We scrutinise an important observation plaguing scene-level sketch research – that a significant portion of scene sketches are "partial". A quick pilot study reveals: (i) a scene sketch does not necessarily contain all objects in the corresponding photo, due to the subjective holistic interpretation of scenes, (ii) there exists significant empty (white) regions as a result of object-level abstraction, and as a result, (iii) existing scene-level fine-grained sketch-based image retrieval methods collapse as scene sketches become more partial. To solve this "partial" problem, we advocate for a simple set-based approach using optimal transport (OT) to model cross-modal region associativity in a partially-aware fashion. Importantly, we improve upon OT to further account for holistic partialness by comparing intra-modal adjacency matrices. Our proposed method is not only robust to partial scene-sketches but also yields state-of-the-art performance on existing datasets.

READ FULL TEXT

page 7

page 8

research
05/28/2017

Cross-modal Subspace Learning for Fine-grained Sketch-based Image Retrieval

Sketch-based image retrieval (SBIR) is challenging due to the inherent d...
research
07/29/2020

Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval

Sketch as an image search query is an ideal alternative to text in captu...
research
11/21/2021

Deep Reinforced Attention Regression for Partial Sketch Based Image Retrieval

Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) aims at finding a sp...
research
09/15/2023

Active Learning for Fine-Grained Sketch-Based Image Retrieval

The ability to retrieve a photo by mere free-hand sketching highlights t...
research
03/25/2021

More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval

A fundamental challenge faced by existing Fine-Grained Sketch-Based Imag...
research
08/16/2021

Scene Designer: a Unified Model for Scene Search and Synthesis from Sketch

Scene Designer is a novel method for searching and generating images usi...
research
11/30/2022

CLIPascene: Scene Sketching with Different Types and Levels of Abstraction

In this paper, we present a method for converting a given scene image in...

Please sign up or login with your details

Forgot password? Click here to reset