Mitigating Test-Time Bias for Fair Image Retrieval

by   Fanjie Kong, et al.

We address the challenge of generating fair and unbiased image retrieval results given neutral textual queries (with no explicit gender or race connotations), while maintaining the utility (performance) of the underlying vision-language (VL) model. Previous methods aim to disentangle learned representations of images and text queries from gender and racial characteristics. However, we show these are inadequate at alleviating bias for the desired equal representation result, as there usually exists test-time bias in the target retrieval set. So motivated, we introduce a straightforward technique, Post-hoc Bias Mitigation (PBM), that post-processes the outputs from the pre-trained vision-language model. We evaluate our algorithm on real-world image search datasets, Occupation 1 and 2, as well as two large-scale image-text datasets, MS-COCO and Flickr30k. Our approach achieves the lowest bias, compared with various existing bias-mitigation methods, in text-based image retrieval result while maintaining satisfactory retrieval performance. The source code is publicly available at <>.


page 1

page 2

page 3

page 4


Probabilistic Compositional Embeddings for Multimodal Image Retrieval

Existing works in image retrieval often consider retrieving images with ...

VisualSparta: Sparse Transformer Fragment-level Matching for Large-scale Text-to-Image Search

Text-to-image retrieval is an essential task in multi-modal information ...

Fair Generative Modeling via Weak Supervision

Real-world datasets are often biased with respect to key demographic fac...

What is a Fair Diffusion Model? Designing Generative Text-To-Image Models to Incorporate Various Worldviews

Generative text-to-image (GTI) models produce high-quality images from s...

Leaner and Faster: Two-Stage Model Compression for Lightweight Text-Image Retrieval

Current text-image approaches (e.g., CLIP) typically adopt dual-encoder ...

Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking

In this paper we address issues with image retrieval benchmarking on sta...

Please sign up or login with your details

Forgot password? Click here to reset