Evaluating language-biased image classification based on semantic representations

01/26/2022
by   Yoann Lemesle, et al.
2

Humans show language-biased image recognition for a word-embedded image, known as picture-word interference. Such interference depends on hierarchical semantic categories and reflects that human language processing highly interacts with visual processing. Similar to humans, recent artificial models jointly trained on texts and images, e.g., OpenAI CLIP, show language-biased image classification. Exploring whether the bias leads to interferences similar to those observed in humans can contribute to understanding how much the model acquires hierarchical semantic representations from joint learning of language and vision. The present study introduces methodological tools from the cognitive science literature to assess the biases of artificial models. Specifically, we introduce a benchmark task to test whether words superimposed on images can distort the image classification across different category levels and, if it can, whether the perturbation is due to the shared semantic representation between language and vision. Our dataset is a set of word-embedded images and consists of a mixture of natural image datasets and hierarchical word labels with superordinate/basic category levels. Using this benchmark test, we evaluate the CLIP model. We show that presenting words distorts the image classification by the model across different category levels, but the effect does not depend on the semantic relationship between images and embedded words. This suggests that the semantic word representation in the CLIP visual processing is not shared with the image representation, although the word representation strongly dominates for word-embedded images.

READ FULL TEXT

page 2

page 7

page 11

page 12

page 13

page 14

page 15

page 18

research
12/08/2022

Task Bias in Vision-Language Models

Incidental supervision from language has become a popular approach for l...
research
11/22/2022

Semantic Guided Level-Category Hybrid Prediction Network for Hierarchical Image Classification

Hierarchical classification (HC) assigns each object with multiple label...
research
04/02/2020

Hierarchical Image Classification using Entailment Cone Embeddings

Image classification has been studied extensively, but there has been li...
research
04/02/2020

What did we learn from forty years of research on semantic interference? A Bayesian metaanalysis

When participants in an experiment have to name pictures while ignoring ...
research
12/13/2021

Context vs Target Word: Quantifying Biases in Lexical Semantic Datasets

State-of-the-art contextualized models such as BERT use tasks such as Wi...
research
01/03/2019

A Hierarchical Grocery Store Image Dataset with Visual and Semantic Labels

Image classification models built into visual support systems and other ...

Please sign up or login with your details

Forgot password? Click here to reset