Relaxed forced choice improves performance of visual quality assessment methods

04/29/2023
by   Mohsen Jenadeleh, et al.
0

In image quality assessment, a collective visual quality score for an image or video is obtained from the individual ratings of many subjects. One commonly used format for these experiments is the two-alternative forced choice method. Two stimuli with the same content but differing visual quality are presented sequentially or side-by-side. Subjects are asked to select the one of better quality, and when uncertain, they are required to guess. The relaxed alternative forced choice format aims to reduce the cognitive load and the noise in the responses due to the guessing by providing a third response option, namely, “not sure”. This work presents a large and comprehensive crowdsourcing experiment to compare these two response formats: the one with the “not sure” option and the one without it. To provide unambiguous ground truth for quality evaluation, subjects were shown pairs of images with differing numbers of dots and asked each time to choose the one with more dots. Our crowdsourcing study involved 254 participants and was conducted using a within-subject design. Each participant was asked to respond to 40 pair comparisons with and without the “not sure” response option and completed a questionnaire to evaluate their cognitive load for each testing condition. The experimental results show that the inclusion of the “not sure” response option in the forced choice method reduced mental load and led to models with better data fit and correspondence to ground truth. We also tested for the equivalence of the models and found that they were different. The dataset is available at http://database.mmsp-kn.de/cogvqa-database.html.

READ FULL TEXT
research
01/10/2020

Subjective Annotation for a Frame Interpolation Benchmark using Artifact Amplification

Current benchmarks for optical flow algorithms evaluate the estimation e...
research
01/16/2019

Technical Report on Visual Quality Assessment for Frame Interpolation

Current benchmarks for optical flow algorithms evaluate the estimation q...
research
11/09/2022

Content-Diverse Comparisons improve IQA

Image quality assessment (IQA) forms a natural and often straightforward...
research
11/25/2020

Evaluation of quality measures for color quantization

Visual quality evaluation is one of the challenging basic problems in im...
research
05/13/2022

AVCAffe: A Large Scale Audio-Visual Dataset of Cognitive Load and Affect for Remote Work

We introduce AVCAffe, the first Audio-Visual dataset consisting of Cogni...
research
10/23/2020

Origins of Algorithmic Instabilities in Crowdsourced Ranking

Crowdsourcing systems aggregate decisions of many people to help users q...

Please sign up or login with your details

Forgot password? Click here to reset