DeepAI AI Chat
Log In Sign Up

Active Sampling for Pairwise Comparisons via Approximate Message Passing and Information Gain Maximization

04/12/2020
by   Aliaksei Mikhailiuk, et al.
University of Cambridge
Microsoft
UCL
0

Pairwise comparison data arise in many domains with subjective assessment experiments, for example in image and video quality assessment. In these experiments observers are asked to express a preference between two conditions. However, many pairwise comparison protocols require a large number of comparisons to infer accurate scores, which may be unfeasible when each comparison is time-consuming (e.g. videos) or expensive (e.g. medical imaging). This motivates the use of an active sampling algorithm that chooses only the most informative pairs for comparison. In this paper we propose ASAP, an active sampling algorithm based on approximate message passing and expected information gain maximization. Unlike most existing methods, which rely on partial updates of the posterior distribution, we are able to perform full updates and therefore much improve the accuracy of the inferred scores. The algorithm relies on three techniques for reducing computational cost: inference based on approximate message passing, selective evaluations of the information gain, and selecting pairs in a batch that forms a minimum spanning tree of the inverse of information gain. We demonstrate, with real and synthetic data, that ASAP offers the highest accuracy of inferred scores compared to the existing methods. We also provide an open-source GPU implementation of ASAP for large-scale experiments.

READ FULL TEXT
10/01/2020

Strategy for Boosting Pair Comparison and Improving Quality Assessment Accuracy

The development of rigorous quality assessment model relies on the colle...
05/23/2019

Replicated Vector Approximate Message Passing For Resampling Problem

Resampling techniques are widely used in statistical inference and ensem...
12/07/2017

Sketched Clustering via Hybrid Approximate Message Passing

In sketched clustering, the dataset is first sketched down to a vector o...
08/12/2011

Compressive Imaging using Approximate Message Passing and a Markov-Tree Prior

We propose a novel algorithm for compressive imaging that exploits both ...
11/16/2017

HodgeRank with Information Maximization for Crowdsourced Pairwise Ranking Aggregation

Recently, crowdsourcing has emerged as an effective paradigm for human-p...
10/20/2018

Hybrid-MST: A Hybrid Active Sampling Strategy for Pairwise Preference Aggregation

In this paper we present a hybrid active sampling strategy for pairwise ...

Code Repositories

asap

Fast and accurate Active ASmpling method for Pairwise comparisons


view repo