Non-deterministic Behavior of Ranking-based Metrics when Evaluating Embeddings

06/19/2018
by   Anguelos Nicolaou, et al.
0

Embedding data into vector spaces is a very popular strategy of pattern recognition methods. When distances between embeddings are quantized, performance metrics become ambiguous. In this paper, we present an analysis of the ambiguity quantized distances introduce and provide bounds on the effect. We demonstrate that it can have a measurable effect in empirical data in state-of-the-art systems. We also approach the phenomenon from a computer security perspective and demonstrate how someone being evaluated by a third party can exploit this ambiguity and greatly outperform a random predictor without even access to the input data. We also suggest a simple solution making the performance metrics, which rely on ranking, totally deterministic and impervious to such exploits.

READ FULL TEXT

page 4

page 7

page 9

08/18/2018

CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement

Typically crowdsourcing-based approaches to gather annotated data use in...
01/19/2021

Ambiguity of Objective Image Quality Metrics: A New Methodology for Performance Evaluation

Objective image quality metrics try to estimate the perceptual quality o...
09/07/2020

Ambiguity Hierarchy of Regular Infinite Tree Languages

An automaton is unambiguous if for every input it has at most one accept...
06/10/2012

Comments on "On Approximating Euclidean Metrics by Weighted t-Cost Distances in Arbitrary Dimension"

Mukherjee (Pattern Recognition Letters, vol. 32, pp. 824-831, 2011) rece...
03/31/2020

Information Leakage in Embedding Models

Embeddings are functions that map raw input data to low-dimensional vect...
01/15/2020

Unbiased evaluation of ranking metrics reveals consistent performance in science and technology citation data

Despite the increasing use of citation-based metrics for research evalua...
02/22/2021

Approximate Knowledge Graph Query Answering: From Ranking to Binary Classification

Large, heterogeneous datasets are characterized by missing or even erron...