    # Optimal Strategies for Matching and Retrieval Problems by Comparing Covariates

In many retrieval problems, where we must retrieve one or more entries from a gallery in response to a probe, it is common practice to learn to do by directly comparing the probe and gallery entries to one another. In many situations the gallery and probe have common covariates -- external variables that are common to both. In principle it is possible to perform the retrieval based merely on these covariates. The process, however, becomes gated by our ability to recognize the covariates for the probe and gallery entries correctly. In this paper we analyze optimal strategies for retrieval based only on matching covariates, when the recognition of the covariates is itself inaccurate. We investigate multiple problems: recovering one item from a gallery of N entries, matching pairs of instances, and retrieval from large collections. We verify our analytical formulae through experiments to verify their correctness in practical settings.

## Authors

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

This document is currently incomplete, and has been uploaded primarily as a supporting document for . The completed version will be uploaded shortly

Consider the following problem: we are given a “gallery” of items and a single “probe” entry, which we expect “matches” some of the g entries, in some sense. Our task is to retrieve the gallery entries that match the probe.

A typical problem, for instance, is when we are given a gallery of biometric identifiers, such as faces, and a probe instance, which is also a biometric instance (e.g. another face image, or even any other modality such as fingerprint or voice). We must retrieve the appropriate gallery entries that are from the same person as the probe entry. Alternately, we may be given a gallery of documents by a number of authors, and a probe document of an unknown author. We must find the gallery entries that match the probe. Many other such problems can be found.

In these problems, the general solution is to find statistical dependencies between entries that relate the two types of data (from the probe and the gallery) and recover the matching entries based on these. Typically, the solution comprises considering some variant of , or for each of the gallery entries, and determining the match based on this value 

. This probability itself may utilize any kind of underlying statistical model. These joint models must often be learned from joint presentation of the types of data present in the probe and the gallery.

Often, however, we can find common covariates to the probe and gallery data, which can be independently determined. For instance, in biometric identification, the gender, ethnicity, nationality, and even characteristics such as body size, affect both the probe and gallery entries. In the document case, the gender and nationality of the author, the writing style, etc., affect the probe and gallery entries.

We expect the covariate values for probe and its matching gallery entries to be identical.

The key feature here is that these covariates, being known entities, may be independently determined from both probe and gallery entries. For instance, in the biometric problem the gender or ethnicity of a subject may be independently determined for both of them. In the document problem, the gender, nationality, and writing style of the author can be independently determined for the probe and gallery entries. To learn to identify these covariate characteristics, the joint distribution of probe and gallery data need not be considered at all.

The question we address here how accurate

can retrieval be when it is based only on matching the covariate information of probe and gallery entries, i.e., if the only information used for the retrieval was the estimated covariate values of the probe and gallery data. E.g., in recovering the correct face from a gallery, how accurate would we be if all we did was to match the estimated gender of the probe and gallery entries.

We analyze this problem in a number of different settings.

• Retrieval of unique match from a gallery of . Here we assume a gallery of entries, where exactly one of the gallery entries matches the probe.

• Verification. The gallery comprises only one entry. I.e., given two data instances, one nominally the probe, and the other the gallery, we must determine if they both match or not.

In all of these settings we will derive an optimal policy for identifying gallery matches to the probe, and the error to be expected in following this policy.

We will make some simplifying assumptions. We assume that the gallery entries are all independently drawn and do not inform about one another. We assume only a single probe entry. Also, although we assume that only a single covariate is considered at any time; this is not a real restriction – groups of covariates fall into the same analysis by simply considering the group as a single extended covariate.

We will assume that the “imposter” entries in the gallery, i.e. gallery entries that do not match the probe, are drawn independently of the probe. We will also assume that the errors in determining the covariate values of gallery entries are independent of the errors made on probe.

## 2 Retrieval of a Unique Match from a Gallery Of N

We first consider the problem of retrieval from a gallery of entries, where it is known that exactly one of the entries is a match to the probe.

Consider a covariate that can take values in the set . We will assume for this document that is discrete, although this is not necessary.

Figure 1 displays our model. We model the automatic classification of the covariate values for probe instances as a noisy channel . We model the automatic classification of the covariate values for gallery instances as a noisy channel .

The overall model has the following statistical components:

• from which probe entries with covariates are selected.

• A probability distribution which specifies the probability that the noisy channel will ouput the value in response to input .

• A probability distribution from which “imposter” gallery entries are drawn.

• A probability distribution which specifies the probability that the noisy channel will ouput the value in response to input .

We assume all of these distributions are known.

The generative model for the process is as follows:

• The probe entry is drawn according to probability .

• is passed through the channel , which outputs a noisy covariate in response.

• The probe entry is passed through the channel , which outputs the noisy covariate in response. is added to the gallery as the matching entry to the probe.

• To fill the rest of the gallery of size , additional entries with covariates are drawn independently according to . Each of these is passed through the channel to obtain the noisy covariate , which is included in the gallery.

Note that using the known distributions, we can also compute the following terms:

• The overall probability of observing a noisy probe value :

 PQH(~cp)=∑cpPQ(cp)PH(~cp|cp)
• The a posteriori probability of the true probe covariate , given the noisy value

 PQH(cp|~cp)=PQ(cp)PH(~cp|cp)PQH(~cp)
• The overall probability that any particular gallery item (other than the entry matching the probe) will take a specific value . From the above formulation, we have

 PGJ(~cg)=∑cgPG(cg)PJ(~cg|cg)

and

 PGJ(cg|~cg)=PG(cg)PJ(~cg|cg)PGJ(~cg)

### 2.1 Defining a Policy for the Matching

We will consider a stochastic policy where, given an output from , we select a covariate value according to a probability distribution , and subsequently select one of the gallery entries for which . Note that this is a generalization of the more conventional deterministic policy (which would return a unique in response to each . As we shall see, however, the optimal strategy is indeed deterministic).

### 2.2 Probability of Correctness as a Function of Policy

If there are gallery entries for which , then the probability of correctly matching the probe, given that the original probe entry was , is given by

 P(correct|cp,csel,K)=PJ(csel|cp)1K. (1)

This factors in both, the probability that the output of the noisy channel in response to is , and that we have chosen the correct instance from the gallery items for which .

The probability that exactly of the gallery items will have value , given that the matching entry is also is given by

 P(K)=B(N−1,K−1,PGJ(csel)), (2)

where is the binomial probability or choosing of entries, with probability parameter :

 B(N,K,p)=(NK)pK(1−p)N−K.

Equation 2 considers the fact that if we are given that one of the is the matching entry, we must only account for the ways in which of the remaining gallery entries can also be .

The overall probability of correctness of the response when we choose is

 P(correct|cp,csel)=PJ(csel|cp)N−1∑K=0P(K)1K, =PJ(csel|cp)N∑K=1B(N−1,K−1,PGJ(csel))K, =PJ(csel|cp)NPGJ(csel)N∑K=1B(N,K,PGJ(csel)) =PJ(csel|cp)(1−PGJ(csel))NNPGJ(csel). (3)

Using the law of iterated expectations we can now write the overall probability of correctness of the response, given a noisy probe as

 P(correct|~cp) =Ecsel|~cpEcp|~cpP(correct|cp,csel) =∑cselr(csel|~cp)∑cpPQH(cp|~cp) PJ(csel|cp)(1−PGJ(csel))NNPGJ(csel). (4)

### 2.3 Optimal Policy

Our objective is to find the policy that maximizes the probability of correctness for any probe :

 argmax{r(csel|~cp)}P(correct|~cp)

Define as

 ^c(~cp)=argmaxcsel(∑cpPQH(cp|~cp)PJ(csel|cp)(1−~PG(csel))NN~PG(csel))

From inspection of Equation 4 we obtain the following optimal policy.

 r(csel|~cp)={1,  csel=^csel(~cp)0  else. (5)

### 2.4 Optimal Error

Given any noisy probe , the probability error under the optimal policy is given by

 Popt(error|~cp)=1− maxcsel(∑cpPQH(cp|~cp)PJ(csel|cp)(1−~PG(csel))NN~PG(csel)) (6)

The overall probability of error is given by

 Popt(error)=1−∑PQH(~cp)Popt(error|~cp) =1− maxcsel⎛⎝∑cp,~cpPQ(cp)PH(~cp|cp)PJ(csel|cp)(1−~PG(csel))NN~PG(csel)⎞⎠ (7)

## 3 The Verification Problem

Figure 2 shows our model for the verification problem. We have two conditions: “match” and “mismatch”. Under match, a single covariate is drawn from and passed through the two noisy channels and to produce the probe entry and the gallery entry . Under mismatch, and are drawn independently from and respectively and passed through and to produce and .

From observing and we must determine which of the two conditions, or , produced them.

### 3.1 Defining The Error

To analyze the problem we must first define the error of matching apporpriately.

When we wrongly identify a case of match as a mismatch (i.e. we “reject” a match), we have an instance of a false rejection. When a mismatch is misidentified as a match (i.e. we “accept” a mismatch), we have a false acceptance.

Let represent the “false rejection rate”, i.e. the probability that a match will be wrongly rejected. Let

represent the “false acceptance rate”, i.e. the probability that a negative match will be wrongly accepted. Any classifier can generally be optimized to trade off

against . The “Equal Error Rate” (EER) is achieved when , i.e. (or ) when .

We will choose as our objective the minimization of the EER. Note that if an operating point other than EER is chosen to quantify performance (e.g. for , or for some fixed or ), the analysis below can generally be modified to accommodate it, provided a feasible solution exists.

### 3.2 Defining a Policy

We will use the following stochastic policy: for any pair of noisy probe and gallery values, and , we will accept the pair as a match with probability . We must find the that minimizes the EER.

### 3.3 Error as a Function of Policy

We first define the probabilities of observing any given pair under conditions of match and mismatch. From the model of Figure 2 we get the following probability under match:

 P(~cp,~cg|match) =∑cpP(~cp,~cg,cp|match), =∑cpPH(~cp|cp)PJ(~cg|cp)PQ(cp). (8)

Above we’re utilizing the fact that and are conditionally independent of , given .

Similarly, from Figure 2, the probability of any under mismatch is given by

 P(~cp,~cg|mismatch) =∑cp,cgP(~cp,~cg,cp,cg|mismatch) =∑cp,cgPH(~cp|cp)PJ(~cg|cg)PQ(cp)PJ(cg) (9)

The probability of a false acceptance is given by

 FA =P(accept|mismatch) =∑~cp,~cqr(~cp,~cg)P(~cp,~cg|mismatch) (10)

The probability of a false rejection is given by

 FR =P(reject|match) =∑~cp,~cq(1−r(~cp,~cg))P(~cp,~cg|match) (11)

We obtain EER when , i.e.

 ∑~cp,~cqr(~cp,~cg)P(~cp,~cg|mismatch)=∑~cp,~cq(1−r(~cp,~cg))P(~cp,~cg|match) ⟹∑~cp,~cg(P(~cp,~cg|mismatch)+P(~cp,~cg|match))r(~cp,~cg)=1 (12)

Thus, optimizing the policy requires solving the following

 argmin{r(~cp,~cg)}∑~cp,~cqr(~cp,~cg)P(~cp,~cg|mismatch)s.t.1≥r(~cp,~cg)≥0,∑~cp,~cg(P(~cp,~cg|mismatch)+P(~cp,~cg|match))r(~cp,~cg)=1.