Conformal testing: binary case with Markov alternatives

1 Introduction

This note treats a problem similar to the one considered in [2]: we would like to test online the null hypothesis of exchangeability of binary observations under Markov alternatives.

The simplest way of online hypothesis testing is to use test martingales, which are defined as nonnegative processes with initial value 1 that are martingales under the null hypothesis; see, e.g., [3]. Such processes, for the null hypothesis of exchangeability, can be constructed using the method of conformal prediction [6], and we will refer to them as conformal test martingales. A previous paper [4] constructs custom-made conformal test martingales for different alternative hypotheses, those of a changepoint.

The method of [2], which is specifically devoted to Markov alternatives, is more general: instead of a test martingale the authors construct a “safe e-process” (to be defined in the next section). Safe e-processes are closely related to test martingales and admit a similar interpretation as the capital of a gambler trying to discredit the null hypothesis. Our methods give similar results to the methods of [2] in the model situations that we consider (following [2]

). The advantage of our methods is that they extend easily to the usual setting of machine learning, where the observations are pairs

(x, y)

consisting of a potentially complex object

x

and its label

y

In this note we only design conformal test martingales for a simple alternative hypothesis (a specific probability measure). This is different from

[2], who are interested in testing against the composite alternative Markov hypothesis. As in [2], we could mix our conformal test martingales over the possible alternative hypotheses, but we leave this step for future research.

2 Model situations

Figure 1: The process $R$ of [2] and the Simple Jumper in the large scenario. Left panel: the hard case. Right panel: the easy case.

This section introduces the model situations considered in this paper, following [2, Section 4.2]

. Our data consist of binary observations generated from a Markov model. We will use the notation

Markov (π_{1 | 0}, π_{1 | 1})

for the probability distribution of a Markov chain with the transition probabilities

π_{1 | 0}

for transitions

0 \to 1

and

π_{1 | 1}

for transitions

1 \to 1

; the probability that the first observation is 1 will always be assumed

0.5

. In the hard case, the model is

Markov (0.4, 0.6)

, and in the easy case, the model is

Markov (0.1, 0.9)

. The number of observations is

N := 10^{4}

(as in [2]) or

N := 10^{3}

N := 10^{2}

; we will refer to these scenarios as large, medium, and small, respectively.

In all our experiments we use 2021 as the seed for the NumPy pseudorandom number generator. (This, however, does not make the trajectories in our plots comparable between different scenarios.) The dependence on the seed will be explored in boxplots reported in Section 5; the seed affects not only the data but also the values of conformal martingales, which are randomized processes, given the data.

Let $B_{π}$

be the Bernoulli distribution on

{0, 1}

with parameter

π \in [0, 1]

B_{π} ({1}) = π

. Set

Ber (π) := B_{π}^{\infty}

. Our null hypothesis is the IID model, under which the observations are generated from

Ber (π)

with unknown parameter

π

Ramdas et al. construct a safe e-process $R = R_{n}$ : namely, under any $Ber (π)$ , $R$ is dominated by a test martingale $M_{n}^{(π)}$ w.r. to $Ber (π)$ , in the sense that $R_{n} \leq M_{n}^{(π)}$ for all $n$ and $π$ . The trajectories of their process for the two cases, hard and easy, are shown in Figure 1 (they coincides with those in Figure 4 in [2] apart from using base 10 logarithms and a different randomly generated dataset). The figure also shows trajectories of the Simple Jumper martingale (see, e.g., [5]) for various values of the jumping rate; it performs poorly in this context.

3 Two benchmarks

In this section we will discuss possible benchmarks that we can use for evaluating the quality of our conformal test martingales. The upper benchmark is

{UB}_{n} := \frac{Markov (π_{1 | 0}, π_{1 | 1}) ([z_{1}, \dots, z_{n}])}{Ber (0.5) ([z_{1}, \dots, z_{n}])},

where $[z_{1}, \dots, z_{n}]$ is the set of all infinite sequences of binary observations starting from $z_{1}, \dots, z_{n}$ , and $z_{1}, z_{2}, \dots$ are the actual observations. The lower benchmark is

{LB}_{n} := \frac{Markov (π_{1 | 0}, π_{1 | 1}) ([z_{1}, \dots, z_{n}])}{Ber (^π) ([z_{1}, \dots, z_{n}])},

where $^π := k / n$

(the maximum likelihood estimate) and

k = k (n)

is the number of 1s among

z_{1}, \dots, z_{n}

. By definition,

{UB}_{0} = {LB}_{0} := 1

Figure 2: The two benchmarks, $R$ process, Bayes–Kelly conformal test martingale, and its simplified version in the large scenario. Left panel: hard case. Right panel: easy case.

Figure 3: The analogue of Figure 2 for the last 1000 observations.

The trajectories of the upper and lower benchmarks are shown in Figure 2 in red and green; the figure also shows the trajectory the $R$ process discussed in the previous section, and the other two trajectories should be ignored for now. The two benchmarks coincide or almost coincide. Figure 3 should the same trajectories “under the lens”, over the last 1000 observations.

4 Bayesian conformal testing

In this section we will use a Bayesian method that is statistically efficient in our experiments but whose computational efficiency will be greatly improved in the next section. The p-values $p_{1}, p_{2}, \dots$ are generated as described in [4]; in particular, we are using the identity nonconformity measure (the nonconformity score of an observation $z$ is $z$ ). Under the alternative hypothesis, the p-values are generated by a completely specified stochastic mechanism. According to [1, Theorem 2], the optimal (in the Kelly-type sense of that paper) betting functions $f_{n}$ are given by the density of the predictive distribution of $p_{n}$ conditional on knowing $p_{1}, \dots, p_{n - 1}$ . Let us find these predictive distributions. We will use the notation $U [a, b]$ , where $a < b$ , for the uniform probability distribution on the interval $[a, b]$ (so that its density is $1 / (b - a)$ ).

We are in a typical situation of Bayesian statistics. The Bayesian parameter is the binary sequence

(z_{1}, z_{2}, \dots) \in {0, 1}^{\infty}

of observations, and the prior distribution on the parameter is

Markov (π_{1 | 0}, π_{1 | 1})

. The Bayesian observations are the conformal p-values

p_{1}, p_{2}, \dots

. Given the parameter, the distribution of

p_{n}

p_{n} \sim {\begin{matrix} U [0, k / n] & if z_{n} = 1 U [k / n, 1] & if z_{n} = 0, \end{matrix}

where $k := z_{1} + \dots + z_{n}$ is the number of 1s among the first $n$ observations.

Let $w_{k, j}^{n}$ , where $n = 1, 2, \dots$ , $k = 0, \dots, n$ , and $j \in {0, 1}$

, be the total posterior probability of the parameter values

z_{1}, z_{2}, \dots

for which

z_{1} + \dots + z_{n} = k

and

z_{n} = j

; we will use them as the weights when computing the predictive distributions for the p-values. We can compute the weights

w_{k, j}^{n}

recursively in

n

as follows. We start from

w_{0, 0}^{1} := w_{1, 1}^{1} := 0.5, w_{0, 1}^{1} := w_{1, 0}^{1} := 0.

At each step $n \geq 2$ , first we compute the unnormalized weights

	${~ w}_{k, 0}^{n}$	$:= (w_{k, 0}^{n - 1} π_{0 \| 0} + w_{k, 1}^{n - 1} π_{1 \| 0}) l_{k}^{n - 1} (0, p_{n}),$
	${~ w}_{k, 1}^{n}$	$:= (w_{k - 1, 0}^{n - 1} π_{1 \| 0} + w_{k - 1, 1}^{n - 1} π_{1 \| 1}) l_{k - 1}^{n - 1} (1, p_{n}),$

where $l$ is the likelihood defined by

	$l_{k}^{n} (1, p)$	$:= {\begin{matrix} \frac{n + 1}{k + 1} & if p \leq \frac{k + 1}{n + 1} 0 & otherwise, \end{matrix}$
	$l_{k}^{n} (0, p)$	$:= {\begin{matrix} \frac{n + 1}{n - k + 1} & if p \geq \frac{k}{n + 1} 0 & otherwise, \end{matrix}$

and then we normalize them:

w_{k}^{n} := {~ w}_{k}^{n} / m \sum k = 0 1 \sum j = 0 {~ w}_{k, j}^{n} .

Given the posterior weights for the previous step, we can find the predictive distribution for $p_{n}$ as

p_{n} \sim n - 1 \sum k = 0 1 \sum j = 0 w_{k, j}^{n - 1} (π_{1 | j} U [0, \frac{k + 1}{n}] + π_{0 | j} U [\frac{k}{n}, 1]),

where we use the shorthand $π_{0 | j} := 1 - π_{1 | j}$ . Therefore, the betting functions for the resulting Bayes–Kelly conformal test martingale are

f_{n} (p) = n - 1 \sum k = 0 1 \sum j = 0 w_{k, j}^{n - 1} (\frac{n}{k + 1} π_{1 | j} 1_{p \leq \frac{k + 1}{n}} + \frac{n}{k} π_{0 | j} 1_{p \geq \frac{k}{n}}) .

(1)

Figure 4: The Bayes–Kelly and Bayes–Kelly simplified conformal test martingales, the $R$ -process, and the two benchmarks in the middle scenario. Left panel: hard case. Right panel: easy case.

For experimental results, see Figure 4, in addition to Figure 2. The Bayes–Kelly conformal test martingale appears to be very close to the two benchmarks. Its simplified version is described in the next section. The relatively poor performance of the $R$ -process in the left panel of Figure 4 should not be interpreted as it being inferior to the Bayes–Kelly conformal test martingale: remember that $R$ works against all Markov alternatives, whereas the other processes in Figures 2–8 are adapted to the specific alternative hypothesis ( $Markov (0.4, 0.6)$ in the hard case and $Markov (0.1, 0.9)$ in the easy case).

5 Simplified Bayesian conformal testing

Figure 5: The weights $w_{k}^{1000}$ , $k = 0, \dots, 1000$ , at the last step for the Bayes–Kelly conformal test martingale in the medium scenario (the hard case on the left and easy on the right).

In this section we consider a radical simplification of the Bayes–Kelly conformal test martingale (1). We still assume that the Markov chain is symmetric, as in our model situations. If we assume that the weights $w_{k, j}^{n}$ , $k = 0, \dots, n$ , are concentrated at

k \approx k + 1 \approx n / 2,

(1) will simplify to

f_{n} (p) = 2 π_{1 | j} 1_{p \leq 0.5} + 2 π_{0 | j} 1_{p > 0.5} .

(2)

Figure 5 shows the weights (averaged over $j \in {0, 1}$ ) for the last step of the Bayes–Kelly conformal test martingale in the medium scenario ( $10^{3}$ observations). They are indeed concentrated around values of $k$ not so different from $0.5 N = 500$ .

Figure 6: The analogue of Figures 2 and 4 for the small scenario.

As a second step, we make (2) straightforward to compute by setting

j := {\begin{matrix} 1 & if p_{n - 1} \leq 0.5 0 & if not . \end{matrix}

(If $k (n - 1) := z_{1} + \dots + z_{n - 1} \approx (n - 1) / 2$ , then $j = z_{n - 1}$ with high probability.) The performance of the simplified version is shown in Figures 2–4 and 6. It is usually worse than that of the Bayes–Kelly conformal test martingale and the two benchmarks, but is comparable on the log scale apart from the right panel of Figure 6.

Figure 7: Boxplots based on $10^{3}$ runs for the final values of the two benchmarks (upper $UB$ and lower $LB$ ), the Bayes–Kelly conformal test martingale (BK), and its simplified version (sBK) in the medium scenario. Left panel: hard case. Right panel: easy case.

Figure 8: The analogue of Figure 7 for $10^{4}$ runs in the small scenario.

The right panel of Figure 6 and Figures 7 and 8 show that the statistical performance of the simplified Bayes–Kelly martingale particularly suffers in the easy case. The notches in the boxplots in Figures 7 and 8

indicate confidence intervals for the median.

References

[1] Valentina Fedorova, Ilia Nouretdinov, Alex Gammerman, and Vladimir Vovk. Plug-in martingales for testing exchangeability on-line. In John Langford and Joelle Pineau, editors, Proceedings of the Twenty Ninth International Conference on Machine Learning, pages 1639–1646. Omnipress, 2012.
[2] Aaditya Ramdas, Johannes Ruf, Martin Larsson, and Wouter Koolen. How can one test if a binary sequence is exchangeable? Fork-convex hulls, supermartingales, and Snell envelopes. Technical Report arXiv:2102.00630 [math.ST], arXiv.org e-Print archive, July 2021 (version 4).
[3] Glenn Shafer, Alexander Shen, Nikolai Vereshchagin, and Vladimir Vovk.
Test martingales, Bayes factors, and p-values.
Statistical Science, 26:84–101, 2011.
[4] Vladimir Vovk. Conformal testing in a binary model situation. Proceedings of Machine Learning Research, 152:131–150, 2021. COPA 2021.
[5] Vladimir Vovk. Testing randomness online. Statistical Science, 36:595–611, 2021.
[6] Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, New York, 2005.

Conformal testing: binary case with Markov alternatives

Authors

Testing the Number of Regimes in Markov Regime Switching Models

Conformal testing in a binary model situation

Testing for Homogeneity with Kernel Fisher Discriminant Analysis

Optimal tests for elliptical symmetry: specified and unspecified location

Testing for Principal Component Directions under Weak Identifiability

Intermediate efficiency of tests under heavy-tailed alternatives

How can one test if a binary sequence is exchangeable? Fork-convex hulls, supermartingales, and Snell envelopes

1 Introduction

2 Model situations

3 Two benchmarks

4 Bayesian conformal testing

5 Simplified Bayesian conformal testing

References

Conformal testing: binary case with Markov alternatives

Authors

Related Research

Testing the Number of Regimes in Markov Regime Switching Models

Conformal testing in a binary model situation

Testing for Homogeneity with Kernel Fisher Discriminant Analysis

Optimal tests for elliptical symmetry: specified and unspecified location

Testing for Principal Component Directions under Weak Identifiability

Intermediate efficiency of tests under heavy-tailed alternatives

How can one test if a binary sequence is exchangeable? Fork-convex hulls, supermartingales, and Snell envelopes

This week in AI

1 Introduction

2 Model situations

3 Two benchmarks

4 Bayesian conformal testing

5 Simplified Bayesian conformal testing

References