Detecting the direction of a signal on high-dimensional spheres: Non-null and Le Cam optimality results

1 Introduction

In directional statistics, the sample space is the unit sphere $S^{p - 1} = {x \in R^{p} : ∥ x ∥^{2} = x^{'} x = 1}$ in $R^{p}$ . By far the most classical distributions on $S^{p - 1}$ are the Fisher–von Mises–Langevin (FvML) ones; see, e.g., Mardia and Jupp (2000) or Ley and Verdebout (2017)

. We say that the random vector

X

with values in

S^{p - 1}

has an

{F v M L}_{p} (θ, κ)

distribution, with

θ \in S^{p - 1}

and

κ \in (0, \infty)

, if it admits the density (throughout, densities on the unit sphere are with respect to the surface area measure)

(1.1)

where, denoting as $Γ (\cdot)$ the Euler Gamma function and as $I_{ν} (\cdot)$ the order- $ν$ modified Bessel function of the first kind, $ω_{p} := (2 π^{p / 2}) / Γ (\frac{p}{2})$ is the surface area of $S^{p - 1}$ and

c_{p, κ} := 1 / \int_{- 1}^{1} (1 - t^{2})^{(p - 3) / 2} exp (κ t) d t = \frac{(κ / 2)^{(p / 2) - 1}}{\sqrt{π} Γ (\frac{p - 1}{2}) I_{\frac{p}{2} - 1} (κ)} \cdot

Clearly, $θ$ is a location parameter ( $θ$ is the modal location on the sphere), that identifies the spike direction of the hyperspherical signal. In contrast, $κ$ is a scale or concentration parameter: the larger $κ$ , the more concentrated the distribution is about the modal location $θ$ . As $κ$ converges to zero, $c_{p, κ}$ converges to $c_{p} := Γ (\frac{p}{2}) / (\sqrt{π} Γ (\frac{p - 1}{2}))$ and the density in (1.1) converges to the density $x \mapsto 1 / ω_{p}$

of the uniform distribution over

S^{p - 1}

. The other extreme case, obtained for arbitrarily large values of

κ

, provides distributions that converge to a point mass in

θ

. Of course, it is expected that the larger

κ

, the easier it is to conduct inference on

θ

— that is, the more powerful the tests on

θ

and the smaller the corresponding confidence zones.

In this paper, we consider inference on $θ$ and focus on the generic testing problem for which the null hypothesis $H_{0} : θ = θ_{0}$ , for a fixed $θ_{0} \in S^{p - 1}$ , is to be tested against $H_{1} : θ \neq θ_{0}$ on the basis of a random sample $X_{n 1}, \dots, X_{n n}$ from the ${F v M L}_{p} (θ, κ)$ distribution — the triangular array notation anticipates non-standard setups where $p$ (hence, also $θ$ ) and/or $κ$ will depend on $n$ . Inference problems on $θ$ in the low-dimensional case have been considered among others in Watson (1983), Chang and Rivest (2001), Larsen, Blæsild and Sørensen (2002), Ley et al. (2013) and Paindaveine and Verdebout (2017). The related spherical regression problem has been tackled in Rivest (1989) and Downs (2003), while testing for location on axial frames has been considered in Arnold and Jupp (2013).

Letting ${¯ X}_{n} := \frac{1}{n} \sum_{i = 1}^{n} X_{n i}$ , the most classical test for the testing problem above is the Watson (1983) test rejecting the null at asymptotic level $α$ whenever

W_{n} := \frac{n (p - 1) {¯ X}_{n}^{'} (I_{p} - θ_{0} θ_{0}^{'}) {¯ X}_{n}}{1 - \frac{1}{n} \sum_{i = 1}^{n} (X_{n i}^{'} θ_{0})^{2}} > χ_{p - 1, 1 - α}^{2},

(1.2)

where $I_{ℓ}$ stands for the $ℓ$

-dimensional identity matrix and

χ_{ℓ, 1 - α}^{2}

denotes the

α

-upper quantile of the chi-square distribution with

ℓ

degrees of freedom. In the classical setup where

p

and

κ

are fixed, the asymptotic properties of the Watson test are well-known, both under the null and under local alternatives; see, e.g., Watson (1983) or Mardia and Jupp (2000). Optimality properties in the Le Cam sense have been studied in Paindaveine and Verdebout (2015). In the non-standard setup where

κ = κ_{n}

converges to zero, Paindaveine and Verdebout (2017) investigated the asymptotic null and non-null behaviors of the Watson test. Interestingly, irrespective of the rate at which

κ_{n}

converges to zero (that is, irrespective of how fast the inference problem becomes more challenging as a function of

n

), the Watson test keeps meeting the asymptotic nominal level constraint and maintains strong optimality properties; see Paindaveine and Verdebout (2017) for details.

For a fixed dimension $p$ , this essentially settles the investigation of the properties of the Watson test and the study of the corresponding hypothesis testing problem. Nowadays, however, increasingly many applications lead to considering high-dimensional directional data: tests of uniformity on high-dimensional spheres have been studied in Chikuse (1991), Cuesta-Albertos, Cuevas and Fraiman (2009), Cai, Fan and Jiang (2013) and Cutting, Paindaveine and Verdebout (2017aa), while high-dimensional FvML distributions (or mixtures of high-dimensional FvML distributions) have been considered in magnetic resonance, gene-expression, and text mining; see, among others, Dryden (2005), Banerjee et al. (2003), and Banerjee et al. (2005). This motivates considering the high-dimensional spherical location problem, based on a random sample $X_{n 1}, \dots, X_{n n}$ from the ${F v M L}_{p_{n}} (θ_{n}, κ_{n})$ distribution, with ( $p_{n}$ ) diverging to infinity (the dimension of $θ_{n}$ then depends on $n$ , which justifies the notation). In this context, Ley, Paindaveine and Verdebout (2015) proved that the Watson test is robust to high-dimensionality in the sense that, as $p_{n}$ goes to infinity with $n$ , this test still has asymptotic size $α$ under $H_{0}^{(n)} : θ_{n} = θ_{n 0}$ . This does not require any condition on the concentration sequence $(κ_{n})$ nor on the rate at which $p_{n}$ goes to infinity, hence covers arbitrarily easy problems ( $κ_{n}$ large) and arbitrarily challenging ones ( $κ_{n}$ small), as well as moderately high dimensions and ultra-high dimensions. On its own, however, this null robustness result is obviously far from sufficient to motivate using the Watson test in high dimensions, as it might very well be that robustness under the null is obtained at the expense of power (in the extreme case, the Watson test, in high dimensions, might actually asymptotically behave like the trivial $α$

-level test that randomly rejects the null with probability

α

These considerations raise many interesting questions, among which: are there alternatives under which the Watson test is consistent in high dimensions? What are the less severe alternatives (if any) under which the Watson test exhibits non-trivial asymptotic powers? Is the Watson test rate-optimal or, on the contrary, are there tests that show asymptotic powers under less severe alternatives than those detected by the Watson test? Does the Watson test enjoy optimality properties in high dimensions? As we will show, answering these questions will require considering several regimes fixing how the concentration $κ_{n}$ behaves as a function of the dimension $p_{n}$ and sample size $n$ . Our results, that will crucially depend on the regime considered, are extensive in the sense that they answer the questions above in all possible regimes.

We achieve this by combining two approaches that are somewhat orthogonal in spirit. (a) The first approach is based on Le Cam’s theory of asymptotic experiments. While this theory is very general, it does not directly apply in the present context since the high-dimensional spherical location problem involves a parametric space, namely ${(θ_{n}, κ) : θ_{n} \in S^{p_{n} - 1}, κ \in (0, \infty)}$ , that depends on $n$ (through $p_{n}$ ). We solve this by exploiting the invariance properties of the testing problem considered. In the image of the model by the corresponding maximal invariant, indeed, the parametric space does not depend on $n$ anymore, which opens the door to studying the problem through the Le Cam approach. We derive stochastic second-order expansions of the resulting log-likelihood ratios, which is the main technical ingredient to establish the local asymptotic normality (LAN) of the invariant model. The LAN property takes different forms and involves different contiguity rates depending on the regime that is considered. In each regime, we determine the Le Cam optimal test for the problem considered and apply the Le Cam third lemma to obtain the asymptotic powers of this test and of the Watson test. This allows us to determine the regime(s) in which the Watson test is Le Cam optimal, or only rate-optimal, or not even rate-optimal. While this is first done under specified concentration $κ_{n}$ , we also provide LAN results with respect to both location and concentration to be able to discuss optimality under unspecified $κ_{n}$ , too.

(b) In regimes where the Watson test is not rate-optimal, that is, in regimes where it is blind to contiguous alternatives, this first, Le Cam, approach leaves open the question of the existence of alternatives that can be detected by the Watson test. This motivates complementing our investigation by a second approach, where we resort to martingale CLTs to study the asymptotic non-null properties of the Watson test. We identify the alternatives (if any) under which the Watson test will show non-trivial asymptotic powers in high dimensions, which again requires considering various regimes according to the concentration pattern. We do so in a broad, semiparametric, model, namely in the class of rotationally symmetric distributions. Therefore, the corresponding results not only allow us to answer the questions left open by the Le Cam approach but they also extend beyond the parametric FvML framework many of the results obtained there through the Le Cam approach.

The outline of the paper is as follows. In Section 2, we consider the high-dimensional version of the FvML spherical location problem. In Section 2.1, we describe the invariance approach that allows us to later rely on Le Cam’s theory of asymptotic experiments. In Section 2.2, we provide a stochastic second-order expansion of the resulting invariant log-likelihood ratios and prove, in various regimes that we identify, that these invariant models are locally asymptotically normal. This allows us to derive the corresponding Le Cam optimal tests for the specified concentration problem and to study the non-null asymptotic behavior of the Watson test in the light of these results. In Section 2.3, we tackle the unspecified concentration problem through the derivation of LAN results that are with respect to both location and concentration. In Section 3, we conduct a systematic investigation of the non-null asymptotic properties of the Watson test in the broader context of rotationally symmetric distributions. Jointly with the results of Section 2, this allows us to fully characterize, in the FvML case, the non-null asymptotic behavior of the Watson test. In Section 4, we conduct a Monte Carlo study to investigate how well the finite-sample behaviors of the various tests reflect our theoretical asymptotic results. In Section 5, we summarize the results derived in the paper and comment on the few remaining open questions. The appendix contains all proofs.

2 Invariance and Le Cam optimality

As already mentioned in the introduction, the high-dimensional spherical location problem requires considering triangular arrays of observations of the form $X_{n i}$ , $i = 1, \dots, n$ , $n = 1, 2, \dots$ For any sequence $(θ_{n})$ such that $θ_{n}$ belongs to $S^{p_{n} - 1}$ for any $n$ and any sequence $(κ_{n})$ in $(0, \infty)$ , we denote as $P_{θ_{n}, κ_{n}}^{(n)}$ the hypothesis under which $X_{n i}$ , $i = 1, \dots, n$ , form a random sample from the ${F v M L}_{p_{n}} (θ_{n}, κ_{n})$ distribution. The resulting sequence of statistical models is then associated with

P^{(n)} = {P_{θ_{n}, κ}^{(n)} : (θ_{n}, κ) \in Θ_{n} := S^{p_{n} - 1} \times (0, \infty)}

(2.1)

(the index in the parameter $θ_{n}$ in principle is superfluous but is used here to stress the dependence of this parameter on $p_{n}$ , hence on $n$ ). The spherical location problem consists in testing the null hypothesis $H_{0}^{(n)} : θ_{n} = θ_{n 0}$ against the alternative $H_{0}^{(n)} : θ_{n} \neq θ_{n 0}$ , where $(θ_{0 n})$ is a fixed sequence such that $θ_{0 n}$ belongs to $S^{p_{n} - 1}$ for any $n$ . Clearly, $θ_{n}$ is the parameter of interest, whereas $κ_{n}$ plays the role of a nuisance. The main objective of this section is to derive Le Cam optimality results for this problem, referring to local alternatives of the form $P_{θ_{n}, κ_{n}}^{(n)}$ , with $θ_{n} = θ_{n 0} + ν_{n} τ_{n}$ , where the sequence $(ν_{n})$ and the bounded sequence $(τ_{n})$ , respectively in $(0, \infty)$ and in $R^{p_{n}}$ , are such that $θ_{n} \in S^{p_{n} - 1}$ for any $n$ , which imposes that

θ_{n 0}^{'} τ_{n} = - \frac{1}{2} ν_{n} ∥ τ_{n} ∥^{2}

(2.2)

for any $n$ . Since the sequence of “statistical experiments” associated with (2.1) involves parametric spaces $Θ_{n}$ that depend on $n$ , applying Le Cam’s theory will require the following reduction of the problem through invariance arguments.

2.1 Reduction through invariance

Denoting as $S O_{p} (θ)$ the collection of $p \times p$ orthogonal matrices satisfying $O θ = θ$ , the null hypothesis $H_{0}^{(n)}$ is invariant under the group $G_{n},_{\circ}$ collecting the transformations

(X_{n 1}, \dots, X_{n n}) \mapsto g_{n O} (X_{n 1}, \dots, X_{n n}) = (O X_{n 1}, \dots, O X_{n n}),

with $O \in S O_{p_{n}} (θ_{n 0})$ . The transformation $g_{n O}$ induces a transformation of the parametric space $Θ_{n}$ defined through $(θ_{n}, κ) \mapsto (O θ_{n}, κ)$ . The orbits of the resulting induced group are $C_{u, κ} (θ_{n 0}) := {θ_{n} \in S^{p_{n} - 1} : θ_{n}^{'} θ_{n 0} = u} \times {κ}$ , with $u \in [- 1, 1]$ and $κ \in (0, \infty)$ . In such a context, the invariance principle (see, e.g., Lehmann and Romano, 2005, Chapter 6) leads to restricting to tests $ϕ_{n}$ that are invariant with respect to the group $G_{n},_{\circ}$ . Denoting as $T_{n} = T_{n} (X_{n 1}, \dots, X_{n n})$ a maximal invariant statistic for $G_{n},_{\circ}$ , the class of invariant tests coincides with the class of $T_{n}$ -measurable tests. Invariant tests thus are to be defined in the image

P^{(n) T_{n}} = {P_{u, κ}^{(n) T_{n}} : (u, κ) \in Ψ := [- 1, 1] \times (0, \infty)}

(2.3)

of the model $P^{(n)}$ by $T_{n}$ , where $P_{u, κ}^{(n) T_{n}}$ denotes the common distribution of $T_{n}$ under any $P_{θ_{n}, κ}^{(n)}$ with $(θ_{n}, κ) \in C_{u, κ} (θ_{n 0})$ . Unlike the original sequence of statistical experiments in (2.1), the invariant one in (2.3) involves a fixed parametric space $Ψ$ , which makes it in principle possible to rely on Le Cam’s asymptotic theory.

Now, the original local log-likelihood ratios $log (d P_{θ_{n}, κ_{n}}^{(n)} / d P_{θ_{n 0}, κ_{n}}^{(n)})$ associated with the generic local alternatives $θ_{n} = θ_{n 0} + ν_{n} τ_{n}$ above correspond, in view of (2.2), to the invariant local log-likelihood ratios

Λ_{θ_{n} / θ_{n 0}; κ_{n}}^{(n) i n v} := log \frac{d P_{1 - ν_{n}^{2} ∥ τ_{n} ∥^{2} / 2, κ_{n}}^{(n) T_{n}}}{d P_{1, κ_{n}}^{(n) T_{n}}} \cdot

(2.4)

Deriving local asymptotic normality (LAN) results requires investigating the asymptotic behavior of such invariant log-likelihood ratios, which in turn requires evaluating the corresponding likelihoods. While obtaining a closed-form expression for $T_{n}$ and its distribution is a very challenging task, these likelihoods can be obtained from Lemma 2.5.1 in Giri (1996), which, denoting as $m_{n}$ the surface area measure on $S^{p_{n} - 1} \times \dots \times S^{p_{n} - 1}$ ( $n$ times), yields

$\frac{d P_{1 - ν_{n}^{2} ∥ τ_{n} ∥^{2} / 2, κ_{n}}^{(n) T_{n}}}{d m_{n}}$	$=$	$\int_{S O_{p_{n}} (θ_{n 0})} \frac{d P_{θ_{n}, κ_{n}}^{(n)}}{d m_{n}} (O X_{n 1}, \dots, O X_{n n}) d O$	(2.5)
	$=$
	$=$	$\frac{c_{p_{n}, κ_{n}}^{n}}{ω_{p_{n} - 1}^{n}} \int_{S O_{p_{n}} (θ_{n 0})} exp (n κ_{n} {¯ X}_{n} O^{'} θ_{n}) d O,$

where integration is with respect to the Haar measure on $S O_{p_{n}} (θ_{n 0})$ . Note that (2.5) shows that the invariant null probability measure $P_{1, κ_{n}}^{(n) T_{n}}$ coincides with the original null probability measure $P_{θ_{n 0}, κ_{n}}^{(n)}$ . In other words, it is only for non-null probability measures that the invariance reduction above is non-trivial.

2.2 Optimal testing under specified $κ_{n}$

The main ingredient needed to obtain LAN results is Theorem 2.1 below, that provides a stochastic second-order expansion of the invariant log-likelihood ratios in (2.4). To state this theorem, we need to introduce the following notation. We will refer to the decomposition $X_{n i} = U_{n i} θ_{n 0} + V_{n i} S_{n}$ , with

U_{n i} = X_{n i}^{'} θ_{n 0}, V_{n i} = (1 - U_{n i}^{2})^{1 / 2} and S_{n i} = \frac{(I_{p_{n}} - θ_{n 0} θ_{n 0}^{'}) X_{n i}}{∥ (I_{p_{n}} - θ_{n 0} θ_{n 0}^{'}) X_{n i} ∥},

as the tangent-normal decomposition of $X_{n i}$ with respect to $θ_{n 0}$ . Under the hypothesis $P_{θ_{n 0}, κ_{n}}^{(n)}$ , $U_{n i}$

has probability density function

(2.6)

where $I [A]$ denotes the indicator function of $A$ , $S_{n i}$ is uniformly distributed over the “equator” ${x \in S^{p_{n} - 1} : x^{'} θ_{n 0} = 0}$ , and $U_{n i}$ and $S_{n i}$ are mutually independent. Throughout, we will denote as $e_{n ℓ} = E [U_{n i}^{ℓ}]$ and ${~ e}_{n ℓ} = E [(U_{n i} - e_{n 1})^{ℓ}]$ , $ℓ = 1, 2, \dots$

the non-centered and centered moments of

U_{n i}

under

P_{θ_{n 0}, κ_{n}}^{(n)}

, and as

f_{n ℓ} = E [V_{n i}^{ℓ}]

the corresponding non-centered moments of

V_{n i}

. Although this is not stressed in the notation, these moments clearly depend on

p_{n}

and

κ_{n}

; for instance,

e_{n 1} = \frac{I_{\frac{p_{n}}{2}} (κ_{n})}{I_{\frac{p_{n}}{2} - 1} (κ_{n})}, {~ e}_{n 2} = 1 - \frac{p_{n} - 1}{κ_{n}} e_{n 1} - e_{n 1}^{2} and f_{n 2} = \frac{p_{n} - 1}{κ_{n}} e_{n 1}

(2.7)

(this readily follows from (2)-(3) in Schou, 1978 by using the standard properties of exponential families; see also Lemma S.2.1 in Cutting, Paindaveine and Verdebout, 2017b). We can now state the stochastic second-order expansion result of the invariant log-likelihood ratios in (2.4).

Theorem 2.1.

Let $(p_{n})$ be a sequence of integers that diverges to infinity and $(κ_{n})$ be an arbitrary sequence in $(0, \infty)$ . Let $(θ_{n 0})$ , $ν_{n}$ and $τ_{n}$ be sequences such that $θ_{n 0}$ and $θ_{n} = θ_{n 0} + ν_{n} τ_{n}$ belong to $S^{p_{n} - 1}$ for any $n$ , with $(τ_{n})$ bounded and $(ν_{n})$ such that

ν_{n}^{2} = O (\frac{\sqrt{p_{n}}}{n κ_{n} e_{n 1}}) .

(2.8)

Then, letting

Z_{n} := \frac{\sqrt{n} ({¯ X}_{n}^{'} θ_{0} - e_{n 1})}{\sqrt{{~ e}_{n 2}}} and {˜ W}_{n} := \frac{W_{n} - (p_{n} - 1)}{\sqrt{2 (p_{n} - 1)}},

we have that

	$Λ_{θ_{n} / θ_{n 0}; κ_{n}}^{(n) i n v} = - \frac{1}{2} \sqrt{n} κ_{n} ν_{n}^{2} \sqrt{{~ e}_{n 2}} ∥ τ_{n} ∥^{2} Z_{n} + \frac{n κ_{n} ν_{n}^{2} e_{n 1}}{\sqrt{2} p_{n}^{1 / 2}} ∥ τ_{n} ∥^{2} (1 - \frac{1}{4} ν_{n}^{2} ∥ τ_{n} ∥^{2}) {˜ W}_{n}$

			$- \frac{1}{8} n κ_{n} ν_{n}^{4} e_{n 1} ∥ τ_{n} ∥^{4} - \frac{n^{2} κ_{n}^{2} ν_{n}^{4} e_{n 1}^{2}}{4 p_{n}} ∥ τ_{n} ∥^{4} (1 - \frac{1}{4} ν_{n}^{2} ∥ τ_{n} ∥^{2})^{2} + o_{P} (1),$

as $n \to \infty$ under $P_{θ_{n 0}, κ_{n}}^{(n)}$ .

Recalling that the log-likelihood ratio $Λ_{θ_{n} / θ_{n 0}; κ_{n}}^{(n) i n v}$ refers to the local perturbation $θ_{n}^{'} θ_{n 0} = 1 - ν_{n}^{2} ∥ τ_{n} ∥^{2} / 2$ of the null reference value $θ_{n 0}^{'} θ_{n 0} = 1$ , the result in Theorem 2.1 essentially shows that the invariant model considered enjoys a local asymptotic quadraticity (LAQ) structure in the vicinity of the null hypothesis $H_{0}^{(n)} : θ_{n} = θ_{n 0}$ ; see, e.g., Le Cam and Yang (2000), page 120. Actually, quadraticity, which is supposed to be in the increment $- ν_{n}^{2} ∥ τ_{n} ∥^{2} / 2$ , only holds for arbitrarily small values of this increment, hence only in regimes where $ν_{n}$ will converge to zero (in regimes below where, in contrast, $ν_{n}$ will be constant, the non-flat manifold structure of the hypersphere actually prevents a standard quadraticity property). This LAQ result hints that optimal testing for the specified- $κ_{n}$ problem at hand is obtained by rejecting the null for small values of $Z_{n}$ (that is, when ${¯ X}_{n}$ and $θ_{n 0}$ project far from each other onto the axis $\pm θ_{n 0}$ ), for large values of ${˜ W}_{n}$ (that is, when ${¯ X}_{n}$ and $θ_{n 0}$ project far from each other onto the orthogonal complement to $θ_{n 0}$ in $R^{p_{n}}$

), or, more generally, for large values of a hybrid test statistic of the form

Q_{n}^{μ, λ} = μ {˜ W}_{n} + λ (- Z_{n}),

with non-negative weights $μ$ and $λ$ . While any $Q_{n}^{μ, λ}$ provides a reasonable test statistic for the problem at hand, only one set of weights will yield a Le Cam optimal test and, interestingly, this set of weights depends on the way $κ_{n}$ behaves with $p_{n}$ and $n$ . This will be one of the many consequences of the following LAN result.

Theorem 2.2.

Let $(p_{n})$ be a sequence of integers that diverges to infinity, $(κ_{n})$ be a sequence in $(0, \infty)$ , and $(θ_{n 0})$ be a sequence such that $θ_{n 0}$ belongs to $S^{p_{n} - 1}$ for any $n$ . Then, there exist a sequence $(ν_{n})$ in $(0, \infty)$

and a sequence of random variables

(Δ_{n})

that is asymptotically normal with zero mean and variance

Γ

under

P_{θ_{n 0}, κ_{n}}^{(n)}

such that, for any bounded sequence

(τ_{n})

such that

θ_{n} = θ_{n 0} + ν_{n} τ_{n}

belongs to

S^{p_{n} - 1}

for any

n

Λ_{θ_{n} / θ_{n 0}; κ_{n}}^{(n) i n v} = ∥ τ_{n} ∥^{2} Δ_{n} - \frac{1}{2} ∥ τ_{n} ∥^{4} Γ + o_{P} (1)

as $n \to \infty$ under $P_{θ_{n 0}, κ_{n}}^{(n)}$ . If (i) $κ_{n} / p_{n} \to \infty$ , then

ν_{n} = \frac{p_{n}^{1 / 4}}{\sqrt{n κ_{n}}}, Δ_{n} = \frac{{˜ W}_{n}}{\sqrt{2}}, and Γ = \frac{1}{2};

if (ii) $κ_{n} / p_{n} \to ξ > 0$ , then, letting $c_{ξ} := \frac{1}{2} + \sqrt{\frac{1}{4} + ξ^{2}}$ ,

ν_{n} = \frac{\sqrt{c_{ξ}} p_{n}^{3 / 4}}{\sqrt{n} κ_{n}}, Δ_{n} = \frac{{˜ W}_{n}}{\sqrt{2}}, and Γ = \frac{1}{2};

if (iii) $κ_{n} / p_{n} \to 0$ with $\sqrt{n} κ_{n} / p_{n} \to \infty$ , then

ν_{n} = \frac{p_{n}^{3 / 4}}{\sqrt{n} κ_{n}}, Δ_{n} = \frac{{˜ W}_{n}}{\sqrt{2}}, and Γ = \frac{1}{2};

if (iv) $\sqrt{n} κ_{n} / p_{n} \to ξ > 0$ , then

ν_{n} = \frac{p_{n}^{3 / 4}}{\sqrt{n} κ_{n}}, Δ_{n} = \frac{{˜ W}_{n}}{\sqrt{2}} - \frac{Z_{n}}{2 ξ}, and Γ = \frac{1}{2} + \frac{1}{4 ξ^{2}};

if (v) $\sqrt{n} κ_{n} / p_{n} \to 0$ with $\sqrt{n} κ_{n} / \sqrt{p_{n}} \to \infty$ , then

ν_{n} = \frac{p_{n}^{1 / 4}}{n^{1 / 4} \sqrt{κ_{n}}}, Δ_{n} = - \frac{Z_{n}}{2}, and Γ = \frac{1}{4};

if (vi) $\sqrt{n} κ_{n} / \sqrt{p_{n}} \to ξ > 0$ , then

ν_{n} = 1, Δ_{n} = - \frac{ξ Z_{n}}{2}, and Γ = \frac{ξ^{2}}{4};

finally, if (vii) $\sqrt{n} κ_{n} / \sqrt{p_{n}} \to 0$ , then, even with $ν_{n} = 1$ , the invariant log-likelihood ratio $Λ_{θ_{n} / θ_{n 0}; κ_{n}}^{(n) i n v}$ is $o_{P} (1)$ as $n \to \infty$ under $P_{θ_{n 0}, κ_{n}}^{(n)}$ .

In the image model (2.3), the spherical location problem consists in testing $H_{0}^{(n)} : u = 1$ against $H_{0}^{(n)} : u < 1$ . In the localized at $u = u_{0} = 1$ experiments, parametrized by $u = u_{0} - \frac{1}{2} ν_{n}^{2} ∥ τ_{n} ∥^{2}$ as in (2.4), this reduces to testing $H_{0}^{(n)} : ∥ τ_{n} ∥ = 0$ against $H_{1}^{(n)} : ∥ τ_{n} ∥ > 0$ . In any given regime (i)-(vii) from Theorem 2.2, it directly follows from this theorem that a locally asymptotically most powerful test for this problem — hence, locally asymptotically most powerful invariant test for the original spherical location problem — rejects the null at asymptotic level $α$ whenever

Δ_{n} / \sqrt{Γ} > Φ^{- 1} (1 - α),

(2.9)

where $Φ$

denotes the cumulative distribution function of the standard normal (in the rest of the paper, the term “

optimal” will refer to this particular Le Cam optimality concept). A routine application of the Le Cam third lemma then shows that, in each regime, the asymptotic distribution of

Δ_{n}

, under the corresponding contiguous alternatives

P_{θ_{n 0} + ν_{n} τ_{n}, κ_{n}}^{(n)}

with

∥ τ_{n} ∥ \to t

, is normal with mean

Γ t^{2}

and variance

Γ

, so that the resulting asymptotic power of the optimal test in (2.9) is

$lim n \to \infty P_{θ_{n 0} + ν_{n} τ_{n}, κ_{n}}^{(n)} [Δ_{n} / \sqrt{Γ} > Φ^{- 1} (1 - α)]$		(2.10)

	$= 1 - Φ (Φ^{- 1} (1 - α) - \sqrt{Γ} t^{2}) .$

In each regime (i)-(vii), $ν_{n}$ is the contiguity rate, which implies that the least severe alternatives under which a test may have non-trivial asymptotic powers are of the form $P_{θ_{n 0} + ν_{n} τ_{n}, κ_{n}}^{(n)}$ , with a sequence $(∥ τ_{n} ∥)$ that is $O (1)$ but not $o (1)$ . Theorem 2.2 shows that this contiguity rate depends on the regime considered and does so in a monotonic fashion, which is intuitively reasonable: the larger $κ_{n}$ (that is, the easier the inference problem), the faster $ν_{n}$ goes to zero, that is, the less severe the alternatives that can be detected by rate-consistent tests. Because the unit sphere $S^{p_{n} - 1}$ has a fixed diameter, $ν_{n} = 1$ characterizes the most severe alternatives that can be considered. In regime (vi), no tests will therefore be consistent under such most severe alternatives, while, in regime (vii), the distribution is so close to the uniform distribution on $S^{p_{n} - 1}$ that no tests can show non-trivial asymptotic powers under such alternatives, so that even the trivial $α$ -test is optimal.

One of the most striking consequences of Theorem 2.2 is that the optimal test depends on the regime considered. In regimes (v)-(vii), the optimal test in (2.9) rejects the null when $Z_{n} < Φ^{- 1} (α)$ ; of course, this optimality is degenerate in regime (vii), where any invariant test with asymptotic level $α$ would also be optimal. In contrast, the optimal $α$ -level test in regimes (i)-(iii) rejects the null when

{˜ W}_{n} = \frac{W_{n} - (p_{n} - 1)}{\sqrt{2 (p_{n} - 1)}} > Φ^{- 1} (1 - α) .

Since the chi-square distribution with $p - 1$ degrees of freedom converges, after standardization via its mean $p - 1$

and standard deviation

\sqrt{2 (p - 1)}

, to the standard normal distribution as

p

diverges to infinity, this test is asymptotically equivalent to the Watson test in (1.2), based obviously on the dimension

p = p_{n}

at hand. This shows that, in regimes (i)-(iii), the traditional, low-dimensional, Watson test is optimal in high dimensions. In regime (iv), which is at the frontier between these regimes where the optimal test is the Watson test and those where the optimal test is based on

Z_{n}

, the optimal test is quite naturally based on a linear combination of

{˜ W}_{n}

and

Z_{n}

Finally, the Le Cam third lemma allows to derive the asymptotic non-null behavior of the Watson test under the contiguous alternatives considered in any regime (i)-(vii). In regimes (i)-(iv), the limiting powers under contiguous alternatives of the form $P_{θ_{n 0} + ν_{n} τ_{n}, κ_{n}}^{(n)}$ , with $∥ τ_{n} ∥ \to t$ , are given by

1 - Φ (Φ^{- 1} (1 - α) - \frac{t^{2}}{\sqrt{2}}) .

(2.11)

In regimes (i)-(iii), the Watson test is the optimal test and these asymptotic powers are equal to those in (2.10), whereas in regime (iv), the Watson test is only rate-consistent, as the corresponding asymptotic powers of the optimal test are

1 - Φ (Φ^{- 1} (1 - α) - t^{2} \sqrt{\frac{1}{2} + \frac{1}{4 ξ^{2}}}) .

(2.12)

In regimes (v)-(vi), the Le Cam third lemma shows that the limiting powers of the Watson test, still under the corresponding contiguous alternatives, are equal to the nominal level $α$ , so that the Watson test is not even rate-consistent in those regimes. Finally, as already discussed, the Watson test is optimal in regime (vii), but trivially so since the trivial $α$ -test there also is.

2.3 Optimal testing under unspecified $κ_{n}$

The optimal test in regimes (i)-(iii), namely the Watson test, is a genuine test in the sense that it can be applied on the basis of the observations only. In contrast, the optimal tests in regimes (iv)-(vi) are “oracle” tests since they require knowing the values of $e_{n 1}$ and ${~ e}_{n 2}$ , or equivalently (see (2.7)), the value of the concentration $κ_{n}$ . This concentration, however, can hardly be assumed to be specified in practice, so that it is natural to wonder what is the optimal test, in regimes (iv)-(vi), when $κ_{n}$ is treated as a nuisance parameter.

We first focus on regime (iv). There, the concentration $κ_{n}$ is asymptotically of the form $κ_{n} = p_{n} ξ / \sqrt{n}$ for some $ξ > 0$ . Within regime (iv), $ξ$ , obviously, is a perfectly valid alternative concentration parameter. Inspired by the classical treatment of asymptotically optimal inference in the presence of nuisance parameters (see, e.g., Bickel et al., 1998), this suggests studying the asymptotic behavior of invariant log-likelihood ratios of the form

Λ_{θ_{n}, κ_{n, s} / θ_{n 0}, κ_{n}}^{(n) i n v} := log \frac{d P_{1 - ν_{n}^{2} ∥ τ_{n} ∥^{2} / 2, κ_{n, s}}^{(n) T_{n}}}{d P_{1, κ_{n}}^{(n) T_{n}}},

where $κ_{n, s} := p_{n} (ξ + ϑ_{n} s) / \sqrt{n}$ is a suitable sequence of perturbed concentrations. We have the following result.

Theorem 2.3.

Let $(p_{n})$ be a sequence of integers that diverges to infinity with $p_{n} = o (n^{2})$ as $n \to \infty$ . Let $κ_{n} := p_{n} ξ / \sqrt{n}$ , with $ξ > 0$ , and $κ_{n, s} := p_{n} (ξ + s / \sqrt{p_{n}}) / \sqrt{n}$ , where $s$ is such that $ξ + s / \sqrt{p_{n}} > 0$ for any $n$ . Let $(θ_{n 0})$ and $τ_{n}$ be sequences such that $θ_{n 0}$ and $θ_{n} = θ_{n 0} + ν_{n} τ_{n}$ , with the $ν_{n}$ below, belong to $S^{p_{n} - 1}$ for any $n$ . Then, putting $t_{n} := (∥ τ_{n} ∥^{2}, s)^{'}$ ,

we have

Λ_{θ_{n}, κ_{n, s} / θ_{n 0}, κ_{n}}^{(n) i n v} = t_{n}^{'} Δ_{n} - \frac{1}{2} t_{n}^{'} Γ t_{n} + o_{P} (1)

(2.13)

as $n \to \infty$ under $P_{θ_{n 0}, κ_{n}}^{(n)}$ , where $Δ_{n}$ , under the same sequence of hypotheses, is asymptotically normal with mean zero and covariance matrix $Γ$ .

Theorem 2.3 shows that, in regime (iv), the sequence of high-dimensional FvML experiments is jointly LAN in the location and concentration parameters. The corresponding Fisher information matrix $Γ = (Γ_{i j})$ is not diagonal, which entails that the unspecification of the concentration parameter has asymptotically a positive cost when performing inference on the location parameter. In the present joint LAN framework, Le Cam optimal inference for location under unspecified concentration is to be based (see again Bickel et al., 1998) on the residual of the regression (in the limiting Gaussian shift experiment) of the location part $Δ_{n 1}$ of the central sequence $Δ = (Δ_{n 1}, Δ_{n 2})^{'}$ with respect to the concentration part $Δ_{n 2}$ , that is, is to be based on the efficient central sequence

Δ_{n 1}^{*} := Δ_{n 1} - \frac{Γ_{12}}{Γ_{22}} Δ_{n 2} .

Under the null, $Δ_{n 1}^{*}$ is asymptotically normal with mean zero and variance $Γ_{11}^{*} = Γ_{11} - Γ_{12}^{2} / Γ_{22}$ , and the Le Cam optimal location test under unspecified $κ_{n}$ rejects the null at asymptotical level $α$ when

Δ_{n 1}^{*} / \sqrt{Γ_{11}^{*}} = {˜ W}_{n} > Φ^{- 1} (1 - α) .

As a corollary, provided that $p_{n} = o (n^{2})$ , the unspecified- $κ_{n}$ optimal test in regime (iv) is the Watson test. Consequently, the difference between the local asymptotic powers in (2.11) and (2.12), associated with the Watson test and the specified- $κ_{n}$ optimal test in regime (iv), respectively, can be interpreted as the asymptotic cost of the unspecification of the concentration when performing inference on location in the regime considered.

We now turn to regime (v), which is associated with $κ_{n} = p_{n} r_{n} ξ / \sqrt{n}$ , where $ξ > 0$ and $r_{n}$ is a positive sequence satisfying $r_{n} = o (1)$ and $r_{n} \sqrt{p_{n}} \to \infty$ . Taking $ν_{n} = p_{n}^{1 / 4} / (n^{1 / 4} \sqrt{κ_{n}})$ (as in Theorem 2.2) and considering perturbed concentrations of the form $κ_{n, s} = p_{n} r_{n} (ξ + s / (\sqrt{p_{n}} r_{n})) / \sqrt{n}$ , one can show, by working along the same lines as in the proof of Theorem 2.3, that the sequence of experiments is also jointly LAN in location and concentration, provided that $p_{n} = o (n^{2} r_{n}^{- 6})$ . The corresponding central sequence and Fisher information matrix are

(2.14)

The co-linearity between the location part $Δ_{n 1}$ and concentration part $Δ_{n 2}$ of the central sequence implies that the efficient central sequence $Δ_{n 1}^{*}$ is zero in regime (v). As a result, for the unspecified concentration problem, no test can detect deviations from the null hypothesis at the $ν_{n}$ -rate in regime (v), which is in line with the corresponding trivial asymptotic powers of the Watson in Section 2.2. In the next section, we will investigate whether or not the Watson test can detect more severe alternatives in this regime.

Finally, consider regime (vi), where the concentration $κ_{n}$ is asymptotically of the form $κ_{n} = \sqrt{p_{n}} ξ / \sqrt{n}$ . Then, taking $ν = 1$ (still as in Theorem 2.2) and perturbed concentrations of the form $κ_{n, s} := \sqrt{p_{n}} (ξ + s) / \sqrt{n}$ , we obtain that, without any condition on $p_{n}$ , the sequence of experiments is jointly LAN in location and concentration with the same central sequence and Fisher information matrix as in (2.14). The resulting efficient central sequence is therefore zero again. Since, in regime (vi), $ν_{n} = 1$ provides the most severe location alternatives than can be considered, we conclude that, for the unspecified concentration problem, no test in regime (vi) can do asymptotically better than the trivial $α$ -level test that randomly rejects the null with probability $α$ . Under unspecified $κ_{n}$ , thus, the Watson test is optimal in regime (vi), too, even if it is in a degenerate way.

Wrapping up, we proved that the Watson test is optimal in regimes (i)-(iii) for the specified concentration problem and that it is optimal in all regimes but possibly regime (v) in the more important unspecified concentration one (in regime (iv), the latter optimality actually requires that $p_{n} = o (n^{2})$ ). The asymptotic cost due to the unspecification of the concentration is nil in regimes (i)-(iii) (and (vii)), affects limiting powers but not consistency rates in regime (iv), and affects consistency rates in regimes (v)-(vi).

3 Non-null investigation via martingale CLTs

The results of the previous section bring much information on the non-null behavior of the Watson test in high dimensions, but two important questions remain open. First, while the Watson test was shown to be Le Cam optimal for the unspecified concentration problem in regimes (i)-(iv) and (vi)-(vii), little is known on its performances in regime (v). More precisely, it is only known that, like any location test addressing the unspecified- $κ_{n}$ problem, the Watson test is blind to the contiguous alternatives in Theorem 2.2(v), so that it is unclear whether or not this test can detect more severe alternatives. Second, the results of the previous section are limited to the FvML case, while the Watson test is valid (in the sense that it meets the asymptotic nominal level constraint) under much broader distributional assumptions. One may therefore wonder about the non-null behavior of the Watson test also under high-dimensional non-FvML distributions.

In this section, we address these two questions by investigating, through martingale CLTs, the non-null behavior of the Watson test under general rotationally symmetric alternatives. Recall that the distribution of a random vector $X$ with values in $S^{p - 1}$ is rotationally symmetric about $θ (\in S^{p - 1})$ if $O X$ and $X$ share the same distribution for any $O \in S O_{θ} (p)$ , and that it is rotationally symmetric if it is rotationally symmetric about some $θ$ in $S^{p - 1}$ . Clearly, if $X$ has an ${F v M L}_{p} (θ, κ)$ distribution, then it is rotationally symmetric about $θ$ , so that the distributional context considered in this section will encompass the one in Section 2. Parallel to what was done there, we will refer to the decomposition $X = U θ + V S,$ with $U = X^{'} θ$ , $V = \sqrt{1 - U^{2}}$ and $S = (I_{p} - θ θ^{'}) X / ∥ (I_{p} - θ θ^{'}) X ∥$ , as the tangent-normal decomposition of $X$ with respect to $θ$ . If $X$ is rotationally symmetric about $θ$ , then $S$ is uniformly distributed over ${x \in S^{p - 1} : x^{'} θ = 0}$ and is independent of $U$ . The distribution of $X$ is then fully determined by $θ$ and by the cumulative distribution function $F$ of $U$ , which justifies denoting the corresponding distribution as ${R o t}_{p} (θ, F)$ . In the sequel, we tacitly restrict to classes of rotationally symmetric distributions making $θ$ identifiable, which typically only excludes distributions satisfying ${R o t}_{p} (- θ, F) = {R o t}_{p} (θ, F)$ .

We consider then a triangular array of observations of the form $X_{n i}$ , $i = 1, \dots, n$ , $n = 1, 2, \dots$ , where $X_{n 1}, \dots, X_{n n}$ form a random sample from the rotationally symmetric distribution ${R o t}_{p_{n}} (θ_{n}, F_{n})$ . The corresponding hypothesis, that will be denoted as $P_{θ_{n}, F_{n}}^{(n)}$ involves a sequence of integers $(p_{n})$ diverging to infinity, a sequence $(θ_{n})$ such that $θ_{n} \in S^{p_{n} - 1}$ for any $n$ , and a sequence $(F_{n})$ of cumulative distribution functions over $[- 1, 1]$ . In this framework, the spherical location problem consists in testing $H_{0}^{(n)} : θ_{n} = θ_{n 0}$ against $H_{1}^{(n)} : θ_{n} \neq θ_{n 0}$ , where $(θ_{n 0})$ is a fixed null parameter sequence. Parallel to the notation that was used in the FvML case, we will write $e_{n ℓ}$ and ${~ e}_{n ℓ}$ , $ℓ = 1, 2, \dots$ for the non-centered and centered moments of $F_{n}$ , respectively. These are the moments, under $P_{θ_{n}, F_{n}}^{(n)}$ , of the quantity $U_{n 1} = X_{n 1}^{'} θ_{n}$ in the tangent-normal decomposition of $X_{n 1}$ with respect to $θ_{n}$ . The corresponding non-centered moments of $V_{n 1} = \sqrt{1 - U_{n 1}^{2}}$ will still be denoted as $f_{n ℓ}$ .

Using the notation $V_{n i}$ and $S_{n i}$ from the tangent-normal decomposition of $X_{n i}$ with respect to the null location $θ_{n 0}$ , the Watson test statistic rewrites

{˜ W}_{n} = \frac{W_{n} - (p_{n} - 1)}{\sqrt{2 (p_{n} - 1)}} = \frac{\sqrt{2 (p_{n} - 1)}}{\sum_{i = 1}^{n} V_{n i}^{2}} \sum 1 \leq i < j \leq n V_{n i} V_{n j} S_{n i}^{'} S_{n j},

where $W_{n}$ denotes the Watson test statistic in (1.2) based on the null location $θ_{n 0}$ . Under the null and under appropriate local alternatives, it is expected that ${˜ W}_{n}$ is asymptotically equivalent in probability to

W_{n}^{*} := \frac{\sqrt{2 (p_{n} - 1)}}{n f_{n 2}} \sum 1 \leq i < j \leq n V_{n i} V_{n j} S_{n i}^{'} S_{n j},

so that an important step in the investigation of the non-null properties of ${˜ W}_{n}$ is the study of the non-null behavior of $W_{n}^{*}$

. A classical martingale central limit theorem (see, e.g., Theorem 35.12 in

Billingsley, 1995) provides the following result.

Theorem 3.1.

Let $(p_{n})$ be a sequence of integers that diverges to infinity and $(θ_{n 0})$ be a sequence such that $θ_{n 0}$ belongs to $S^{p_{n} - 1}$ for any $n$ . Let $(F_{n})$ be a sequence of cumulative distribution functions on $[- 1, 1]$ such that (a) $f_{n 2} > 0$ for any $n$ , (b) $f_{n 4} / f_{n 2}^{2} = o (n)$ and (c) $\sqrt{p_{n}} e_{n 2} = o (1)$ . Then, we have the following, where in each case $(τ_{n})$ refers to an arbitrary sequence such that $θ_{n} = θ$

Detecting the direction of a signal on high-dimensional spheres: Non-null and Le Cam optimality results

Authors

On the power of axial tests of uniformity on spheres

Inference for spherical location under high concentration

Sign tests for weak principal directions

On the power of Sobolev tests for isotropy under local rotationally symmetric alternatives

Sparse Uniformity Testing

Finite Sample Smeariness on Spheres

Joint limiting laws for high-dimensional independence tests

1 Introduction

2 Invariance and Le Cam optimality

2.1 Reduction through invariance

2.2 Optimal testing under specified $κ_{n}$

Theorem 2.1.

Theorem 2.2.

2.3 Optimal testing under unspecified $κ_{n}$

Theorem 2.3.

3 Non-null investigation via martingale CLTs

Theorem 3.1.

Detecting the direction of a signal on high-dimensional spheres: Non-null and Le Cam optimality results

Authors

Related Research

On the power of axial tests of uniformity on spheres

Inference for spherical location under high concentration

Sign tests for weak principal directions

On the power of Sobolev tests for isotropy under local rotationally symmetric alternatives

Sparse Uniformity Testing

Finite Sample Smeariness on Spheres

Joint limiting laws for high-dimensional independence tests

This week in AI

1 Introduction

2 Invariance and Le Cam optimality

2.1 Reduction through invariance

2.2 Optimal testing under specified κn

Theorem 2.1.

Theorem 2.2.

2.3 Optimal testing under unspecified κn

Theorem 2.3.

3 Non-null investigation via martingale CLTs

Theorem 3.1.

2.2 Optimal testing under specified $κ_{n}$

2.3 Optimal testing under unspecified $κ_{n}$