Uniform Inference in High-Dimensional Gaussian Graphical Models

1 Introduction

We provide methodology and theory for uniform inference on high-dimensional graphical models with the number of target parameters being possible much larger than sample size. We demonstrate uniform asymptotic normality of the proposed estimator over $d$ -dimensional rectangles and construct simultaneous confidence bands on all of the $d$ target parameters. The proposed method can be applied to test simultaneously the presence of a large set of edges in the graphical model

X = (X_{1}, \dots, X_{p})^{T} \sim N (μ_{X}, Σ_{X}) .

Assuming that the covariance matrix $Σ_{X}$ is nonsingular, the conditional independence structure of the distribution can be conveniently represented by a graph $G = (V, E)$ , where $V = {1, \dots, p}$ is the set of nodes and $E$ the set of edges in $V \times V$

. Every pair of variables not contained in the edge set is conditionally independent given all remaining variables. If the vector

X

is normally distributed, every edge corresponds to a non-zero entry in the inverse covariance matrix (Lauritzen (1996))

[8].

In the last decade, significant progress has been made on estimation of a large precision matrix in order to analyze the dependence structure of a high-dimensional normal distributed random variable. There are mainly two common approaches to estimate the entries of a precision matrix. The first approach is a penalized likelihood estimation approach with a lasso-type penalty on entries of the precision matrix, typically referred to as the graphical lasso. This approach has been studied in several papers, see e.g Lam and Fan (2009)

[7], Rothman et al. (2008) [12], Ravikumar et al. (2011) [10] and Yuan and Lin (2007) [16]. The second approach, first introduced by Meinshausen and Bühlmann (2006) [9], is neighborhood based. It estimates the conditional independence restrictions separately for each node in the graph and is hence equivalent to variable selection for Gaussian linear models. The idea of estimating the precision matrix column by column by running a regression for each variable against the rest of variables was further studied in Yuan (2010) [15], Cai, Liu and Zhou (2011) [4] and Sun and Zhang (2013) [13].

In this paper, we do not aim to estimate the whole precision matrix but we focus on quantifying the uncertainty of recovering its support by providing a significance test for a set of potential edges. In recent years, statistical inference for the precision matrix in high-dimensional settings has been studied, e.g in Janková and van de Geer (2016) [6] and Ren et al. (2015) [11]. Both approaches lead to an estimate that is elementwise asymptotically normal and enables testing for low-dimensional parameters of the precision matrix using standard procedures such as Bonferroni-Holm correction.
In contrast to these existing results, our method explicitly allows for testing a joint hypothesis without correction for multiple testing and conducting inference for a growing number of parameters using high dimensional central limit results. In order to provide theoretical results, fitting the problem of support discovery in Gaussian graphical models into a general Z-estimation setting with a high-dimensional nuisance function is key. Inference on a (multivariate) target parameter in general Z-estimation problems in high dimensions is covered in Belloni et al. (2014) [2], Belloni et al. (2015) [1] and Chernozhukov et al. (2017) [5].
Additionally, our results rely on approximate sparsity instead of row sparsity which restricts the number of non-zero entries of each row of the precision matrix to be at most

s ≪ N

that is in many applications a questionable assumption. In this context, we establish a theorem regarding uniform estimation rates of the lasso/ post lasso estimator in high-dimensional regression models under approximate sparsity conditions.

2 Setting

Let

X = (X_{1}, \dots, X_{p})^{T} \sim N (μ_{X}, Σ_{X})

be a $p$ -dimensional random variable. For all $(j, k) \in E$ with $j \neq k$ , assume that

X_{j} = p \sum \begin{matrix} l = 1 l \neq j \end{matrix} β_{l}^{(j)} X_{l} + ε^{(j)} = β^{(j)} X_{- j} + ε^{(j)}

and

X_{k} = γ^{(j, k)} X_{- {j, k}} + ν^{(j, k)},

where $E [ε^{(j)} | X_{- j}] = 0$ and $E [X_{- {j, k}} ν^{(j, k)}] = 0$ . Define the column vector

Γ^{(j)} = {(- β_{1}^{(j)}, \dots, - β_{j - 1}^{(j)}, 1, - β_{j + 1}^{(j)}, \dots, - β_{p}^{(j)})}^{T} .

One may show

Φ_{0} = (Φ_{0}^{1}, \dots, Φ_{0}^{p}) = (Γ^{(1)} / V a r (ε^{(1)}), \dots, Γ^{(p)} / V a r (ε^{(p)})),

where $Φ_{0}^{j}$ is the $j$ -th column of the precision matrix $Φ_{0} = Σ_{X}^{- 1}$ [6]. Hence

β_{k}^{(j)} = 0 \Leftrightarrow β_{k}^{(j)} = 0 \Leftrightarrow X_{j} ⊥ X_{k} | X_{- {j, k}}

(2.1)

for all $j \neq k$ . Assume that we are interested in the following set of potential edges

M := {m_{1}, \dots, m_{d_{n}}}

where the number of edges $d_{n}$ may increase with sample size $n$ . In the following the dependence on $n$ is omitted to simplify the notation. In order to test whether all variables $X_{j}$ and $X_{k}$ are conditionally independent with $m_{r} = (j_{r}, k_{r})$ for a $r \in {1, \dots, d}$ , we have to estimate our target parameter

θ_{0} = (θ_{m_{1}}, \dots, θ_{m_{d}})^{T} := (β_{k_{1}}^{(j_{1})}, \dots, β_{k_{d}}^{(j_{d})})^{T} .

The setting above fits in the general Z-estimation problem of the form

E [ψ_{m_{r}} (X, θ_{m_{r}}, η_{m_{r}})] = 0

with nuisance parameters

η_{m_{r}} = (β_{- k}^{(j)}, γ^{(j, k)})

where $β_{- k}^{(j)} \equiv β^{(m_{r})}$ and $γ^{(j, k)} \equiv γ^{(m_{r})}$ . The score functions are defined by

ψ_{m_{r}} (X, θ, η) :

= (X_{j} - θ X_{k} - η^{(1)} X_{- m_{r}}) (X_{k} - η^{(2)} X_{- m_{r}})

for $m_{r} = (j_{r}, k_{r}) \equiv (j, k)$ , $η = (η^{(1)}, η^{(2)})$ and $r = 1, \dots, d$ . Without loss of generality we assume $j > k$ for all tuples $m_{r} \in M$ .

Comment 2.1.

The score function $ψ$ is linear, meaning

ψ_{m_{r}} (X, θ, η) = ψ_{m_{r}}^{a} (X, η^{(2)}) θ + ψ_{m_{r}}^{b} (X, η)

with

ψ_{m_{r}}^{a} (X, η^{(2)}) = - X_{k} (X_{k} - η^{(2)} X_{- m_{r}})

and

ψ_{m_{r}}^{b} (X, η) = (X_{j} - η^{(1)} X_{- m_{r}}) (X_{k} - η^{(2)} X_{- m_{r}})

for $m_{r} = (j, k)$ and $r = 1, \dots, d$ .

It is well known that in partially linear regression models

θ_{0}

satisfies the moment condition

E [ψ_{m_{r}} (X, θ_{m_{r}}, η_{m_{r}})] = 0

(2.2)

and also the Neyman orthogonality condition

\partial_{t} {E [ψ_{m_{r}} (X, θ_{m_{r}}, η_{m_{r}} + t ~ η)]} {∣ ∣}_{t = 0}

for all $~ η$ in an appropriate set where $\partial_{t}$ denotes the derivate with respect to $t$ . These properties are crucial for valid inference in high dimensional settings. We will show these properties explicitly in the proof of Theorem 1.

3 Estimation

Let $X^{(i)}$ , $i = 1, \dots, n$ be i.i.d. random vectors.
At first we estimate the nuisance parameter $η_{m_{r}} = (η_{m_{r}}^{(1)}, η_{m_{r}}^{(2)})$

by running a lasso/ post-lasso regression of

X_{j}

X_{- j}

to compute

({~ θ}_{m_{r}}, {^η}_{m_{r}}^{(1)})

and a lasso/ post-lasso regression of

X_{k}

X_{- m_{r}}

to compute

{^η}_{m_{r}}^{(2)}

for each

(j, k) = m_{r} \in M

. The estimator

{^θ}_{0}

of the target parameter

θ_{0} = (θ_{m_{1}}, \dots, θ_{m_{d}})^{T}

is defined as the solution of

supr=1,…,d{∣∣En[ψmr(X,^θmr,^ηmr)]∣∣−infθ∈Θmr∣∣En[ψmr(X,θ,^ηmr)]∣∣}≤ϵn,

(3.1)

where $ϵ_{n} = o (δ_{n} n^{- 1 / 2})$ is the numerical tolerance and $(δ_{n})_{n \geq 1}$ a sequence of positive constants converging to zero.

Assumptions A1-A4.
Let $a_{n} := max (d, p, n, e)$ . The following assumptions hold uniformly in $n \geq n_{0}, P \in P_{n}$ :

[label=A0,ref=A0]
For all $m_{r} = (j, k) \in M$ with $j \neq k$ we have the following approximate sparse representations
- It holds
  
  $X_{j}$ $= β^{(j)} X_{- j} + ε^{(j)}$
  
  $= θ_{m_{r}} X_{k} + β^{(m_{r})} X_{- m_{r}} + ε^{(m_{r})}$
  
  $= θ_{m_{r}} X_{k} + (β^{(1, m_{r})} + β^{(2, m_{r})}) X_{- m_{r}} + ε^{(m_{r})}$
  
  with $∥ β^{(1, m_{r})} ∥_{0} \leq s$ and $E [{(β^{(2, m_{r})} X_{- m_{r}})}_{- m_{r}}^{2}] = O (\frac{s log (a_{n})}{n})$ .
- It holds
  
  $X_{k}$ $= γ^{(j, k)} X_{- {j, k}} + ν^{(j, k)}$
  
  $= γ^{(m_{r})} X_{- m_{r}} + ν^{(m_{r})}$
  
  $= (γ^{(1, m_{r})} + γ^{(1, m_{r})}) X_{- m_{r}} + ν^{(m_{r})}$
  
  with $∥ γ^{(1, m_{r})} ∥_{0} \leq s$ and $E [{(γ^{(2, m_{r})} X_{- m_{r}})}_{- m_{r}}^{2}] = O (\frac{s log (a_{n})}{n})$ .
There exists a positive numbers $~ q > 0$ and $c_{q} > 0$ such that the following growth conditions are fulfilled:

$n^{\frac{1}{~ q}} \frac{s {log}^{4} (a_{n})}{n} ({log}^{2} (log (a_{n})) \lor s) = o (1), log (d) = o (n^{1 / 9} \land n^{\frac{2 c_{q}}{(1 + c_{q}) ~ q}}) .$
For all $(j, k) = m_{r} \in M$ it holds

$∥ β^{(m_{r})} ∥_{2} + ∥ γ^{(m_{r})} ∥_{2} \leq C$

and

$sup r = 1, \dots, d sup θ \in Θ_{m_{r}} | θ | \leq C .$

Additionally $Θ_{m_{r}}$ contains a ball of radius $n^{- 1 / 2} {log}^{1 / 2} (d) log (n)$ centered at $θ_{m_{r}}$ .
It holds

$inf ∥ ξ ∥_{2} = 1 E [(ξ X)^{2}] \geq c and sup ∥ ξ ∥_{2} = 1 E [(ξ X)^{2}] \leq C .$

The condition 1 is a standard approximate sparsity condition. Condition 3 restricts the parameter spaces and ensures that the coefficients are well behaved. The last condition 4 restricts the correlation between the components of $X$

and bounds the variances of each

X_{j}

from below and above. Assumptions 1-4 combined with the normal distribution of

X

imply the conditions 1-3 from theorem 2 which enables us to estimate the nuisance parameter sufficiently fast.

Comment 3.1.

If we have exact sparsity for each $β^{(k)}$ with $(j, k) \in M_{r}$ the sparsity of $γ^{(m_{r})}$ follows directly.
Observe that for $k \in {1, \dots, p} ∖ {j}$ and $l \in {1, \dots, p} ∖ {j, k}$ we have

β_{l}^{(k)} = 0 \Leftrightarrow X_{k} ⊥ X_{l} | X_{- {k, l}} \Leftrightarrow E [X_{k} X_{l} | X_{- {k, l}}] = 0

which implies

E [X_{k} X_{l} | X_{- {j, k, l}}] = E [E [X_{k} X_{l} | X_{- {k, l}}] | X_{- {j, k, l}}] = 0

and thereby

γ_{l}^{(j, k)} = 0 \Leftrightarrow X_{k} ⊥ X_{l} | X_{- {j, k, l}} \Leftrightarrow E [X_{k} X_{l} | X_{- {j, k, l}}] = 0.

4 Main results

We will prove that the assumptions of Corollary $2.2$ from Belloni et al. (2015) [1]

hold and hence we are able to use their results to construct confidence intervals even for a growing number of hypothesis

d = d_{n}

. Define

	$J_{m_{r}}$	$:= \partial_{θ} E [ψ_{m_{r}} (X), θ, η_{m_{r}}] {∣ ∣}_{θ = θ_{m_{r}}} = - E [X_{k} (X_{k} - η_{m_{r}}^{(2)} X_{- m_{r}})]$
	$σ_{m_{r}}^{2}$	$:= E [J_{m_{r}}^{- 2} ψ_{m_{r}}^{2} (X, θ_{m_{r}}, η_{m_{r}})]$

and the corresponding estimators

	${^J}_{m_{r}}$	$= - E_{n} [X_{k} (X_{k} - {^η}_{m_{r}}^{(2)} X_{- m_{r}})]$
	${^σ}_{m_{r}}^{2}$	$= E_{n} [{^J}_{m_{r}}^{- 2} ψ_{m_{r}}^{2} (X, {^θ}_{m_{r}}, {^η}_{m_{r}})]$

for $r = 1, \dots, d$ . To construct confidence intervals we will employ the Gaussian multiplier bootstrap. Define

{^ψ}_{m_{r}} (X) := - {^σ}_{m_{r}}^{- 1} {^J}_{m_{r}}^{- 1} ψ_{m_{r}} (X, {^θ}_{m_{r}}, {^η}_{m_{r}})

and the process

where $(ξ_{i})_{i = 1}^{n}$ are independent standard normal random variables which are independent from $(X^{(i)})_{i = 1}^{n}$ . We define $c_{α}$ as the $(1 - α)$

-conditional quantile of

{sup}_{m_{r} \in M} | {^N}_{m_{r}} |

given the observations

(X^{(i)})_{i = 1}^{n}

. The following theorem is the main result of our paper and establishes simultaneous confidence bands for the target parameter

θ_{0}

Theorem 1.

Under the assumptions 1-4

with probability

1 - o (1)

uniformly in

P \in P_{n}

the estimator

^θ

in (3.1) obeys

P ({^θ}_{m_{r}} - \frac{c_{α} {^σ}_{m_{r}}}{\sqrt{n}} \leq θ_{m_{r}} \leq {^θ}_{m_{r}} + \frac{c_{α} {^σ}_{m_{r}}}{\sqrt{n}}, r = 1, \dots, d) \to 1 - α .

(4.1)

Using theorem 1

we are able to construct standard confidence regions which are uniformly valid over a large set of variables and we can check null hypothesis of the form:

H_{0} : M \cap E = \emptyset .

Comment 4.1.

Theorem 1 is basically an application of the gaussian approximation and multiplier bootstrap for maxima of sums of high-dimensional random vectors [chernozhukov2013gaussian]

. The central limit theorem and bootstrap in high dimension introduced by Chernozhukov, Chetverikov, Kato et al. (2017)

[chernozhukov2017central] extend these results to more general sets, more precisely sparsely convex sets. Hence our main theorem can be easily generalized to various confidence regions that contain the true target parameter with probability

1 - α

. Theorem 1 provides critical regions of the form

sup r = 1, \dots, d ∣ ∣ ∣ ∣ \sqrt{n} \frac{{^θ}_{m_{r}}}{{^σ}_{m_{r}}} ∣ ∣ ∣ ∣ > c_{1 - α} .

(4.2)

Alternatively, we can reject the null hypothesis if

sup r = 1, \dots, d ∣ ∣ ∣ ∣ \sqrt{n} \frac{{^θ}_{m_{r}}}{{^σ}_{m_{r}}} ∣ ∣ ∣ ∣ < c_{\frac{α}{2}} or sup r = 1, \dots, d ∣ ∣ ∣ ∣ \sqrt{n} \frac{{^θ}_{m_{r}}}{{^σ}_{m_{r}}} ∣ ∣ ∣ ∣ > c_{1 - \frac{α}{2}} .

(4.3)

Both of these regions are based on the central limit theorem for hyperrectangles in high dimensions. The confidence region (4.3) is motivated by the fact that the standard normal distribution $N (0, I_{d})$ in high dimensions is concentrated in a thin spherical shell around the sphere of radius $\sqrt{d}$ as described by Roman Vershynin (2017) [vershynin2017high] and therefore might have smaller volume. More generally, define

{^θ}_{m_{r}}^{*} (S, e x p) = S \sum s = 1 {(\sqrt{n} \frac{{^θ}_{m_{r - s}}}{{^σ}_{m_{r - s}}})}^{e x p}

for a fix $S$ , $e x p \in {1, 2}$ and

r - s := {\begin{matrix} r - s & if r - s > 0 d + (r - s) & otherwise \end{matrix} .

A test that reject the null hypothesis if

sup r = 1, \dots, d ∣ ∣ {^θ}_{m_{r}}^{*} (S, e x p) ∣ ∣ > c_{1 - α}^{*}

(4.4)

has level $α$ by [chernozhukov2017central], since the constructed confidence regions correspond to S-sparsely convex sets. Here, $c_{1 - α}^{*}$ is the $(1 - α)$ -conditional quantile of ${sup}_{m_{r} \in M} | {^N}_{m_{r}}^{*} |$ given the observations $(X^{(i)})_{i = 1}^{n}$ with

{^N}_{m_{r}}^{*} = S \sum s = 1 {({^N}_{m_{r - s}})}_{m_{r - s}}^{e x p}

where

r - s := {\begin{matrix} r - s & if r - s > 0 d + (r - s) & otherwise. \end{matrix}

5 The function $G G M t e s t$

The function $G G M t e s t$ that will be added to the $R$ -package $h d m$ estimates the target coefficients

(θ_{m_{1}}, \dots, θ_{m_{d}})^{T} = (β_{k_{1}}^{(j_{1})}, \dots, β_{k_{d}}^{(j_{d})})^{T}

corresponding the considered set of potential edges

M := {m_{1}, \dots, m_{d_{n}}}

by the proposed method described in section 3. It can be used to perform hypothesis tests with asymptotic level $α$ based on the different confidence regions described in comment 4.1. The nuisance function can be estimated by lasso, post-lasso or square root lasso. The verification of uniform convergence rates of the square root lasso estimator for functional parameters in high-dimensional settings is an interesting problem that we plan to address in future work.

5.1 cross-fitting

In general Z- estimation problems where a so called debiased or double machine learning (DML) method is used to construct confidence intervals, it is common to use cross-fitting in order to improve small sample properties. A detailed discussion of cross-fitted DML can be found in Chernozhukov et al. (2017) [5]. The following algorithm generalizes our proposed method to a $K$ -fold cross fitted version. We assume that $n$ is divisible by $K$ in order to simplify notation.

Algorithm 1.

1) Take a $K$ -fold random partition $(I_{k})_{k = 1}^{K}$ of observation indices $[n] = {1, \dots, n}$ such that the size of each fold $I_{k}$ is $N$ . Also, for each $k \in [K] = {1, \dots, K}$ , define $I_{k}^{c} := {1, \dots, N} ∖ I_{k}$ . 2) For each $k \in [K]$ and $r = 1, \dots, d$ , construct an estimator

{^η}_{k, m_{r}} = {^η}_{m_{r}} ((X_{i})_{i \in I_{k}^{c}})

by lasso/ post-lasso or square root lasso. 3) For each $k \in [K]$ , construct an estimator ${^θ}_{k} = ({^θ}_{k, m_{1}}, \dots, {^θ}_{k, m_{d}})$ as in 3.1:


	$\leq ϵ_{n}$

with $E_{N, k} [ψ_{m_{r}} (X_{i})] = N^{- 1} \sum_{i \in I_{k}} ψ_{m_{r}} (X_{i})$ . 4) Aggregate the estimators:

{^θ}^{K} = \frac{1}{K} K \sum k = 1 {^θ}_{k} .

5) For $r = 1, \dots, d$ construct the uniform valid confidence interval

[{^θ}_{m_{r}}^{K} - \frac{c_{α} {^σ}_{m_{r}}^{K}}{n}, {^θ}_{m_{r}}^{K} + \frac{c_{α} {^σ}_{m_{r}}^{K}}{n}]

with

	${^J}_{m_{r}}^{K}$	$= - \frac{1}{K} K \sum k = 1 (X_{k} (X_{k} - {^η}_{k, m_{r}}^{(2)} X_{- m_{r}})),$
	${^σ}_{m_{r}}^{K}$	$= \sqrt{({^J}_{m_{r}}^{K})^{- 2} \frac{1}{K} K \sum k = 1 (ψ_{m_{r}}^{2} (X, {^θ}_{m_{r}}^{K}, {^η}_{k, m_{r}}))} .$

$c_{α}$ is the $1 - α$ bootstrap quantile of $sup r = 1, \dots, d {^N}_{m_{r}}$ with

{^N}_{m_{r}} = \frac{1}{\sqrt{n}} n \sum i = 1 ξ_{i} {^ψ}_{m_{r}}^{K} (X^{(i)})

where $(ξ_{i})_{i = 1}^{n}$ are independent standard normal random variables which are independent from $(X^{(i)})_{i = 1}^{n}$ and

{^ψ}_{m_{r}}^{K} (X) := - {({^σ}_{m_{r}}^{K} {^J}_{m_{r}}^{K})}^{- 1} ψ_{m_{r}} (X, {^θ}_{m_{r}}^{K}, {^η}_{m_{r}}^{K}) .

The confidence region above corresponds to (4.2). Confidence regions corresponding to (4.3) or (4.4) can be constructed in an analogous way.

6 Simulation Study

This section provides a simulation study on the proposed method. In each example the precision matrix of the Gaussian graphical model is generated as in the $R$ -package $h u g e$ [17]. Hence, the corresponding adjacency matrix $A$ is generated by setting the nonzero off-diagonal elements to be one and each other element to be zero. To obtain a positive definite pre-version of the precision matrix we set

Φ_{p r e} := v \cdot A + (| Λ_{min} (v \cdot A) | + 0.1 + u) \cdot I_{p \times p} .

Here $v = 0.3$ and $u = 0.1$ are chosen to control the magnitude of partial correlations. The covariance matrix $Σ$ is generated by inverting $Φ_{p r e}$ and scaling the variances to one. The corresponding precision matrix $Φ$ is given by $Σ^{- 1}$ . For some given $p$ we generate $n = 200$ independent samples of

X = (X_{1}, \dots, X_{p}) \sim N (0, Σ)

and evaluate whether our test statistic would reject the null hypothesis for a specific set of edges

M

which satisfies the null hypothesis. Finally the acceptance rate is calculated over

l = 1000

independent simulations for a given confidence level

1 - α = 0.95

6.1 Simulation settings

In our simulation study we estimate the correlation structure of four different designs that are described in the following.

6.1.1 Example 1: Random Graph

Each pair of off-diagonal elements of the covariance matrix of the first $p - 1$ regressors is randomly set to non-zero with probability $p r o b = 5 / p$ . The last regressor is added as an independent random variable. It results in about $(p - 1) \cdot (p - 2) \cdot p r o b / 2$ edges in the graph. The corresponding precision matrix is of the form

Φ:=⎛⎜ ⎜ ⎜ ⎜ ⎜ ⎜⎝\raisebox−15.0pt{B}0⋮00⋯01⎞⎟ ⎟ ⎟ ⎟ ⎟ ⎟⎠

where $B$ is a sparse matrix. We test the hypothesis, whether the last regressor is independent from all other regressors, corresponding to

M = {(p, 1), \dots, (p, p - 1)} .

6.1.2 Example 2: Cluster Graph

The regressors are evenly partitioned into $g = 4$ disjoint groups. Each pair of off-diagonal elements $Φ_{(i, j)}$ is set non-zero with probability $p r o b = 5 / p$ , if both $i$ and $j$ belong to the same group. It results in about $g \cdot (p / g) \cdot (p / g - 1) \cdot p r o b / 2$ edges in the graph. The precision Matrix is of the form

Φ:=⎛⎜ ⎜ ⎜⎝B1\huge\mbox{{0}}B2B3\huge\mbox{{0}}B4⎞⎟ ⎟ ⎟⎠

where each block $B_{i}$ is a sparse matrix. We test the hypothesis that the first two hubs are conditionally independent. This corresponds to testing the tuples

M = {(1, p / 4 + 1), \dots, (1, p / 2), (2, p / 4 + 1), \dots, (p / 4, p / 2)} .

6.1.3 Example 3: Approximately Sparse Random Graph

In this example we generate a random graph structure as in example $1$ , but instead of setting the other elements of the adjacency matriy $A$

to zero we generate independent random entries from a uniform distribution on

[- a, a]

with

a = 1 / 20

. This results in a precision matrix of the form

Φ:=⎛⎜ ⎜ ⎜ ⎜ ⎜ ⎜⎝\raisebox−15.0pt{B}0⋮00⋯01⎞⎟ ⎟ ⎟ ⎟ ⎟ ⎟⎠

where $B$ is not a sparse matrix anymore. We then again test the hypothesis, whether the last regressor is independent from all other regressors, corresponding to

M = {(p, 1), \dots, (p, p - 1)} .

6.1.4 Example 4: Independent Graph

By setting

Φ := I_{p \times p}

we generate samples of $p$ independent normal distributed random variables. We can test the hypothesis whether the regressors are independent by choosing

M = {(1, 2), \dots, (1, p), (2, 3), \dots, (p - 1, p)} .

6.2 Simulation results

We provide simulated acceptance rates for all of the examples above calculated by the function GGMtest with $B = 500$ bootstrap samples. Confidence Interval I corresponds to the standard case in (4.2), whereas Confidence Interval II is based on the approximation of the sphere in (4.3). In summary, the results reveal that the empirical acceptance rate is, on average, close to the nominal level of $95 %$ with a mean absolute deviation of $2.676 %$ over all simulations. The Confidence Interval II has got a mean absolute deviation of $2.000 %$ and performs significantly better than Confidence Interval I with a mean absolute deviation of $3.347 %$ . More complex S-sparsely convex sets seem to result in better acceptance rates, whereas higher exponents do not improve the rates. The lowest mean absolute deviation ( $1.463 %$ ) is achieved in table 2 for $S = 5$ , $e x p = 2$ and without cross-fitting.

Model	p	d	lasso	post-lasso	sqrt-lasso	lasso	post-lasso	sqrt-lasso
			Confidence Interval I			Confidence Interval II
random	20	19	0.929	0.936	0.931	0.923	0.931	0.930
	50	49	0.909	0.914	0.909	0.920	0.918	0.928
	100	99	0.915	0.918	0.915	0.924	0.926	0.926
cluster	20	25	0.909	0.940	0.913	0.911	0.932	0.915
	40	100	0.914	0.918	0.918	0.930	0.938	0.940
	60	225	0.895	0.893	0.899	0.917	0.922	0.925
approx	20	19	0.929	0.929	0.929	0.942	0.942	0.942
	50	49	0.906	0.906	0.906	0.919	0.919	0.919
	100	99	0.897	0.897	0.897	0.933	0.933	0.933
indepent	5	10	0.929	0.929	0.929	0.929	0.929	0.929
	10	45	0.926	0.926	0.926	0.931	0.931	0.931
	20	190	0.899	0.899	0.899	0.919	0.919	0.919

Table 1: Simulation results for S=1,exp=1 and 1-fold

Model	p	d	lasso	post-lasso	sqrt-lasso	lasso	post-lasso	sqrt-lasso
			Confidence Interval I			Confidence Interval II
random	20	19	0.956	0.935	0.951	0.956	0.935	0.950
	50	49	0.911	0.927	0.922	0.904	0.939	0.925
	100	99	0.924	0.912	0.921	0.922	0.920	0.922
cluster	20	25	0.965	0.954	0.966	0.938	0.942	0.941
	40	100	0.935	0.941	0.947	0.932	0.939	0.944
	60	225	0.929	0.930	0.935	0.937	0.941	0.946
approx	20	19	0.964	0.964	0.964	0.954	0.954	0.954
	50	49	0.946	0.945	0.946	0.945	0.945	0.945
	100	99	0.909	0.909	0.909	0.937	0.937	0.937
indepent	5	10	0.947	0.947	0.947	0.957	0.957	0.957
	10	45	0.932	0.932	0.932	0.934	0.934	0.934
	20	190	0.934	0.934	0.934	0.941	0.941	0.941

Table 2: Simulation results for S=5,exp=1 and 1-fold

Model	p	d	lasso	post-lasso	sqrt-lasso	lasso	post-lasso	sqrt-lasso
			Confidence Interval I			Confidence Interval II
random	20	19	0.919	0.916	0.918	0.914	0.919	0.926
	50	49	0.901	0.916	0.907	0.908	0.923	0.928
	100	99	0.920	0.912	0.916	0.920	0.920	0.926
cluster	20	25	0.918	0.936	0.922	0.906	0.919	0.907
	40	100	0.908	0.932	0.923	0.916	0.933	0.936
	60	225	0.895	0.896	0.897	0.909	0.928	0.925
approx	20	19	0.925	0.925	0.925	0.939	0.936	0.939
	50	49	0.903	0.904	0.906	0.926	0.919	0.922
	100	99	0.907	0.901	0.909	0.925	0.924	0.923
indepent	5	10	0.930	0.930	0.930	0.931	0.931	0.931
	10	45	0.923	0.920	0.922	0.946	0.945	0.946
	20	190	0.896	0.893	0.894	0.922	0.920	0.919

Table 3: Simulation results for S=5,exp=2 and 1-fold

Model	p	d	lasso	post-lasso	sqrt-lasso	lasso	post-lasso	sqrt-lasso
			Confidence Interval I			Confidence Interval II
random	20	19	0.927	0.917	0.923	0.945	0.939	0.937
	50	49	0.912	0.905	0.918	0.935	0.928	0.930
	100	99	0.902	0.901	0.903	0.918	0.930	0.925
cluster	20	25	0.921	0.921	0.926	0.927	0.932	0.927
	40	100	0.904	0.899	0.907	0.926	0.923	0.918
	60	225	0.888	0.884	0.885	0.932	0.922	0.928
approx	20	19	0.915	0.918	0.917	0.937	0.930	0.932
	50	49	0.891	0.898	0.897	0.931	0.931	0.932
	100	99	0.894	0.894	0.892	0.931	0.933	0.932
indepent	5	10	0.920	0.919	0.916	0.931	0.929	0.929
	10	45	0.913	0.911	0.910	0.929	0.930	0.931
	20	190	0.891	0.896	0.896	0.922	0.918	0.917

Table 4: Simulation results for S=1,exp=1 and 3-fold

Model	p	d	lasso	post-lasso	sqrt-lasso	lasso	post-lasso	sqrt-lasso
			Confidence Interval I			Confidence Interval II
random	20	19	0.946	0.919	0.940	0.945	0.931	0.943
	50	49	0.922	0.931	0.938	0.917	0.938	0.935
	100	99	0.929	0.918	0.920	0.934	0.943	0.934
cluster	20	25	0.951	0.950	0.962	0.927	0.936	0.939
	40	100	0.924	0.920	0.936	0.925	0.929	0.943
	60	225	0.919	0.917	0.927	0.939	0.937	0.940
approx	20	19	0.955	0.955	0.951	0.960	0.960	0.958
	50	49	0.917	0.917	0.923	0.932	0.932	0.930
	100	99	0.932	0.932	0.934	0.945	0.945	0.942
indepent	5	10	0.935	0.935	0.936	0.950	0.950	0.951
	10	45	0.926	0.926	0.926	0.929	0.929	0.928
	20	190	0.925	0.925	0.926	0.939	0.939	0.940

Table 5: Simulation results for S=5,exp=1 and 3-fold

Model	p	d	lasso	post-lasso	sqrt-lasso	lasso	post-lasso	sqrt-lasso
			Confidence Interval I			Confidence Interval II
random	20	19	0.909	0.906	0.911	0.937	0.923	0.937
	50	49	0.894	0.897	0.901	0.898	0.927	0.932
	100	99	0.895	0.884	0.895	0.928	0.924	0.930
cluster	20	25	0.905	0.892	0.902	0.920	0.899	0.898
	40	100	0.904	0.898	0.924	0.909	0.927	0.935
	60	225	0.887	0.858	0.875	0.920	0.900	0.916
approx	20	19	0.919	0.915	0.918	0.932	0.928	0.933
	50	49	0.907	0.907	0.908	0.930	0.932	0.930
	100	99	0.890	0.891	0.892	0.921	0.928	0.921
indepent	5	10	0.925	0.927	0.926	0.939	0.939	0.938
	10	45	0.910	0.910	0.908	0.924	0.924	0.925
	20	190	0.880	0.880	0.876	0.915	0.913	0.914

Table 6: Simulation results for S=5,exp=2 and 3-fold

Appendix A Proof of Theorem 1

Proof.

We want to use corollary $2.2$ from Belloni et al. (2015) [1]. Consequently, we will show that their assumptions 2.1-2.4 and the growth conditions of corollary $2.2$ hold by modifying the proof of corollary $3.2$ in [1]. Let $m_{r} = (j, k)$ be an arbitrary set in $M$ . We have

due to the assumptions 3 and 4. Define the convex set

T_{m_{r}} = {η = (η^{(1)}, η^{(2)}) : η^{(1)} \in R^{p - 2}, η^{(2)} \in R^{p - 2}}

and endow $T_{m_{r}}$ with the norm

| | η | |_{e} = | | η^{(1)} | |_{2} \lor | | η^{(2)} | |_{2} .

Further let $τ_{n} := \sqrt{\frac{s log (a_{n})}{n}}$ and define the nuisance realization set

	$T_{m_{r}} =$	${(β^{(m_{r})}, γ^{(m_{r})})} \cup {η \in T_{r} : \| \| η^{(1)} \| \|_{0} \lor \| \| η^{(2)} \| \|_{0} \leq C s$
		$, \| \| η^{(1)} - β^{(m_{r})} \| \|_{2} \lor \| \| η^{(2)} - γ^{(m_{r})} \| \|_{2} \leq C τ_{n}, \| \| η^{(2)} - γ^{(m_{r})} \| \|_{1} \leq C \sqrt{s} τ_{n}} .$

for a sufficiently large constant $C > 0$ . First we verify Assumption 2.1 (i). The moment condition holds since

	$E [ψ_{m_{r}} (X, θ_{m_{r}}, η_{m_{r}})]$
	$= E [ε^{(m_{r})} ν^{(m_{r})}]$
	$= E [E [ε^{(m_{r})} ν^{(m_{r})} \| X_{- j}]] = E [ν^{(m_{r})} E [ε^{(m_{r})} \| X_{- j}]      = 0] = 0.$

In addition, we have

	$S_{n} :$	$= E [max r \| \sqrt{n} E_{n} [ψ_{m_{r}} (X, θ_{m_{r}}, η_{m_{r}})] \|]$
		$= E [sup f \in F G_{n} (f)]$

with $F = {ε^{(m_{r})} ν^{(m_{r})} | r = 1, \dots, d}$ . By the same arguments as in the beginning of proof of theorem 2 we conclude that the envelope $sup f \in F | f |$ of $F$ fulfills

	$\| \| max r \| ε^{(m_{r})} ν^{(m_{r})} \| \| \|_{P, q}$	$= E {[max r {(\| ε^{(m_{r})} ν^{(m_{r})} \|)}^{q}]}^{1 / q}$
		$\leq E {[max r {(\| ε^{(m_{r})} \|)}^{2 q}]}^{1 / 2 q} E {[max r {(\| ν^{(m_{r})} \|)}^{2 q}]}^{1 / 2 q}$
		$\leq C (q) log (d) .$

Since $| F | = d$ , using lemma O.2 (Maximal Inequality I) in [1], we have

S_{n} \leq C {log}^{1 / 2} (d) + C (q) {log}^{1 / 2} (d) {(n^{\frac{2}{q}} \frac{{log}^{3} (d)}{n})}^{1 / 2} ≲ {log}^{1 / 2} (d)

by the assumption 2. Hence, assumption 3 implies that for all $r = 1, \dots, d$ , $Θ_{m_{r}}$ contains an interval of radius $C n^{- \frac{1}{2}} S_{n} log (n)$ centered at $θ_{m_{r}}$ for all sufficiently large $n$ for any constant $C$ . Assumption 2.1 (i) follows.

For all $m_{r} \in M$ , the map $(θ, η) \mapsto ψ_{m_{r}} (X, θ, η)$ is twice continuously Gateaux-differentiable on $Θ_{m_{r}} \times T_{m_{r}}$ , and so is the map $(θ, η) \mapsto E [ψ_{m_{r}} (X, θ, η)]$ . Further we have

	$D_{m_{r}, 0} [η, η_{m_{r}}] :$	$= \partial_{t} E [ψ_{m_{r}} (X, θ_{m_{r}}, η_{m_{r}} + t (η - η_{m_{r}}))] {∣ ∣}_{t = 0}$
		$= E [\partial_{t} {(X_{j} - θ_{m_{r}} X_{k} - (η_{m_{r}}^{(1)} + t (η^{(1)} - η_{m_{r}}^{(1)})) X_{- m_{r}})$
		$(X_{k} - (η_{m_{r}}^{(2)} + t (η^{(2)} - η_{m_{r}}^{(2)})) X_{- m_{r}})}] {∣ ∣}_{t = 0}$
		$= E [ε^{(m_{r})} (η_{m_{r}}^{(2)} - η^{(2)}) X_{- m_{r}}] + E [(η_{m_{r}}^{(1)} - η^{(1)}) X_{- m_{r}} ν^{(m_{r})}]$
		$= 0.$

Therefore, Assumptions 2.1 (ii) and 2.1 (iii) hold. Remark that

	$\| J_{m_{r}} \|$	$= \| \partial_{θ} E [ψ_{m_{r}} (X, θ, η_{m_{r}})] \|_{θ = θ_{m_{r}}} \|$
		$= \| E [- X$

	$\| \| max r \| ε^{(m_{r})} ν^{(m_{r})} \| \| \|_{P, q}$	$= E {[max r {(\| ε^{(m_{r})} ν^{(m_{r})} \|)}^{q}]}^{1 / q}$
		$\leq E {[max r {(\| ε^{(m_{r})} \|)}^{2 q}]}^{1 / 2 q} E {[max r {(\| ν^{(m_{r})} \|)}^{2 q}]}^{1 / 2 q}$
		$\leq C (q) log (d) .$

Uniform Inference in High-Dimensional Gaussian Graphical Models

Learning mixed graphical models from data with p larger than n

Characterization of differentially expressed genes using high-dimensional co-expression networks

Temporal Point Process Graphical Models

Exact Inference of Hidden Structure from Sample Data in Noisy-OR Networks

On Identifying Significant Edges in Graphical Models of Molecular Networks

Connecting actuarial judgment to probabilistic learning techniques with graph theory

Graphical model inference with external network data

1 Introduction

2 Setting

Comment 2.1.

3 Estimation

Comment 3.1.

4 Main results

Theorem 1.

Comment 4.1.

5 The function $G G M t e s t$

5.1 cross-fitting

Algorithm 1.

6 Simulation Study

6.1 Simulation settings

6.1.1 Example 1: Random Graph

6.1.2 Example 2: Cluster Graph

6.1.3 Example 3: Approximately Sparse Random Graph

6.1.4 Example 4: Independent Graph

6.2 Simulation results

Appendix A Proof of Theorem 1

Proof.

	$X_{j}$	$= β^{(j)} X_{- j} + ε^{(j)}$
		$= θ_{m_{r}} X_{k} + β^{(m_{r})} X_{- m_{r}} + ε^{(m_{r})}$
		$= θ_{m_{r}} X_{k} + (β^{(1, m_{r})} + β^{(2, m_{r})}) X_{- m_{r}} + ε^{(m_{r})}$

	$X_{k}$	$= γ^{(j, k)} X_{- {j, k}} + ν^{(j, k)}$
		$= γ^{(m_{r})} X_{- m_{r}} + ν^{(m_{r})}$
		$= (γ^{(1, m_{r})} + γ^{(1, m_{r})}) X_{- m_{r}} + ν^{(m_{r})}$

Uniform Inference in High-Dimensional Gaussian Graphical Models

Related Research

Learning mixed graphical models from data with p larger than n

Characterization of differentially expressed genes using high-dimensional co-expression networks

Temporal Point Process Graphical Models

Exact Inference of Hidden Structure from Sample Data in Noisy-OR Networks

On Identifying Significant Edges in Graphical Models of Molecular Networks

Connecting actuarial judgment to probabilistic learning techniques with graph theory

Graphical model inference with external network data

1 Introduction

2 Setting

Comment 2.1.

3 Estimation

Comment 3.1.

4 Main results

Theorem 1.

Comment 4.1.

5 The function GGMtest

5.1 cross-fitting

Algorithm 1.

6 Simulation Study

6.1 Simulation settings

6.1.1 Example 1: Random Graph

6.1.2 Example 2: Cluster Graph

6.1.3 Example 3: Approximately Sparse Random Graph

6.1.4 Example 4: Independent Graph

6.2 Simulation results

Appendix A Proof of Theorem 1

Proof.

5 The function $G G M t e s t$