M-estimation in high-dimensional linear model

1 Introduction

For the classical linear regression model $Y = X β + ε$ , we are interested in the problem of variable selection and estimation, where $Y = (y_{1}, y_{2}, . . ., y_{n})^{T}$

is the response vector,

X = (X_{1}, X_{2}, . . ., X_{p_{n}}) = (x_{1}, x_{2}, . . ., x_{n})^{T} = (x_{i j})_{n \times p_{n}}

is an

n \times p_{n}

design matrix, and

ε = (ε_{1}, ε_{2}, . . ., ε_{n})^{T}

is a random vector. The main topic is how to estimate the coefficients vector

β \in R_{n}^{p}

when

p_{n}

increases with sample size

n

and many elements of

β

equal zero. We can transfer this problem into a minimization of a penalized least squares objective function

_{n} = arg min β Q_{n} (β_{n}), Q_{n} (β_{n}) = ∥ Y - X β_{n} ∥^{2} + p_{n} \sum j = 1 p_{λ_{n}} (| β_{n} j |),

where $| | \cdot | |$ is the $l_{2}$ norm of the vector, $λ_{n}$ is a tuning parameter, and $p_{λ_{n}} (| t |)$ a penalty term. We have known that least squares estimation is not robust, especially when the data exists abnormal values or the error term has the heavy tailed distribution.

In this paper we consider the loss function be least absolute deviation,i.e., minimize the following objective function:

_{n} = arg min β Q_{n} (β_{n}), Q_{n} (β_{n}) = \frac{1}{n} n \sum i = 1 | y_{i} - x_{i}^{T} β_{n} | + p_{n} \sum j = 1 p_{λ_{n}} (| β_{n} j |),

where the loss function is least absolute deviation(LAD for short), that does not need the noise obeys a gaussian distribution and be more robust than least squares estimation. In fact, LAD estimation is the special case of M-estimation, which is named by Huber(1964, 1973, 1981)

[1] [2] [3]firstly and can be obtained by minimizing the objective function

Q_{n} (β_{n}) = \frac{1}{n} n \sum i = 1 ρ (y_{i} - x_{i}^{T} β_{n}),

where the function $ρ$ can be selected. For example, if we choose $ρ (x) = \frac{1}{2} x^{2} 1_{| x | \leq c} + (c | x | - c^{2} / 2) 1_{| x | > c}$ , where $c > 0$ , Huber estimator can be obtained; if we choose $ρ (x) = | x |^{q}$ , where $1 \leq q \leq 2$ , $L_{q}$ estimator will be obtained, with two special cases: LAD estimator for $q = 1$ and OLS estimator for $q = 2$ . If we choose $ρ (x) = α x^{+} + (1 - α) (- x)^{+}$ , where $0 < α < 1, x^{+} = max (x, 0)$ , we call it quantile regression, and can also get LAD estimator for $α = 1 / 2$ especially.

When $p_{n}$ approaches infinity as $n$ tends to infinity, we assume that the function $ρ$ is convex and not monotone, and the monotone function $φ$ is the derivative of $ρ$ . By imposing the appropriate regularity conditions, Huber(1973), Portnoy(1984)[4],Welsh(1989)[5] and Mammen(1989)[6] have proved that the M-estimator enjoyed the properties of consistency and asymptotic normality, where Welsh(1989) gave the weaker condition imposed on $φ$ and the stronger condition on $p_{n} / n$ . Bai and Wu [7] further pointed that the condition on $p_{n}$ could be a part of the integrable condition imposed on design matrix. Moreover, He and Shao(2000)[8] studied the asymptotic properties of M-estimator in the case of the generalized model setting and the dimension $p_{n}$ getting bigger and bigger. Li(2011)[9] obtained the Oracle property of non-concave penalized M-estimator in high-dimensional model with the condition of $p_{n} log n / n \to 0, p_{n}^{2} / n \to 0$ , and proposed RSIS to make variable selection by applying rank sure independence screening method in the ultra high-dimensional model. Zou and Li(2008)[10] combined penalized function and local linear approximation method(LLA) to prove that the obtained estimator enjoyed good asymptotic properties, and demonstrated this method improved the computational efficiency of local quadratic approximation(LQA) in the part of simulation.
Inspired by this, in this paper we consider the following problem:

_{n} = arg min β_{n} Q_{n} (β_{n}), Q_{n} (β_{n}) = \frac{1}{n} n \sum i = 1 ρ (y_{i} - x_{i}^{T} β_{n}) + p_{n} \sum j = 1 p_{λ_{n}}^{'} (| {~ β}_{n j} |) | β_{n j} |,

where $p_{λ_{n}}^{'} (\cdot)$ is the derivative of the penalized function, and ${~ β}_{n} = ({~ β}_{n 1}, {~ β}_{n 2}, . . ., {~ β}_{n p_{n}})^{T}$ is the non-penalized estimator.

In this paper, we assume that the function $ρ$ is convex, hence the objective function is still convex and the obtained local minimizer is global minimizer.

2 Main results

For the convenience of statement, we first give some notations. Let $β_{0} = (β_{01}, β_{02}, . . ., β_{0 p})^{T}$ be the true parameter. Without loss of generality, we assume the first $k_{n}$ coefficients of covariates are nonzero, $p_{n} - k_{n}$ be coviariates with zero coefficients. $β_{0} = (β_{0 (1)}^{T}, β_{0 (2)}^{T})^{T},_{n} = ({^β}_{n (1)}^{T}, {^β}_{n (2)}^{T})^{T}$ correspondingly. For the given symmetric matrix $Z$ , denote by $λ_{m i n} (Z)$ and $λ_{m a x} (Z)$

the minimum and maximum eigenvalue of

Z

, respectively. Denote

\frac{X^{T} X}{n} := D

and

D = (\begin{matrix} D_{11} & D_{12} D_{21} & D_{22} \end{matrix}),

where

D_{11} = \frac{1}{n} X_{(1)}^{T} X_{(1)}

. Finally we denote that

c_{n} = max {| p_{λ_{n}}^{'} (| {~ β}_{n j} |) | : {~ β}_{n j} \neq 0, 1 \leq j \leq p_{n}}

Next, we state some assumptions which will be needed in the following results.
$(A_{1})$ The function $ρ$ is convex on $R$ , and its left derivative and right derivative $φ_{+} (\cdot), φ_{-} (\cdot)$ satisfies that $φ_{-} (t) \leq φ (t) \leq φ_{+} (t), \forall t \in R$ .
$(A_{2})$ The error term $ε$ is i.i.d, and the distribution function $F$ of $ε_{i}$ satisfies $F (S) = 0$ , where $S$ is the set of discontinuous points of $φ$ .

Moreover, $E [φ (ε_{i})] = 0, 0 < E [φ^{2} (ε_{i})] = σ^{2} < \infty$ , and $G (t) \equiv E [φ (ε_{i} + t)] = γ t + o (| t |)$ , where $γ > 0$ . Besides these, we assume that $lim t \to 0 E [φ (ε_{i} + t) - φ (ε_{i})]^{2} = 0$ .
$(A_{3})$ There exist constants $τ_{1}, τ_{2}, τ_{3}, τ_{4}$ such that $0 < τ_{1} \leq λ_{m i n} (D) \leq λ_{m a x} (D) \leq τ_{2}$ and $0 < τ_{3} \leq λ_{m i n} (D_{11}) \leq λ_{m a x} (D_{11}) \leq τ_{4}$ .
$(A_{4})$ $λ_{n} \to 0 (n \to \infty)$ , $p_{n} = O (n^{1 / 2}), c_{n} = O (n^{- 1 / 2})$ .
$(A_{5})$ Let $z_{i}$ be the transpose of the $i$ th row vector of $X_{(1)}$ , such that $lim n \to \infty n^{- \frac{1}{2}} max 1 \leq i \leq n z_{i}^{T} z_{i} = 0.$

It is worth mentioning that conditions $(A_{1})$ and $(A_{2})$ are classical assumptions for M-estimation in linear model, which can be found in many references, for example Bai, Rao and Wu(1992)[11]and Wu(2007)[12]. The condition $(A_{3})$ is frequently used for sparse model in the linear model regression theory, which requires that the eigenvalues of the matrices $D$ and $D_{11}$ are bounded. The condition $(A_{4})$ is weaker than that in previous references. In the condition $(A_{4})$ we broad the order of $p_{n}$ to $n^{1 / 2}$ , but in the references Huber(1973) and Li,Peng and Zhu(2011)[9] they required that $p_{n}^{2} / n \to 0$ , Portnoy(1984) required $p_{n} log p_{n} / n \to 0$ , and Mammen(1989) required $p_{n}^{3 / 2} log p_{n} / n \to 0$ . Compared with these results, it is obvious that our sparse condition is much weaker. The condition $(A_{5})$ is the same as that in Huang, Horowitz and Ma(2008)[13], which is used to prove the asymptotic properties of the nonzero part of M-estimation.

Theorem 2.1 (Consistency of estimator) If the conditions $(A_{1}) - (A_{4})$ hold, there exists a non-concave penalized M-estimation $_{n}$ , such that

∥_{n} - β_{0} ∥ = O_{P} ((p_{n} / n)^{1 / 2}) .

Remark 2.1 From Theorem 2.1, we can obtain that there exists a global M-estimation $_{n}$ if we choose the appropriate tuning parameter $λ_{n}$ , moreover this M-estimation is $(n / p_{n})^{1 / 2}$ -consistent. This convergence rate is the same as that in the references Huber(1973) and Li,Peng and Zhu(2011).

Theorem 2.2 (The sparse of the model) If the conditions $(A_{1}) - (A_{4})$ hold and $λ_{m i n} (D) > λ_{m a x} (\frac{1}{n} \sum_{i = 1}^{n} J_{i} J_{i}^{T})$ , for the non-concave penalized M-estimation $_{n}$ we have

P ({^β}_{n (2)} = 0) \to 1.

Remark 2.2

By Theorem 3.2, we can get that under the suitable conditions the global M-estimation of zero-coefficient variables goes to zero with a high probability when

n

is large enough. This also shows that the model is sparse.

Theorem 2.3 (Oracle property) If the conditions $(A_{1}) - (A_{5})$ hold and $λ_{m i n} (D) > λ_{m a x} (\frac{1}{n} \sum_{i = 1}^{n} J_{i} J_{i}^{T})$ , with probability converging to one the non-concave penalized M-estimation $_{n} = ({^β}_{n (1)}^{T}, {^β}_{n (2)}^{T})^{T}$ has the following properties:

(1)(The consistency of the model selection) ${^β}_{n (2)} = 0$ ;

(2)(Asymptotic normality)

	$\sqrt{n} s_{n}^{- 1} u^{T} ({^β}_{n (1)} - β_{0 (1)})$	$= n \sum i = 1 n^{- 1 / 2} s_{n}^{- 1} γ^{- 1} u^{T} D_{11} z_{i}^{T} φ (ε_{i}) + o_{P} (1)$
		$\small d⟶N(0,1),$

where $s_{n}^{2} = σ^{2} γ^{- 1} u^{T} D_{11}^{- 1} u$ , and $u$ is any $k_{n}$ dimensional vector such that $∥ u ∥ \leq 1$ . Meanwhile, $z_{i}$ is the transpose of the $i$ th row vector of a $k_{n} \times k_{n}$ matrix $X_{(1)}$ .

Remark 2.3 From Theorem 2.3, M-estimation enjoys Oracle property, that is, the adaptive bridge estimator can correctly select covariates with nonzero coefficients with probability converging to one and that the estimator of nonzero coefficients has the same asymptotic distribution that they would have if the zero coefficients were known in advance.

Remark 2.4 In Fan and Peng(2004)[14], the authors obtained that the non-concave penalized M-estimation has the property of consistency with the condition $p_{n}^{4} / n \to 0$ , and enjoyed the property of asymptotic normality with the condition $p_{n}^{5} / n \to 0$ . By Theorem 3.1-3.3, we can see that the corresponding conditions we exert is quite weak.

3 Proofs of main results

The proof of Theorem 2.1: Let $α_{n} = (p_{n} / n)^{1 / 2} + p_{n}^{1 / 2} c_{n}$ , where $u$ is a any $p_{n}$ -dimensional vector such that $∥ u ∥ = C$ . In the following part we only need to prove that there exists a great enough positive constant $C$ such that

liminf n \to \infty P {inf ∥ u ∥ = C Q_{n} (β_{0} + α_{n} u) > Q_{n} (β_{0})} \geq 1 - ε,

for any $ε > 0$ , that is, there at least exists a local minimizer $_{n}$ such that $∥_{n} - β_{0} ∥ = O_{P} (α_{n})$ in the closed ball ${β_{0} + α_{n} u : ∥ u ∥ \leq C}$ . Firstly by the triangle inequality we can get that

		$Q_{n} (β_{0} + θ u) - Q_{n} (β_{0})$		(3.2)
		$= \frac{1}{n} n \sum i = 1 [ρ (y_{i} - x_{i}^{T} (β_{0} + α_{n} u)) - ρ (y_{i} - x_{i}^{T} β_{0})] + p_{n} \sum j = 1 p_{λ_{n}}^{'} (\| {~ β}_{n j} \|) (\| β_{0 j} + α_{n} u_{j} \| - \| β_{0 j} \|)$
		$\geq \frac{1}{n} n \sum i = 1 [ρ (y_{i} - x_{i}^{T} (β_{0} + α_{n} u)) - ρ (y_{i} - x_{i}^{T} β_{0})] - α_{n} p_{n} \sum j = 1 p_{λ_{n}}^{'} (\| {~ β}_{n j} \|) \| u_{j} \|$
		$:= T_{1} + T_{2},$

where $T_{1} = \frac{1}{n} \sum_{i = 1}^{n} [ρ (y_{i} - x_{i}^{T} (β_{0} + α_{n} u)) - ρ (y_{i} - x_{i}^{T} β_{0})]$ , $T_{2} = - α_{n} \sum_{j = 1}^{p_{n}} p_{λ_{n}}^{'} (| {~ β}_{n j} |) | u_{j} |$ . Noticing that

$T_{1}$	$= \frac{1}{n} n \sum i = 1 [ρ (y_{i} - x_{i}^{T} (β_{0} + α_{n} u)) - ρ (y_{i} - x_{i}^{T} β_{0})]$	(3.3)
	$= \frac{1}{n} n \sum i = 1 [ρ (ε_{i} - α_{n} x_{i}^{T} u) - ρ (ε_{i})]$
	$= \frac{1}{n} n \sum i = 1 \int_{0}^{- α_{n} x_{i}^{T} u} [φ (ε_{i} + t) - φ (ε_{i})] d t - \frac{1}{n} α_{n} n \sum i = 1 φ (ε_{i}) x_{i}^{T} u$
	$:= T_{11} + T_{12},$

where $T_{11} = \frac{1}{n} \sum_{i = 1}^{n} \int_{0}^{- α_{n} x_{i}^{T} u} [φ (ε_{i} + t) - φ (ε_{i})] d t$ , $T_{12} = - \frac{1}{n} α_{n} \sum_{i = 1}^{n} φ (ε_{i}) x_{i}^{T} u$ ,
combining with Von-Bahr Esseen inequality and the fact that $| T_{12} | \leq \frac{1}{n} α_{n} ∥ u ∥ ∥ \sum_{i = 1}^{n} φ (ε_{i}) x_{i} ∥$ , we instantly have

E [∥ n \sum i = 1 φ (ε_{i}) x_{i} ∥^{2}] \leq n n \sum i = 1 E [∥ φ (ε_{i}) x_{i} ∥^{2}] = n n \sum i = 1 E [φ^{2} (ε_{i}) x_{i}^{T} x_{i} \leq n^{2} p_{n} σ^{2},

hence

| T_{12} | = O_{P} (α_{n} p_{n}^{1 / 2}) ∥ u ∥ = O_{P} ((p_{n}^{2} / n)^{1 / 2}) .

Secondly for $T_{11}$ , let $T_{11} = \sum_{i = 1}^{n} A_{i n}$ , where $A_{i n} = \frac{1}{n} \int_{0}^{- α_{n} x_{i}^{T} u} [φ (ε_{i} + t) - φ (ε_{i})] d t$ , so

T_{11} = n \sum i = 1 [A_{i n} - E (A_{i n})] + n \sum i = 1 E (A_{i n}) := T_{111} + T_{112} .

We can easily obtain $E (T_{111}) = 0$ . From Von-Bahr Esseen inequality, Schwarz inequality and the condition $(B_{3})$ , it follows that

	$v a r (T_{111})$	$= v a r (n \sum i = 1 A_{i n}) \leq \frac{1}{n} n \sum i = 1 E (\int_{0}^{- α_{n} x_{i}^{T} u} [φ (ε_{i} + t) - φ (ε_{i})] d t)^{2}$
		$\leq \frac{1}{n} n \sum i = 1 \| α_{n} x_{i}^{T} u \| \| \int_{0}^{- α_{n} x_{i}^{T} u} E [φ (ε_{i} + t) - φ (ε_{i})]^{2} d t \|$
		$= \frac{1}{n} n \sum i = 1 o_{P} (1) (α_{n} x_{i}^{T} u)^{2} = \frac{1}{n} o_{P} (1) α_{n}^{2} n \sum i = 1 u^{T} x_{i} x_{i}^{T} u$
		$= o_{P} (1) α_{n}^{2} u^{T} D u \leq λ_{m a x} (D) o_{P} (1) α_{n}^{2} ∥ u ∥^{2} = o_{P} (α_{n}^{2}) ∥ u ∥^{2},$

together by Markov inequality yields that

P (| T_{111} | > C_{1} α_{n} ∥ u ∥) \leq \frac{v a r (T_{111})}{C_{1}^{2} α_{n}^{2} ∥ u ∥^{2}} \leq \frac{o_{P} (α_{n}^{2}) ∥ u ∥^{2}}{C_{1}^{2} α_{n}^{2} ∥ u ∥^{2}} \to 0 (n \to \infty),

hence

T_{111} = o_{P} (α_{n}) ∥ u ∥ .

As for $T_{112}$ ,

	$T_{112}$	$= n \sum i = 1 E (A_{i n}) = \frac{1}{n} n \sum i = 1 \int_{0}^{- α_{n} x_{i}^{T} u} [γ t + o (\| t \|)] d t$
		$= \frac{1}{n} n \sum i = 1 (\frac{1}{2} γ α_{n}^{2} u^{T} x_{i} x_{i}^{T} u + o_{P} (1) α_{n}^{2} u^{T} x_{i} x_{i}^{T} u)$
		$= \frac{1}{2} γ α_{n}^{2} u^{T} D u + o_{p} (1) α_{n}^{2} u^{T} D u$
		$\geq [\frac{1}{2} γ λ_{m i n} (D) + o_{P} (1)] α_{n}^{2} ∥ u ∥^{2} .$

Finally considering $T_{2}$ , we can easily obtain

T_{2} \leq (p_{n})^{1 / 2} α_{n} max {| p_{λ_{n}}^{'} (| {~ β}_{n j} |) |, 1 \leq j \leq k_{n}} ∥ u ∥ = (p_{n})^{1 / 2} α_{n} c_{n} ∥ u ∥ \leq α_{n}^{2} ∥ u ∥ .

This together with (3.3)-(3.7) yields that we can choose a great enough constant $C$ such that $T_{111}$ and $T_{2}$ is controlled by $T_{112}$ , which follows that there at least exists a local minimizer $_{n}$ such that $∥_{n} - β_{0} ∥ = O_{P} (α_{n})$ in the closed ball ${β_{0} + α_{n} u : ∥ u ∥ \leq C}$ . $□$

The proof of Theorem 2.2: From Theorem 2.1, as long as we choose a great enough constant $C$ and appropriate $α_{n}$ , then $_{n}$ will be in the ball ${β_{0} + α_{n} u : ∥ u ∥ \leq C}$ with probability converging to one, where $α_{n} = (p_{n} / n)^{1 / 2} + p_{n}^{1 / 2} c_{n}$ . For any $p_{n}$ -dimensional vector $β_{n}$ , now we denote $β_{n} = (β_{n (1)}^{T}, β_{n (2)}^{T})^{T}, β_{n (1)} = β_{0 (1)} + α_{n} u_{(1)}, β_{n (2)} = β_{0 (2)} + α_{n} u_{(2)} = α_{n} u_{(2)}$ , where $β_{0} = (β_{0 (1)}^{T}, β_{0 (2)}^{T})^{T}, ∥ u ∥^{2} = ∥ u_{(1)} ∥^{2} + ∥ u_{(2)} ∥^{2} \leq C^{2}$ . Meanwhile let

V_{n} (u_{(1)}, u_{(2)}) = Q_{n} ((β_{n (1)}^{T}, β_{n (2)}^{T})^{T}) - Q_{n} ((β_{0 (1)}^{T}, 0^{T})^{T}),

then by minimizing $V_{n} (u_{(1)}, u_{(2)})$ we can obtain the estimator $_{n} = ({^β}_{n (1)}^{T}, {^β}_{n (2)}^{T})^{T}$ , where $∥ u_{(1)} ∥^{2} + ∥ u_{(2)} ∥^{2} \leq C^{2}$ . In the following part, we will prove that as long as $∥ u ∥ \leq C, ∥ u_{(2)} ∥ > 0$ ,

P (V_{n} (u_{(1)}, u_{(2)}) - V_{n} (u_{(1)}, 0) > 0) \to 1 (n \to \infty)

holds, for any $p_{n}$ -dimensional vector $u = (u_{(1)}^{T}, u_{(2)}^{T})^{T}$ . We can easily find the fact that

		$V_{n} (u_{(1)}, u_{(2)}) - V_{n} (u_{(1)}, 0) = Q_{n} ((β_{n (1)}^{T}, β_{n (2)}^{T})^{T}) - Q_{n} ((β_{n (1)}^{T}, 0^{T})^{T})$
		$= \frac{1}{n} n \sum i = 1 [ρ (ε_{i} - α_{n} H_{i}^{T} u_{(1)} - α_{n} J_{i}^{T} u_{(2)}) - ρ (ε_{i} - α_{n} H_{i}^{T} u_{(1)})] + p_{n} \sum j = k_{n} + 1 p_{λ_{n}}^{'} (\| {~ β}_{n j} \|) \| α_{n} u_{j} \|$
		$= \frac{1}{n} n \sum i = 1 \int_{- α_{n} H_{i}^{T} u_{(1)}}^{- α_{n} H_{i}^{T} u_{(1)} - α_{n} J_{i}^{T} u_{(2)}} [φ (ε_{i} + t) - φ (ε_{i})] d t - \frac{1}{n} α_{n} n \sum i = 1 φ (ε_{i}) J_{i}^{T} u_{(2)}$
		$+ p_{n} \sum j = k_{n} + 1 p_{λ_{n}}^{'} (\| {~ β}_{n j} \|) \| α_{n} u_{j} \| := W_{1} + W_{2} + W_{3},$

where $H_{i}$ and $J_{i}$ are $k_{n}$ and $p_{n} - k_{n}$ dimensional vectors respectively such that $x_{i} = (H_{i}^{T} + J_{i}^{T})^{T}$ . Similar to the proof of Theorem 3.1, we get that

	$W_{1}$	$= \frac{1}{n} n \sum i = 1 \int_{- α_{n} H_{i}^{T} u_{(1)}}^{- α_{n} H_{i}^{T} u_{(1)} - α_{n} J_{i}^{T} u_{(2)}} [φ (ε_{i} + t) - φ (ε_{i})] d t$
		$= \frac{1}{2 n} n \sum i = 1 γ α_{n}^{2} u^{T} x_{i} x_{i}^{T} u - \frac{1}{2 n} n \sum i = 1 γ α_{n}^{2} u_{(2)}^{T} J_{i} J_{i}^{T} u_{(2)} + o_{P} (1) α_{n}^{2} ∥ u ∥^{2} + o_{P} (1) α_{n} ∥ u ∥$
		$\geq \frac{1}{2} γ α_{n}^{2} [λ_{m i n} (D) - λ_{m a x} (\frac{1}{n} n \sum i = 1 J_{i} J_{i}^{T})] ∥ u ∥^{2} + o_{P} (1) α_{n}^{2} ∥ u ∥^{2} + o_{P} (1) α_{n} ∥ u ∥$

| W_{2} | = | - \frac{1}{n} α_{n} n \sum i = 1 φ (ε_{i}) J_{i}^{T} u_{(2)} | = O_{P} ((p_{n}^{2} / n)^{1 / 2}) ∥ u ∥,

and

	$\| W_{3} \|$	$= \| p_{n} \sum j = k_{n} + 1 p_{λ_{n}}^{'} (\| {~ β}_{n j} \|) \| α_{n} u_{j} \| \| \leq (p_{n})^{1 / 2} α_{n} max {\| p_{λ_{n}}^{'} (\| {~ β}_{n j} \|) \|, k_{n} + 1 \leq j \leq p_{n}} ∥ u ∥$
		$= (p_{n})^{1 / 2} α_{n} c_{n} ∥ u ∥ \leq α_{n}^{2} ∥ u ∥ .$

By formula (3.10)-(3.12) and the condition $λ_{m i n} (D) > λ_{m a x} (\frac{1}{n} \sum_{i = 1}^{n} J_{i} J_{i}^{T})$ , it follows that

		$V_{n} (u_{(1)}, u_{(2)}) - V_{n} (u_{(1)}, 0) \geq \frac{1}{2} γ α_{n}^{2} [λ_{m i n} (D) - λ_{m a x} (\frac{1}{n} n \sum i = 1 J_{i} J_{i}^{T})] ∥ u ∥^{2}$
		$+ o_{P} (1) α_{n}^{2} ∥ u ∥^{2} + o_{P} (1) α_{n} ∥ u ∥] + O_{P} ((p_{n}^{2} / n)^{1 / 2}) ∥ u ∥ + O_{P} (α_{n}^{2}) ∥ u ∥ > 0,$

which yields that as long as $∥ u ∥ \leq C, ∥ u_{(2)} ∥ > 0$ ,

P (V_{n} (u_{(1)}, u_{(2)}) - V_{n} (u_{(1)}, 0) > 0) \to 1 (n \to \infty)

holds, for any $p_{n}$ -dimensional vector $u = (u_{(1)}^{T}, u_{(2)}^{T})^{T}$ . $□$

The proof of Theorem 2.3: It is obvious that the conclusion (1) can be obtained instantly by Theorem 2.2, so we only need to prove the conclusion (2). It follows from Theorem 2.1 that $_{n}$ is consistent of $β_{0}$ and ${^β}_{n (2)} = 0$ with probability converging to one from Theorem 2.2. Therefore $_{n (1)}$ holds that

\frac{\partial Q_{n} (β_{n})}{\partial β_{n (1)}} ∣_{β_{n (1)} = {^β}_{n (1)}} = 0,

that is

- \frac{1}{n} n \sum i = 1 H_{i} φ (y_{i} - H_{i}^{T} {^β}_{n (1)}) + W_{(1)} = 0,

where

W = (p_{λ_{n}}^{'} (| {~ β}_{n 1} |) s g n ({^β}_{n 1}), p_{λ_{n}}^{'} (| {~ β}_{n 2} |) s g n ({^β}_{n 2}), \dots, p_{λ_{n}}^{'} (| {~ β}_{n p_{n}} |) s g n ({^β}_{n p_{n}}))^{T} .

In the following part we give the Taylor expansion of upper left first term:

−1nn∑i=1{Hiφ(yi−HTi^β0(1))−[φ′(yi−HTiβ0(1))HiHTi+oP(1)](^βn(1)−β0(1))}+W(1)=0.

Noticing that $y_{i} = H_{i}^{T} β_{0 (1)} + ε_{i}$ , we have

- \frac{1}{n} n \sum i = 1 H_{i} φ (ε_{i}) + \frac{1}{n} n \sum i = 1 [φ^{'} (ε_{i}) H_{i} H_{i}^{T} + o_{P} (1)] ({^β}_{n (1)} - β_{0 (1)}) + W_{(1)} = 0,

which yields that

	$\frac{1}{n} γ n \sum i = 1 H_{i} H_{i}^{T} ({^β}_{n (1)} - β_{0 (1)})$	$= \frac{1}{n} n \sum i = 1 H_{i} φ (ε_{i}) - W_{(1)} + ({^β}_{n (1)} - β_{0 (1)}) o_{P} (1)$
		$+ \frac{1}{n} n \sum i = 1 (γ - φ^{'} (ε_{i})) H_{i} H_{i}^{T} ({^β}_{n (1)} - β_{0 (1)}) .$

Then as long as $∥ u ∥ \leq 1$ </

M-estimation in high-dimensional linear model

Authors

Analysis of Least square estimator for simple Linear Regression with a uniform distribution error

Scale calibration for high-dimensional robust regression

Debiased and threshold ridge regression for linear model with heteroskedastic and dependent error

Smooth Adjustment for Correlated Effects

Using parameter elimination to solve discrete linear Chebyshev approximation problems

Machine Learning for Partial Identification: Example of Bracketed Data

Support estimation in high-dimensional heteroscedastic mean regression

1 Introduction

2 Main results

3 Proofs of main results

M-estimation in high-dimensional linear model

Authors

Related Research

Analysis of Least square estimator for simple Linear Regression with a uniform distribution error

Scale calibration for high-dimensional robust regression

Debiased and threshold ridge regression for linear model with heteroskedastic and dependent error

Smooth Adjustment for Correlated Effects

Using parameter elimination to solve discrete linear Chebyshev approximation problems

Machine Learning for Partial Identification: Example of Bracketed Data

Support estimation in high-dimensional heteroscedastic mean regression

This week in AI

1 Introduction

2 Main results

3 Proofs of main results