A finite sample analysis of the double descent phenomenon for ridge function estimation

1 Introduction

The tremendous achievements of deep learning models in the solution of complex prediction tasks have been the focus of great attention in the applied Computer Science, Artificial Intelligence and Statistics communities in the recent years. Many success stories related to the use of Deep Neural Networks have even been reported in the media and no data scientist can ignore the Deep Learning tools available via opensource machine learning libraries such as Tensorflow, Keras, Pytorch and many others.

One of the key ingredient in their success is the huge number of parameters involved in all current architectures, a very counterintuitive approach that defies traditional statistical wisdom. Indeed, as intuition suggests, overparametrisation often results in interpolation, i.e. zero training error and the expected outcome of this approach should be very poor generalisation performance. However, the main suprise came from the observation that interpolating networks can still generalise well, as shown in the following table

[soltanolkotabi2019imaginginparis] reporting the error rate of various networks on the CIFAR 10 dataset, where

p

is the number of parameters, the training sample size is

n = 50.000

, the feature size is

d = 3072

and the number of classes is 10:

Model	parameters	Prain	error
CudaConvNet	145,578	2.9	$23 %$
Microlnception	1,649,402	33	$14 %$
ResNet	2,401,440	48	$13 %$

Belkin et al. [belkin2019reconciling]

recently addressed the problem of resolving this paradox, and brought some new light on the relationships between interpolation and generalization to unseen data. In the particular instance of kernel ridge regression,

[liang2018just] proved that interpolation can coexist with good generalization. In a subsequent line of work, recent connections between kernel regression and wide neural networks extensively were studied by [jacot2018neural], [du2018gradient], [allen2018convergence], [belkin2018understand] and provide additional motivation for a deeper understanding of the double descent phenomenon for kernel methods. Further motivations provided by Chizat and Bach [chizat2018note] about any nonlinear model of the form

E [Y ∣ X] = f (X; θ)

(1)

with parameter $θ \in R^{p}$ . As elegantly summarised in the introduction of [mei2019generalization], if we assume that $p$ is so large that training by gradient flow moves each of them by just a small amount with respect to some random initialization $θ_{0} \in R^{p}$ , linearising the model around $θ_{0}$ gives

E (Y ∣ X) \approx f (X; θ_{0}) + \nabla_{θ} f {(X; θ_{0})}_{0}^{t} β,

which leads, in the Empirical Risk Minimisation setting, to consider a simpler linear regression problem, with high dimensional random features

\nabla_{θ} f (X_{i}; θ_{0}), i = 1, \dots, n

, which owe their randomness to the randomness of the initialisation

θ_{0}

. This approximation is now well known to be missing the main features of deep neural networks [chizat2019lazy] but it is still a good test bench for new methods of analysing the double descent phenomenon.

In this paper, we consider a statistical model of the form

E [Y_{i} ∣ X_{i}]

= f (X_{i}^{t} θ^{*}), i = 1, \dots, n,

(2)

where $θ^{*} \in R^{p}$ and the function $f$ is assumed increasing and thrice differentiable, with bounded derivatives up to third order. The data $X_{1}, \dots, X_{n}$ will be assumed isotropic and subGaussian, and the observation errors will be assumed subGaussian as well. When the estimation of $θ^{*}$ is performed using Empirical Risk Estimation, i.e. by solving

^θ

= {a r g m i n}_{θ \in Θ} \frac{1}{n} n \sum i = 1 ℓ (Y_{i} - f (X_{i}^{t} θ))

(3)

for a given smooth loss function

ℓ

, we show that the double descent phenomenon takes place and we give precise order of dependencies with respect to all the intrinsic parameters of the model, such as the dimensions

n

and

p

, various bounds on the derivatives of

f

and of the loss function used in the Empirical Risk Estimation.

Our contribution is the first non-asymptotic analysis of the double descent phenomenon for non-linear models. Our results precisely characterise the proximity in

ℓ_{2}

of a certain solution

^θ

of (3) to

θ^{*}

, from which the performace of the minimum norm solution follows naturally. Our proofs are very elementary as they utilise an elegant continuous Newton method argument initially promoted by Neuberger in a series of papers intended to provide a new proof of the Nash Moser theorem [castro2001inverse], [neuberger2007continuous].

2 Main Results

In this section, we describe our mathematical model and set the notations.

2.1 Mathematical presentation of the problem

We assume that (2) holds and that $f$ satisfies the following properties

$f$ is increasing,
the first (resp. second, third) derivative is uniformly bounded by a positive constant $C_{f^{'}}$ (resp. $C_{f^{''}}$ and $C_{f^{(3)}}$ ), and
for all

$w \in [E [X_{i}]^{t} θ^{*} - μ \sqrt{log (n)}, E [X_{i}]^{t} θ^{*} + μ \sqrt{log (n)}],$ (4)

we have

$f^{'} (w) \geq c_{f^{'}}$ (5)

for some positive constant $c_{f^{'}}$ .

Concerning the statistical data, we will assume that

the random vectors
$X_{1}, \dots, X_{n}$ are independent subGaussian vectors in $R^{p}$ , to be subGaussian, with $ϕ_{2}$ -norm upper bounded by $K_{X}$ , and are such that the matrix

$X^{t}$ $= [\begin{matrix} X_{1}, \dots, X_{n} \end{matrix}]$ (6)

is full rank with probability one.
for all $i = 1, \dots, n$ , the random vectors $X_{i}$ are assumed
- to have a second moment matrix equal to the identity
  ¹¹1i.e. $ρ_{X}^{- 1 / 2} X_{i}$ , $i = 1, \dots, n$ are isotropic up to a scaling factor $ρ_{X}$ , i.e. $E [X_{i} X_{i}^{t}] = ρ_{X} I_{p}$ ,
- to have $ℓ_{2}$ -norm exactly equal to²²2notice that this is different from the usual regression model, where the columns are assumed to be normalised $\sqrt{ρ_{X} p}$
the errors $ϵ_{i} = Y_{i} - E [Y_{i}]$
are independent subGaussian centered random variables with
$ψ_{2}$ -norm upper bounded by $K_{ϵ}$ .

The performance of the estimators are often measured by the theoretical risk $R : Θ \mapsto R$ by

R (θ) = E [ℓ (Y - f (X^{t} θ))] .

Here, we will assume that the loss function $ℓ : R \mapsto R$ satisfies

$ℓ$ is a fourth differentiable convex loss function,
$ℓ^{'} (0) = 0$ ,
the derivative $ℓ^{'}$ is $C_{ℓ^{''}}$ -lipschitz function, which implies that $ℓ^{''}$ is bounded by $C_{ℓ^{''}}$ ,
the third and fourth derivatives are also assumed bounded with $C_{ℓ^{(3)}}$ the bound for $ℓ^{(3)}$ and $C_{ℓ^{(4)}}$ for $ℓ^{(4)}$ ,
$ℓ^{''}$ is lower bounded by a constant $c_{ℓ^{''}} > 0$ .

In order to estimate $θ^{*}$ , the Empirical Risk Minimizer $^θ$ is defined as a solution to the following optimisation problem

^θ

\in {a r g m i n}_{θ \in Θ} {^R}_{n} (θ)

(7)

with

{^R}_{n} (θ)

= {a r g m i n}_{θ \in Θ} \frac{1}{n} n \sum i = 1 ℓ (Y_{i} - f (X_{i}^{t} θ)) .

(8)

2.2 Statement of our main theorems

Our main result is the following. Their proofs are given in Section B and Section C in the appendix.

Theorem 2.1.

(Underparametrised setting) Let $μ > 0$ , $ν > 0$ and let $β \in (0, 1)$ . Let

	$r$	$= \frac{K_{ϵ}}{ρ_{X}} \frac{12 C^{'} \sqrt{C} C_{ℓ^{''}} C_{f^{'}}}{c_{ℓ^{''}} c_{f^{'}}^{2} (1 - β)} \sqrt{\frac{p}{n}}$
		$+ \frac{K_{ϵ}^{2}}{ρ_{X}^{2}} \frac{2 (2 κ_{1} + C_{f^{'''}} ν \sqrt{log (n)})}{c_{ℓ^{''}}^{3} c_{f^{'}}^{6} (1 - υ)^{3 / 2}} \frac{(12 C^{'} \sqrt{C} C_{ℓ^{''}} C_{f^{'}})^{2}}{(1 - β)^{2}} \frac{p^{3 / 2}}{n}$

and

	$Δ_{p, n}$	$= (2 κ_{1} + C_{f^{'''}} ν \sqrt{log (n)}) \sqrt{} p r +$
		$\frac{1}{2} (6 κ_{2} + C_{f^{(4)}} ν \sqrt{log (n)}) p r^{2} .$		(9)

where

κ_{1} = max (3 C_{ℓ^{''}} C_{f^{'}} C_{f^{''}}, C_{ℓ^{'''}} C_{f^{'}}^{3})

(10)

and

	$κ_{2}$	$= max {3 C_{ℓ^{'''}} C_{f^{'}}^{2} C_{f^{''}}, 3 C_{ℓ^{''}} C_{f^{''}}^{2}, 3 C_{ℓ^{''}} C_{f^{'}} C_{f^{'''}},$
		$C_{ℓ^{(4)}} C_{f^{'}}^{4}, 3 C_{ℓ^{'''}} C_{f^{'}}^{2} C_{f^{''}}, C_{ℓ^{''}} C_{f^{'}} C_{f^{'''}}} .$		(11)

Assume that $p$ and $n$ are such that

(α + C_{K_{X}})^{2} p < β n,

(12)

and

Δ_{p, n}

\leq c_{ℓ^{''}} c_{f^{'}}^{2} .

(13)

Then, with probability at least

$1$	$- (2 exp (- c_{K_{X}} α^{2} n) + 2 n exp (- \frac{μ^{2} log (n)}{∥ θ^{*} ∥_{2}^{2} K_{X}^{2}})$
	$+ exp (- \frac{p}{2}) + 2 n (exp (- \frac{ν^{2} log (n)}{C_{ℓ^{''}}^{2} K_{ϵ}^{2}})$
	$+ exp (- \frac{(c_{ℓ^{''}} c_{f^{'}}^{2} - Δ_{p, n})^{2}}{4 C_{f^{''}}^{2} C_{ℓ^{''}}^{2} K_{ϵ}^{2}}))),$	(14)

the unique solution $^θ$ to the optimisation problem (7) satisfies

∥^θ - θ^{*} ∥_{2}

\leq r .

Theorem 2.2.

(Overparametrised setting) Let $μ > 0$ , $ν > 0$ and let $β \in (0, 1)$ . Let

r

= \frac{K_{ϵ}}{ρ_{X}} \frac{12 C^{'} \sqrt{C} C_{ℓ^{''}} C_{f^{'}}}{c_{ℓ^{''}} c_{f^{'}}^{2} (1 - β)} \sqrt{\frac{n}{p}}

(15)

+ \frac{K_{ϵ}^{2}}{ρ_{X}^{2}} \frac{2 (2 κ_{1} + C_{f^{'''}} ν \sqrt{log (n)})}{c_{ℓ^{''}}^{3} c_{f^{'}}^{6} (1 - υ)^{3 / 2}} \frac{(12 C^{'} \sqrt{C} C_{ℓ^{''}} C_{f^{'}})^{2}}{(1 - β)^{2}} \frac{n^{2}}{\sqrt{p}}

and $Δ_{p, n}$ and $κ_{1}$ and $κ_{2}$ defined by (13), (10) and (11), respectively. Assume that $p$ and $n$ are such that

(α + C_{K_{X}})^{2} n < β p,

(16)

and

Δ_{p, n}

\leq c_{ℓ^{''}} c_{f^{'}}^{2} .

(17)

Then, there exists a first order stationary point $^θ$ to the optimisation problem (7) such that, with probability larger than or equal to

	$1$	$- (2 exp (- c_{K_{X}} α^{2} n) + 2 n exp (- \frac{μ^{2} log (n)}{∥ θ^{*} ∥_{2}^{2} K_{X}^{2}})$
		$+ exp (- \frac{n}{2})$
		$+ 2 n (exp (- \frac{ν^{2} log (n)}{C_{ℓ^{''}}^{2} K_{ϵ}^{2}}) + exp (- \frac{(c_{ℓ^{''}} c_{f^{'}}^{2} - Δ_{r})^{2}}{4 C_{f^{''}}^{2} C_{ℓ^{''}}^{2} K_{ϵ}^{2}}))),$

we have

∥^θ - θ^{*} ∥_{2}^{2}

\leq r .

(18)

Using these two theorems, we now establish the following risk bound.

Corollary 2.3.

Let the assumptions of Theorem 2.2 hold and let $r$ be defined by (15). Then, there exists a minimum norm first order stationary point ${^θ}^{♯}$ of problem (7) which satisfies

∥ X {^θ}^{♯} - X θ^{*} ∥_{2}

\leq (1 + 2 C \sqrt{\frac{p}{n}}) r

(19)

with probability larger than or equal to

	$1$	$- (2 exp (- c_{K_{X}} α^{2} n) + 2 n exp (- \frac{μ^{2} log (n)}{∥ θ^{*} ∥_{2}^{2} K_{X}^{2}})$
		$+ exp (- \frac{n}{2})$
		$+ 2 n (exp (- \frac{ν^{2} log (n)}{C_{ℓ^{''}}^{2} K_{ϵ}^{2}}) + exp (- \frac{(c_{ℓ^{''}} c_{f^{'}}^{2} - Δ_{r})^{2}}{4 C_{f^{''}}^{2} C_{ℓ^{''}}^{2} K_{ϵ}^{2}})) + 2 exp (- c p)) .$

Proof.

Let ${^θ}^{♯}$ denote the minimum norm solution in the variable $θ$ to the system

X θ

= X^θ,

(20)

where $^θ$ is the first order stationary point of problem (7) of Theorem 2.2. Then, ${^θ}^{♯}$ is a minimum norm first order stationary point of problem (7). Moreover

	$X {^θ}^{♯} - X θ^{*}$	$= X^θ - X θ^{*}$
		$= X (^θ - θ^{*}) .$

Therefore, we obtain

\frac{1}{\sqrt{n}} ∥ X {^θ}^{♯} - X θ^{*} ∥_{2}

\leq \frac{1}{\sqrt{n}} ∥ X ∥ ∥^θ - θ^{*} ∥_{2} .

(21)

Using [vershynin2010introduction, Theorem 5.39], we get that with probability at least $1 - 2 exp (- c p)$ ,

∥ X ∥

\leq \sqrt{n} + 2 C \sqrt{p}

(22)

for some absolute constants $c$ and $C$ depending of $K_{X}$ , and combining this bound with the result of Theorem 2.2, the announced result follows. ∎

2.3 The case of linear regression

In the linear case where $f (X_{i}^{t} θ) = X_{i}^{t} θ$ and the loss is quadratic $ℓ (z) = \frac{1}{2} z^{2}$ , the optimisation problem (7) is

^θ = {a r g m i n}_{θ \in R^{p}} \frac{1}{n} n \sum i = 1 \frac{1}{2} (Y_{i} - X_{i}^{t} θ)^{2} .

we have, $C_{f^{'}} = 1$ and $C_{f^{(k)}} = 0$ for $k \geq 2$ .

The quadratic loss function is used with

ℓ (z) = \frac{1}{2} z^{2}, ℓ^{'} (z) = z, ℓ^{''} (z) = 1

and $C_{ℓ^{(k)}} = 0$ , $k \geq 3$ . We therefore have

Corollary 2.4.

In addition to the assumptions about our model, let us assume that $f (X_{i}^{t} θ) = X_{i}^{t} θ$ and that $ℓ (z) = \frac{1}{2} z^{2}$ . Let $μ > 0$ , $ν > 0$ and let $β \in (0, 1)$ . Let

r

= \frac{K_{ϵ}}{ρ_{X}} \frac{12 C^{'} \sqrt{C}}{(1 - β)} \sqrt{\frac{p}{n}}

Assume that $p$ and $n$ are such that

(α + C_{K_{X}})^{2} p < β n,

(23)

Then, with probability at least

1

- 2 exp (- c_{K_{X}} α^{2} n) - 2 n exp (- \frac{μ^{2} log (n)}{∥ θ^{*} ∥_{2}^{2} K_{X}^{2}}) - e^{- \frac{p}{2}},

the unique solution $^θ$ to the optimisation problem (7) satisfies

∥^θ - θ^{*} ∥_{2}

\leq r .

Corollary 2.5.

In addition to the assumptions about our model, let us assume that $f (X_{i}^{t} θ) = X_{i}^{t} θ$ and that $ℓ (z) = \frac{1}{2} z^{2}$ . Let $μ > 0$ , $ν > 0$ and let $β \in (0, 1)$ . Let

r

= \frac{K_{ϵ}}{ρ_{X}} \frac{12 C^{'} \sqrt{C}}{(1 - β)} \sqrt{\frac{n}{p}}

Assume that $p$ and $n$ are such that

(α + C_{K_{X}})^{2} n < β p .

(24)

Then, there exists a solution $^θ$ to the optimisation problem (7) such that, with probability larger than or equal to

1

- 2 exp (- c_{K_{X}} α^{2} n) - 2 n exp (- \frac{μ^{2} log (n)}{∥ θ^{*} ∥_{2}^{2} K_{X}^{2}}) - e^{- \frac{n}{2}},

we have

∥^θ - θ^{*} ∥_{2}^{2}

\leq r .

(25)

Corollary 2.6.

In addition to the assumptions about our model, let us assume that $f (X_{i}^{t} θ) = X_{i}^{t} θ$ and that $ℓ (z) = \frac{1}{2} z^{2}$ . Let the other assumptions of Corollary 2.5. Then, the minimum norm solution ${^θ}^{♯}$ of problem (7) satisfies

\frac{1}{\sqrt{n}} ∥ X {^θ}^{♯} - X θ^{*} ∥_{2}

\leq (1 + 2 C \sqrt{\frac{p}{n}}) \frac{K_{ϵ}}{ρ_{X}} \frac{12 C^{'} \sqrt{C}}{(1 - β)} \sqrt{\frac{n}{p}}

(26)

with probability at least

1

- 2 exp (- c_{K_{X}} α^{2} n) - 2 n exp (- \frac{μ^{2} log (n)}{∥ θ^{*} ∥_{2}^{2} K_{X}^{2}}) - e^{- \frac{n}{2}} - 2 exp (- c p)

for some constants $c$ and $C$ depending on $K_{X}$ only.

2.4 Discussion of the results and new implications for some classical models

Theorem 2.1 and Theorem 2.2 provide a new finite sample analysis of the problem of estimating ridge functions in both underparametrised and overparametrised regimes, i.e. where the number of parameters is smaller (resp. larger) than the sample size.

Our analysis of the underparametrised setting shows that:
- When $p \leq \sqrt{n / log (n)}$ , we can obtain an error of order less than or equal to $\sqrt{p / n}$ .
- When $p \geq \sqrt{n / log (n)}$ , our bound undergoes a transition to a worse order of $\sqrt{log (n)} p^{3 / 2} / n$ . Moreover, this error goes to $+ \infty$ at a logarithmic rate as $n$ grows to $+ \infty$ if $p$ becomes proportional to $n^{2 / 3}$ , and at a $\sqrt{n}$ rate if $p$ becomes proportional to $n$ .
- One condition for this to work is (13).
  - In the $p \leq \sqrt{n / log (n)}$ regime, $Δ_{p, n}$ will be of the order $\sqrt{log (n)} p / \sqrt{n}$ and we can choose $c_{l^{''}}$ to be of that same order so that (13) is satisfied. The last term in (14) will be of small order when $K_{ϵ} \leq 1 / \sqrt{log (n)}$ which can be assumed under the natural assumption that the matrix $X$ is rescaled using $ρ_{X} = K_{ϵ}$ .
  - In the $p \geq \sqrt{n / log (n)}$ regime, $Δ_{p, n}$ is of the order $n^{4} log (n)^{3 / 2}$ . We can choose to multiply $ℓ$ by this quantity in order to ensure that (13) is satisfied, a choose $K_{ϵ} \leq 1 / \sqrt{log (n)}$ and $ρ_{X} = K_{ϵ}$ as argued for the previous case.
In the overparametrised setting, we get the following results:
- The error bound is of order $n^{2} / \sqrt{p}$ and decreases as $p$ grows to $+ \infty$ . We therefore recover the "double descent phenomenon".

In the linear model setting, Corollary 2.4 and Corollary 2.5 give the following simpler results

In the underparametrised setting, the error bound is of order $\sqrt{p / n}$ , which is a simpler behavior than in the non-linear setting.
In the overparametrised setting, the error bound is of order $\sqrt{n / p}$ , which is a simpler behavior than in the non-linear setting, with a faster decay towards zero.

Concerning the prediction error, Corollary 2.3 and Corollary 2.6 provide corresponding prediction bounds. In the linear setting of Corollary 2.6, the obtained bound is decreasing as a function of $p$ and is of the order of the noise level. In the general case, the results of Corollary 2.3 depend of quantities that vanish in the linear case and that may be taken to be small as a function of $p$ and for $n$ fixed. We leave it for further study in relevant specific cases.

2.5 Comparison with previous results

Our results are based on a new zero finding approach inspired from [neuberger2007continuous] and we obtain precise quantitative results in the finite sample setting for linear and non-linear models. Following the initial discovery of the "double descent phenomenon" in [belkin2019reconciling], many authors have addressed the question of precisely characterising the error decay as a function of the number of parameters in the linear and non-linear setting (mostly based on random feature models). Some of the latest works [mei2019generalization] address the problem in the asymptotic regime. Recently, the finite sample analysis has been addressed in the very interesting works [bartlett2020benign] and [chinot2020benign] for the linear model only. The work of [bartlett2020benign] and [chinot2020benign] give very precise upper and lower bounds on the prediction risk for general covariate covariance matrices under the subGaussian assumption.

In the present work, we show that similar, very precise results can be obtained for non-linear models of the ridge function class, using elementary perturbation results and some (now) standard random matrix theory. Our results provide an explicit control of the distance between some empirical estimators and the ground truth in terms of the subGaussian norms of the error and the covariate vector in the case where the covariate vectors are assumed isotropic (more general results can easily be recovered by a simple change of variable). Our analysis is made very elementary by using Neuberger’s result

[neuberger2007continuous] and the subGaussian isotropic assumption which allows to leverage previous results of [vershynin2010introduction] about finite random matrix with subGaussian rows or columns, depending of the setting (underparametrised vs overparametrised).

3 Conclusion and perspectives

This work presents a precise quantitative, finite sample analysis of the double descent phenomenon in the estimation of linear and non-linear models. We make use of a zero-finding result of Neuberger [neuberger2007continuous] which can be applied to a large number of settings in machine learning.

Extending our work to the case of Deep Neural Networks is an exciting avenue for future research. We are currently working on the analysis of the double descent phenomenon in the case of Residual Neural Networks and we expect to post our new findings in a near future. Another possible direction is to include penalisation, which can be treated using the same techniques via Karush-Kuhn-Tucker conditions. This can be applied to Ridge Regression and $ℓ_{1}$ -penalised estimation and makes a promising avenue for future investigations. Weakening the assumptions on our data, which are here of subGaussian type, could also lead to interesting new results; this could be achieved by utilising, e.g. the work of [mendelson2017extending].

References

Appendix A Common framework: Chasing close-to-ideal solutions using Neuberger’s quantitative perturbation theorem

a.1 Neuberger’s theorem

The following theorem of Neuberger [neuberger2007continuous] will be instrumental in our study of the ERM. In our context, this theorem can be restated as follows.

Theorem A.1 (Neuberger’s theorem for ERM).

Suppose that $r > 0$ , that $θ^{*} \in R^{p}$ and that the Jacobian $D {^R}_{n} (\cdot)$ is a continuous map on $B_{r} (θ^{*})$ with the property that for each $θ$ in $b_{r} (θ^{*})$ there exists a vector $d$ in $B_{r} (0)$ such that,

lim t ↓ 0 \frac{D {^R}_{n} (θ + t d) - D {^R}_{n} (θ)}{t} = - D {^R}_{n} (θ^{*}) .

(27)

Then there exists $u$ in $B_{r} (θ^{*})$ such that $D {^R}_{n} (u) = 0$ .

a.2 Computing the second derivative

Since the loss is twice differentiable, the empirical risk ${^R}_{n}$ is itself twice differentiable. The Gradient of the empirical risk is given by

	$\nabla {^R}_{n} (θ)$	$= - \frac{1}{n} n \sum i = 1 ℓ^{'} (Y_{i} - f (X_{i}^{t} θ)) f^{'} (X_{i}^{t} θ) X_{i}$
		$= - \frac{1}{n} X^{t} D (ν) l^{'} (ϵ)$

where $ℓ^{'} (ϵ)$ is to be understood componentwise, and

ν_{i}

= f^{'} (X_{i}^{t} θ)

(28)

and the Hessian is given by

	$\nabla^{2} {^R}_{n} (θ) = \frac{1}{n} n \sum i = 1 (ℓ^{''} (Y_{i} - f (X_{i}^{t} θ)) f^{'} (X_{i}^{t} θ)^{2}$
	$- ℓ^{'} (Y_{i} - f (X_{i}^{t} θ)) f^{''} (X_{i}^{t} θ)) X_{i} X_{i}^{t} .$		(29)

The condition we have to satisfy in order to use Neuberger’s theorem, i.e. the version of (27) associated with our setting, is the following

\nabla^{2} {^R}_{n} (θ) d

= - \nabla {^R}_{n} (θ^{*}) .

(30)

The Hessian matrix can be rewritten as

\nabla^{2} {^R}_{n} (θ)

= \frac{1}{n} X^{t} D (μ) X

(31)

where $D_{Y, X}$ is a diagonal matrix given by

	$μ_{i}$	$= (ℓ^{''} (Y_{i} - f (X_{i}^{t} θ)) f^{'} (X_{i}^{t} θ)^{2}$
		$- ℓ^{'} (Y_{i} - f (X_{i}^{t} θ)) f^{''} (X_{i}^{t} θ))$

Notice that

	$μ_{i}$	$= ℓ^{''} (Y_{i} - f (X_{i}^{t} θ^{})) f^{'} (X_{i}^{t} θ^{})^{2}$
		$- ℓ^{'} (Y_{i} - f (X_{i}^{t} θ^{})) f^{''} (X_{i}^{t} θ^{}) + δ_{i}$

with

	$δ_{i}$	$= ℓ^{''} (Y_{i} - f (X_{i}^{t} θ)) f^{'} (X_{i}^{t} θ)^{2}$
		$- ℓ^{'} (Y_{i} - f (X_{i}^{t} θ)) f^{''} (X_{i}^{t} θ)$
		$- (ℓ^{''} (Y_{i} - f (X_{i}^{t} θ^{})) f^{'} (X_{i}^{t} θ^{})^{2}$
		$- ℓ^{'} (Y_{i} - f (X_{i}^{t} θ^{})) f^{''} (X_{i}^{t} θ^{})) .$

The proof of the main results Theorem 2.1 and 2.2 will be based on controlling these quantities and using these estimates for controlling particular solutions of (27). The proof of Theorem 2.1 is given in Section B and the proof of Theorem 2.2 is given in Section C.

Appendix B Proof of Theorem 2.1: The under-parametrised case

b.1 Four technical lemmæ

Lemma B.1.

With probability larger than or equal to $1 - 2 exp (- \frac{η^{2}}{∥ θ^{*} ∥_{2}^{2} K_{X}^{2}})$ , we have

f^{'} (X_{i}^{t} θ^{*})

\geq c_{f^{'}} .

(32)

Proof.

As the rows of $X$ are subGaussian, $X_{i}^{t} θ^{*}$

is subGaussian with variance proxy

K_{X} ∥ θ^{*} ∥_{2}

. Consequently, for all

t > 0

(33)

Set $t = η$ and use the assumption that $f^{'} (w) \geq c_{f^{'}}$ for all $w \in [E [X_{i}]^{t} θ^{*} - η, E [X_{i}]^{t} θ^{*} + η]$ . ∎

Lemma B.2.

For all $i = 1, \dots, n$ , the variable $ℓ^{'} (ϵ_{i})$ is subGaussian, with variance proxy $∥ ℓ^{'} (ϵ_{i}) ∥_{ψ_{2}} = C_{ℓ^{''}} ∥ ϵ_{i} ∥_{ψ_{2}}$ .

Proof.

Let us compute

∥ ℓ^{'} (ϵ_{i}) ∥_{ψ_{2}}

= sup γ \geq 1 γ^{- 1 / 2} (E | ℓ^{'} (ϵ_{i}) |^{γ})^{1 / γ}

Lipschitzianity of $ℓ^{'}$ implies that

| ℓ^{'} (ϵ_{i}) - ℓ^{'} (0) |

\leq C_{ℓ^{''}} | ϵ_{i} - 0 |

and since $ℓ^{'} (0) = 0$ , we get

| ℓ^{'} (ϵ_{i}) |

\leq C_{ℓ^{''}} | ϵ_{i} |

which implies that

∥ ℓ^{'} (ϵ_{i}) ∥_{ψ_{2}}

= C_{ℓ^{''}} sup γ \geq 1 γ^{- 1 / 2} (E | ϵ_{i} |^{γ})^{1 / γ}

and thus,

∥ ℓ^{'} (ϵ_{i}) ∥_{ψ_{2}}

= C_{ℓ^{''}} ∥ ϵ_{i} ∥_{ψ_{2}} < + \infty

as announced. ∎

Lemma B.3.

Define

	$δ_{i} (θ)$	$= μ_{i} (X_{i}^{t} θ) - (ℓ^{''} (Y_{i} - f (X_{i}^{t} θ^{})) f^{'} (X_{i}^{t} θ^{})^{2}$
		$- ℓ^{'} (Y_{i} - f (X_{i}^{t} θ^{})) f^{''} (X_{i}^{t} θ^{})) .$

Then, with probability at least $1 - 2 \sum_{i = 1}^{n} exp (- \frac{t^{2}}{C_{ℓ^{''}}^{2} ∥ ϵ_{i} ∥_{ψ_{2}}^{2}})$ , conditional on $X_{1}, \dots, X_{n}$ , we have

n max i = 1 | δ_{i} (θ) |

\leq Δ_{r} (t)

(34)

with

Δ_{r} (t) = (2 κ_{1} + C_{f^{'''}} t) \sqrt{p} r + \frac{1}{2} (6 κ_{2} + C_{f^{(4)}} t) p r^{2} .

(35)

for all $θ$ such that

∥ θ - θ^{*} ∥_{2} \leq r,

(36)

with

κ_{1} = max (3 C_{ℓ^{''}} C_{f^{'}} C_{f^{''}}, C_{ℓ^{'''}} C_{f^{'}}^{3})

(37)

and

	$κ_{2}$	$= max {3 C_{ℓ^{'''}} C_{f^{'}}^{2} C_{f^{''}}, 3 C_{ℓ^{''}} C_{f^{''}}^{2}, 3 C_{ℓ^{''}} C_{f^{'}} C_{f^{'''}},$
		$C_{ℓ^{(4)}} C_{f^{'}}^{4}, 3 C_{ℓ^{'''}} C_{f^{'}}^{2} C_{f^{''}}, C_{ℓ^{''}} C_{f^{'}} C_{f^{'''}}} .$

Proof.

Recall that

	$μ_{i} (X_{i}^{t} θ)$	$= ℓ^{''} (Y_{i} - f (X_{i}^{t} θ)) f^{'} (X_{i}^{t} θ)^{2}$
		$- ℓ^{'} (Y_{i} - f (X_{i}^{t} θ)) f^{''} (X_{i}^{t} θ)$

By Taylor’s formula

	$μ_{i} (X_{i}^{t} θ)$	$= μ_{i} (X_{i}^{t} θ^{}) + \frac{d μ_{i}}{d (X_{i}^{t} θ)} (X_{i}^{t} θ^{}) (X_{i}^{t} θ - X_{i}^{t} θ^{*})$
		$+ \frac{1}{2} \frac{d^{2} μ_{i}}{d (X_{i}^{t} θ)^{2}} (X_{i}^{t} θ^{} + c_{i} (X_{i}^{t} θ - X_{i}^{t} θ^{})) (X_{i}^{t} θ - X_{i}^{t} θ^{*})^{2}$

where $c_{i}$ is a number between $0$ and $1$ . Then

	$\| δ_{i} (θ) \|$	$= ∣ ∣ \frac{d μ_{i}}{d (X_{i}^{t} θ)} (X_{i}^{t} θ^{}) (X_{i}^{t} θ - X_{i}^{t} θ^{})$
		$+ \frac{1}{2} \frac{d^{2} μ_{i}}{d (X_{i}^{t} θ)^{2}} (X_{i}^{t} θ^{} + c_{i} (X_{i}^{t} θ - X_{i}^{t} θ^{})) (X_{i}^{t} θ - X_{i}^{t} θ^{*})^{2} ∣ ∣$
		$\leq ∣ ∣ \frac{d μ_{i}}{d (X_{i}^{t} θ)} (X_{i}^{t} θ^{}) ∣ ∣ ∣ ∣ X_{i}^{t} θ - X_{i}^{t} θ^{} ∣ ∣$
		$+ \frac{1}{2} (X_{i}^{t} θ - X_{i}^{t} θ^{})^{2} ∣ ∣ \frac{d^{2} μ_{i}}{d (X_{i}^{t} θ)^{2}} (X_{i}^{t} θ^{} + c_{i} (X_{i}^{t} θ - X_{i}^{t} θ^{*})) ∣ ∣$

We have to control the quantities

∣ ∣ \frac{d μ_{i}}{d (X_{i}^{t} θ)} (X_{i}^{t} θ

	$\| δ_{i} (θ) \|$	$= ∣ ∣ \frac{d μ_{i}}{d (X_{i}^{t} θ)} (X_{i}^{t} θ^{}) (X_{i}^{t} θ - X_{i}^{t} θ^{})$
		$+ \frac{1}{2} \frac{d^{2} μ_{i}}{d (X_{i}^{t} θ)^{2}} (X_{i}^{t} θ^{} + c_{i} (X_{i}^{t} θ - X_{i}^{t} θ^{})) (X_{i}^{t} θ - X_{i}^{t} θ^{*})^{2} ∣ ∣$
		$\leq ∣ ∣ \frac{d μ_{i}}{d (X_{i}^{t} θ)} (X_{i}^{t} θ^{}) ∣ ∣ ∣ ∣ X_{i}^{t} θ - X_{i}^{t} θ^{} ∣ ∣$
		$+ \frac{1}{2} (X_{i}^{t} θ - X_{i}^{t} θ^{})^{2} ∣ ∣ \frac{d^{2} μ_{i}}{d (X_{i}^{t} θ)^{2}} (X_{i}^{t} θ^{} + c_{i} (X_{i}^{t} θ - X_{i}^{t} θ^{*})) ∣ ∣$

A finite sample analysis of the double descent phenomenon for ridge function estimation

Asymptotics of Ridge Regression in Convolutional Models

Multiple Descent in the Multiple Random Feature Model

Exact expressions for double descent and implicit regularization via surrogate random design

Minimum ℓ_1-norm interpolators: Precise asymptotics and multiple descent

Sharp Asymptotics of Kernel Ridge Regression Beyond the Linear Regime

Triple descent and the two kinds of overfitting: Where why do they appear?

Interpolation and Learning with Scale Dependent Kernels

1 Introduction

2 Main Results

2.1 Mathematical presentation of the problem

2.2 Statement of our main theorems

Theorem 2.1.

Theorem 2.2.

Corollary 2.3.

Proof.

2.3 The case of linear regression

Corollary 2.4.

Corollary 2.5.

Corollary 2.6.

2.4 Discussion of the results and new implications for some classical models

2.5 Comparison with previous results

3 Conclusion and perspectives

References

Appendix A Common framework: Chasing close-to-ideal solutions using Neuberger’s quantitative perturbation theorem

a.1 Neuberger’s theorem

Theorem A.1 (Neuberger’s theorem for ERM).

a.2 Computing the second derivative

Appendix B Proof of Theorem 2.1: The under-parametrised case

b.1 Four technical lemmæ

Lemma B.1.

Proof.

Lemma B.2.

Proof.

Lemma B.3.

Proof.

A finite sample analysis of the double descent phenomenon for ridge function estimation

Related Research

Asymptotics of Ridge Regression in Convolutional Models

Multiple Descent in the Multiple Random Feature Model

Exact expressions for double descent and implicit regularization via surrogate random design

Minimum ℓ_1-norm interpolators: Precise asymptotics and multiple descent

Sharp Asymptotics of Kernel Ridge Regression Beyond the Linear Regime

Triple descent and the two kinds of overfitting: Where why do they appear?

Interpolation and Learning with Scale Dependent Kernels

1 Introduction

2 Main Results

2.1 Mathematical presentation of the problem

2.2 Statement of our main theorems

Theorem 2.1.

Theorem 2.2.

Corollary 2.3.

Proof.

2.3 The case of linear regression

Corollary 2.4.

Corollary 2.5.

Corollary 2.6.

2.4 Discussion of the results and new implications for some classical models

2.5 Comparison with previous results

3 Conclusion and perspectives

References

Appendix A Common framework: Chasing close-to-ideal solutions using Neuberger’s quantitative perturbation theorem

a.1 Neuberger’s theorem

Theorem A.1 (Neuberger’s theorem for ERM).

a.2 Computing the second derivative

Appendix B Proof of Theorem 2.1: The under-parametrised case

b.1 Four technical lemmæ

Lemma B.1.

Proof.

Lemma B.2.

Proof.

Lemma B.3.

Proof.