# Dimension-free PAC-Bayesian bounds for matrices, vectors, and linear least squares regression

This paper is focused on dimension-free PAC-Bayesian bounds, under weak polynomial moment assumptions, allowing for heavy tailed sample distributions. It covers the estimation of the mean of a vector or a matrix, with applications to least squares linear regression. Special efforts are devoted to the estimation of Gram matrices, due to their prominent role in high-dimension data analysis.

Comments

There are no comments yet.

## Authors

• 6 publications
• 2 publications
• ### Dimension-free PAC-Bayesian bounds for the estimation of the mean of a random vector

In this paper, we present a new estimator of the mean of a random vector...
02/12/2018 ∙ by Olivier Catoni, et al. ∙ 0

read it

• ### Simpler PAC-Bayesian Bounds for Hostile Data

PAC-Bayesian learning bounds are of the utmost interest to the learning ...
10/23/2016 ∙ by Pierre Alquier, et al. ∙ 0

read it

• ### ℓ_1-regression with Heavy-tailed Distributions

In this paper, we consider the problem of linear regression with heavy-t...
05/02/2018 ∙ by Lijun Zhang, et al. ∙ 0

read it

• ### Distribution-Free Robust Linear Regression

We study random design linear regression with no assumptions on the dist...
02/25/2021 ∙ by Jaouad Mourtada, et al. ∙ 0

read it

• ### Dimension-free Bounds for Sums of Independent Matrices and Simple Tensors via the Variational Principle

We consider the deviation inequalities for the sums of independent d by ...
08/18/2021 ∙ by Nikita Zhivotovskiy, et al. ∙ 0

read it

• ### Berry-Esseen Bounds for Projection Parameters and Partial Correlations with Increasing Dimension

The linear regression model can be used even when the true regression fu...
07/19/2020 ∙ by Arun Kumar Kuchibhotla, et al. ∙ 0

read it

• ### Suboptimality of Constrained Least Squares and Improvements via Non-Linear Predictors

We study the problem of predicting as well as the best linear predictor ...
09/19/2020 ∙ by Tomas Vaškevičius, et al. ∙ 0

read it

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

The subject of this paper is to discuss dimension-free PAC-Bayesian bounds for matrices and vectors. It comes after Catoni (2016) and Giulini (2017a)

, the first paper discussing dimension dependent bounds and the second one dimension-free bounds, under a kurtosis like assumption about the data distribution. Here, in contrast, we envision even weaker assumptions, and focus on dimension-free bounds only.

Our main objective is the estimation of the mean of a random vector and of a random matrix. Finding sub-Gaussian estimators for the mean of a non necessarily sub-Gaussian random vector has been the subject of much research in the last few years, with important contributions from Joly, Lugosi and Oliveira (2017), Lugosi and Mendelson (2017) and Minsker (2015). While in Joly, Lugosi and Oliveira (2017) the statistical error bound still has a residual dependence on the dimension of the ambient space, in Lugosi and Mendelson (2017) this dependence is removed, for an estimator of the median of means type. However, this estimator is not easy to compute and the bound contains large constants. We propose here another type of estimator, that can be seen as a multidimensional extension of Catoni (2012). It provides a nonasymptotic confidence region with the same diameter (including the values of the constants) as the Gaussian concentration inequality stated in equation (1.1) of Lugosi and Mendelson (2017)

, although in our case, the confidence region is not necessarily a ball, but still a convex set. The Gaussian bound concerns the estimation of the expectation of a Gaussian random vector by the mean of an i.i.d. sample, whereas in our case, we only assume that the variance is finite, a much weaker hypothesis.

In Minsker (2016) the question of estimating the mean of a random matrix is addressed. The author uses exponential matrix inequalities in order to extend Catoni (2012) to matrices and to control the operator norm of the error. In the bounds at confidence level , the complexity term is multiplied by . Here, we extend Catoni (2012) using PAC-Bayesian bounds to measure complexity, and define an estimator with a bound where the term is multiplied by some directional variance term only, and not the complexity factor, that is larger.

After recalling in Section 2 the PAC-Bayesian inequality that will be at the heart of many of our proofs, we deal successively with the estimation of a random vector (Section 3) and of a random matrix (Section 4). Section 6 is devoted to the estimation of the Gram matrix, due to its prominent role in multidimensional data analysis. In Section 7 we introduce some applications to least squares regression.

## 2 Some well known PAC-Bayesian inequality

This is a preliminary section, where we state the PAC-Bayesian inequality that we will use throughout this paper to obtain deviation inequalities holding uniformly with respect to some parameter.

Consider a random variable

and a measurable parameter space . Let

be a probability measure on

and a bounded measurable function. For any other probability measure on , define the Kullback divergence function as usual by the formula

 K(ρ,μ)=⎧⎪⎨⎪⎩∫log(dρdμ)dρ,ρ≪μ,+∞, otherwise.

Let be independent copies of .

###### Proposition 2.1.

For any , with probability at least , for any probability measure ,

 1nn∑i=1∫f(θ,Xi)dρ(θ)≤∫log[E(exp(f(θ,X)))]dρ(θ)+K(ρ,μ)+log(δ−1)n.
###### Proof.

It is a consequence of equation (5.2.1) page 159 of Catoni (2004). Indeed, let us recall the identity

 log(∫exp(h(θ))dμ(θ))=supρ{∫h(θ)dρ(θ)−K(ρ,μ)},

where may be any bounded measurable function (extensions to unbounded are possible but will not be required in this paper), and where the supremum in is taken on all probability measures on the measurable parameter space . The proof may be found in (Catoni, 2004, page 159). Combined with Fubini’s lemma, it yields

Since implies that

 P(W≥log(δ−1))=E(1[δexp(W)≥1])≤E(δexp(W))≤δ,

we obtain the desired result, considering

 W=supρ[∫(n∑i=1f(θ,Xi)−nlog[E(exp(f(θ,X)))])dρ(θ)−K(ρ,μ)].

## 3 Estimation of the mean of a random vector

Let be a random vector and let be independent copies of . In this section, we will estimate the mean and obtain dimension-free non-asymptotic bounds for the estimation error.

Let be the unit sphere of and let

be the identity matrix of size

. Let

be the normal distribution centered at

, whose covariance matrix is , where is a positive real parameter.

Instead of estimating directly the mean vector , our strategy will be rather to estimate its component in each direction of the unit sphere. For this, we introduce the estimator of defined as

 E(θ)=1nλn∑i=1∫ψ(λ⟨θ′,Xi⟩)dρθ(θ′),θ∈Sd, λ>0,

where is the symmetric influence function

 ψ(t)=⎧⎪⎨⎪⎩t−t3/6,−√2≤t≤√22√2/3,t>√2−2√2/3,t<−√2 (1)

and where the positive constants and will be chosen afterward.

As stated in the following lemma, we chose this influence function because it is close to the identity in a neighborhood of zero and is such that is bounded by polynomial functions.

###### Lemma 3.1.

For any ,

 −log(1−t+t2/2)≤ψ(t)≤log(1+t+t2/2).
###### Proof.

Put . Remark that for and that for . As and

 [f′(t)−ψ′(t)](1+t+t2/2)=t3(2−t)4, ψ′(t)≤f′(t),0≤t≤√2,ψ′(t)≥f′(t),−√2≤t≤0,

proving that

 ψ(t)≤f(t),−√2≤t≤√2.

Since is increasing on and decreasing on , while is constant on these two intervals, the above inequality can be extended to all . From the symmetry , we deduce the converse inequality

 −f(−t)≤ψ(t),t∈R

that ends the proof. ∎

Since follows a normal distribution with mean

, and since the influence function is piecewise polynomial, the estimator can be computed explicitly in terms of the standard normal distribution function. This is done in the following lemma.

###### Lemma 3.2.

Let be a standard Gaussian real valued random variable. For any and any , define

 φ(m,σ)=E[ψ(m+σW)].

The function can be computed as

 φ(m,σ)=m(1−σ2/2)−m3/6+r(m,σ),

where, introducing , , the correction term is

Remark that the correction term is small when is small and is small, since

###### Proof.

The proof of this lemma is a simple computation, based on the expression

 ψ(t)=(t−t36)[1(t≤√2)−1(t≤−√2)]+2√23[1−1(t≤√2)−1(t≤−√2)],t∈R,

on the identities

 E[1(W≤a)] =F(a), E[1(W≤a)W] =−1√2πexp(−a22), E[1(W≤a)W2] =F(a)−a√2πexp(−a22), E[1(W≤a)W3] =−(a2+2)√2πexp(−a22),

and on the fact that . ∎

Accordingly, the estimator can be computed as

### 3.1 Estimation without centering

###### Proposition 3.3.

Assume that

 E(∥X∥2)=Tr[E(XX⊤)]≤T<∞ and supθ∈SE(⟨θ,X⟩2)≤v≤T<∞,

where and are two known constants and where is an arbitrary symmetric subset of the unit sphere, meaning that if then . Choose any confidence parameter and set the constants and used in the definition of the estimator to

 λ =√2log(δ−1)nv, β =√nTλ=√2Tlog(δ−1)v.

Non asymptotic confidence region: With probability at least ,

 supθ∈S∣∣E(θ)−⟨θ,E(X)⟩∣∣≤√Tn+√2vlog(δ−1)n.

Consider an estimator of satisfying

 supθ∈S∣∣E(θ)−⟨θ,ˆm⟩∣∣≤√Tn+√2vlog(δ−1)n.

With probability at least , such a vector exists and

 supθ∈S∣∣⟨θ,ˆm−E(X)⟩|≤supθ∈S∣∣E(θ)−⟨θ,ˆm⟩∣∣+√Tn+√2vlog(δ−1)n≤2√Tn+2√2vlog(δ−1)n.
###### Remark 3.1.

In particular in the case when is the whole unit sphere, we obtain with probability at least the bound

By choosing as the middle of a diameter of the confidence region, we could do a little better and replace the factor in this bound by a factor .

###### Proof.

According to the PAC-Bayesian inequality of Proposition 2.1, with probability at least , for any ,

 E(θ)≤1λ∫log[E(expψ(λ⟨θ′,X⟩))]dρθ(θ′)+K(ρθ,ρ0)+log(δ−1)nλ.

We can then use the polynomial approximation of given by Lemma 3.1, remarking that and that , to deduce that

We conclude by considering both and to get the reverse inequality, using the assumption that is symmetric and remarking that .
The existence with probability of satisfying the required inequality is granted by the fact that on the event defined by the above PAC-Bayesian inequality, the expectation belongs to the confidence region that, as a result, cannot be empty. ∎

### 3.2 Centered estimate

The bounds in the previous section are simple, but they are stated in terms of uncentered moments of order two where we would have expected a variance. In this section, we explain how to deduce centered bounds from the uncentered bounds of the previous section, through the use of a sample splitting scheme.
Assume that

 E(∥X−E(X)∥2)≤¯¯¯¯T<∞, and supθ∈SdE(⟨θ,X−E(X)⟩2)≤¯¯¯v≤¯¯¯¯T<∞,

where and are known constants. Remark that when these bounds hold, the bounds

 v=¯¯¯v+∥E(X)∥2 and T=¯¯¯¯T+∥E(X)∥2 (2)

hold in the previous section. Assume that we know also some bound such that

 ∥∥E(X)∥∥2≤b.

Split the sample in two parts and . Use the first part to construct an estimator of as described in Proposition 3.3, choosing . According to this proposition and by equation (2), with probability at least ,

 ∥˜m−E(X)∥≤2√¯¯¯¯T+bk+2√2(¯¯¯v+b)log(δ−1)k=√Ak,

where we have put .
We then construct an estimator of , , built as described in Proposition 3.3, based on the sample and on the constants and . With probability at least ,

 supθ∈Sd∣∣E(θ)−⟨θ,E(X)⟩∣∣≤Bn,k=√¯¯¯¯T+A/kn−k+√2(¯¯¯v+A/k)log(δ−1)n−k,

and we can, if needed, deduce from an estimator such that with probability at least ,

 ∥ˆm−E(X)∥≤2Bn,k.

If we want the correction term to behave as a second order term when tends to , we can for example take , in which case is equivalent to at infinity, so that is equivalent to

 √¯¯¯¯Tn+√2¯¯¯vlog(δ−1)n.

Let us also mention that a simpler estimator, obtained by shrinking the norm of , is also possible. It comes with a sub-Gaussian deviation bound under the slightly stronger hypothesis that for some (non necessarily integer) exponent , and is described in Catoni and Giulini (2017).

## 4 Mean matrix estimate

Let be a random matrix and let be independent copies of . In this section, we will provide an estimator for .

From the previous section, we already have an estimator of with a bounded Hilbert-Schmidt norm , since from the point of view of the Hilbert-Schmidt norm, is nothing but a random vector of size . Here, we will be interested in another natural norm, the operator norm

 ∥M∥∞=supθ∈Sq∥Mθ∥.

Indeed, recalling that

 ∥M∥∞=supθ∈Sq,ξ∈Sp⟨ξ,Mθ⟩=supξ∈Sp∥M⊤ξ∥=supθ∈Sq,ξ∈SpTr(θξ⊤M),

we see that we can deduce results from the previous section on vectors, considering the scalar product between matrices

 ⟨M,N⟩=Tr(M⊤N),M,N∈Rp×q,

and the part of the unit sphere defined as

 S={ξθ⊤:ξ∈Sp,θ∈Sq}.

Doing so, we obtain in the uncentered case a bound of the form

We will show in the next section that the second -dependent term is satisfactory whereas the first -independent term can be improved.

### 4.1 Estimation without centering

Consider the influence function defined by equation (1) on page 1.

For any , let , where is the identity matrix of size . In the same way, let , . Consider the estimator of defined as

 E(ξ,θ)=1λnn∑i=1∫ψ(λ⟨ξ′,Miθ′⟩)dνξ(ξ′)dρθ(θ′),ξ∈Rp,θ∈Rq.
###### Proposition 4.1.

For any parameters , , with probability at least , for any and any ,

 ∣∣E(ξ,θ)−E(⟨ξ,Mθ⟩)∣∣≤λ2[E(⟨ξ,Mθ⟩2)+E(∥Mθ∥2)β+E(∥M⊤ξ∥2)γ+E(∥M∥2HS)βγ+β+γ+2log(δ−1)2nλ.
###### Proof.

The PAC-Bayesian inequality of Proposition 2.1 tells us that with probability at least , for any and any ,

 E(ξ,θ)≤λ−1∫log{E[exp(ψ(λ⟨ξ′,Mθ′⟩))]}dνξ(ξ′)dρθ(θ′)+K(νξ,ν0)nλ+K(ρθ,ρ0)nλ+log(δ−1)nλ,

Using the properties of (Lemma 3.1) and Fubini’s lemma, we get

 E(ξ,θ)≤⟨ξ,E(M)θ⟩+λ2E(∫⟨ξ′,Mθ′⟩2dνξ(ξ′)dρθ(θ′))+β+γ+2log(δ−1)2nλ.

As

 ∫⟨ξ′,Mθ′⟩2dνξ(ξ′)dρθ(θ′)=⟨ξ,Mθ⟩2+∥Mθ∥2β+∥M⊤ξ∥2γ+∥M∥2HSβγ,

this concludes the proof. ∎

Let us now discuss the question of computing . Remark that, according to Lemma 3.2, for any ,

 ∫ψ(⟨ξ′,x⟩)dνξ(ξ′)=φ(⟨ξ,x⟩,β−1/2∥x∥)=⟨ξ,x⟩−⟨ξ,x⟩∥x∥22β−⟨ξ,x⟩36+r(⟨ξ,x⟩,β−1/2∥x∥).

It is also easy to check that

 ∫∥Miθ′∥2dρθ(θ′)=∥Miθ∥2+∥Mi∥2HSγ,∫⟨ξ,Miθ′⟩∥Miθ′∥2dρθ(θ′)=⟨ξ,Miθ⟩∥Miθ∥2+1γ⟨ξ,Miθ⟩∥Mi∥2HS+2γ⟨ξ,MiM⊤iMiθ⟩,and ∫⟨ξ,Miθ′⟩3dρθ(θ′)=⟨ξ,Miθ⟩3+3γ⟨ξ,Miθ⟩∥M⊤iξ∥2.

Consider a standard random vector . We obtain that

 ∫ψ(λ⟨ξ′,Miθ′⟩)dνξ(ξ′)dρθ(θ′)=∫(λ⟨ξ,Miθ′⟩−λ3⟨ξ,Miθ′⟩∥Miθ′∥22β−λ36⟨ξ,Miθ′⟩3+r(λ⟨ξ,Miθ′⟩,λβ−1/2∥Miθ′∥))dρθ(θ′),

so that

 E(ξ,θ)=1nn∑i=1⟨ξ,Miθ⟩−λ26⟨ξ,Miθ⟩3−λ22β⟨ξ,Miθ⟩∥Miθ∥2−λ22γ⟨ξ,Miθ⟩∥M⊤iξ∥2−λ22βγ⟨ξ,Miθ⟩∥Mi∥2HS−λ2βγ⟨ξ,MiM⊤iMiθ⟩+1λE[r(λ⟨M⊤iξ,θ+γ−1/2Wq⟩,λβ−1/2∥Mi(θ+γ−1/2Wq)∥)].

The last term is not explicit, since it contains an expectation, but should be most of the time a small reminder and can be evaluated using a Monte-Carlo numerical scheme. This gives a more explicit and efficient method than evaluating directly using a Monte-Carlo simulation for the couple of random variables .

###### Proposition 4.2.

Assume that the following finite bounds are known

 v ≥supξ∈Sp,θ∈SqE(⟨ξ,Mθ⟩2)=supξ∈Sp,θ∈Sq(ξ⊤⊗ξ⊤)E(M⊗M)(θ⊗θ), t u ≥supξ∈SpE(∥M⊤ξ∥2)=supξ∈Sp⟨ξ,E(MM⊤)ξ⟩=∥E(M⊤M)∥∞, T ≥E(∥M∥2HS),

and choose

 λ= ⎷β+γ+2log(δ−1)n(v+t/β+u/γ+T/(βγ)).

For any values of , , with probability at least , for any , any ,

 ∣∣E(ξ,θ)−⟨ξ,E(M)θ⟩∣∣≤Bn=√(v+tβ+uγ+Tβγ) β+γ+2log(δ−1)n.

Consider now any estimator of . With probability at least ,

 ∥ˆm−E(M)∥∞≤supξ∈Sp,θ∈Sq∣∣E(ξ,θ)−⟨ξ,ˆmθ⟩∣∣+Bn.

In particular, if we choose such that,

 supξ∈Sp,θ∈Sq∣∣E(ξ,θ)−⟨ξ,ˆmθ⟩∣∣≤Bn,

with probability at least , this choice is possible and

 ∥ˆm−E(M)∥∞≤2Bn.
###### Remark 4.1.

In particular, choosing , we get

 Bn≤ ⎷2vn(2log(δ−1)+4max{t+uv,√Tv}).

The bound is of the type , with a complexity (or dimension) term equal to

 C=4max{t+uv,√Tv}+log(δ−1).
###### Remark 4.2.

Let us envision a simple case to compare the precision of the bounds in a setting where dimension-free and dimension-dependent bounds coincide. Assume more specifically that the entries of the matrix ,

 Mi,j1≤i≤p,1≤j≤q,

are centered and i.i.d.. Assume that is known, and take

 v =supξ∈Sp,θ∈SqE(⟨ξ,Mθ⟩2)=σ2, t =supθ∈SqE(∥Mθ∥2)=pσ2, u =supξ∈SpE(∥M⊤ξ∥2)=qσ2, T =E(∥M∥2HS)=pqσ2.

Choosing , we get a complexity term equal to

 C=4(p+q)+log(δ−1),

whereas the bound of the previous section made for vectors has a complexity factor equal to .

### 4.2 Controlling both the operator norm error and the Hilbert-Schmidt error

There are situations where it is desirable to control both and . To do so we can very easily combine Propositions 3.3 and Proposition 4.2, since these two propositions are based on the construction of confidence regions.

More precisely, first consider as a vector and use the scalar product

 ⟨θ,M⟩HS=Tr(θ⊤M),θ∈Rp×q.

Applying Proposition 3.3, we can build an estimator such that with probability at least ,

On the other hand, we can also apply Proposition 4.2 and build an estimator , such that with probability at least ,

 supξ∈Sp,θ∈Sq∣∣E(ξ,θ)−⟨ξ,E(M)θ⟩∣∣≤Bn=√2vn(2log(δ−1)+4max{t+uv,√Tv}).
###### Proposition 4.3.

Consider a matrix such that

 and supξ∈Sp,θ∈Sq∣∣E(ξ,θ)−⟨ξ,ˆmθ⟩∣∣≤Bn.

Combining Propositions 3.3 and 4.2 shows that, with probability at least , such a matrix exists and satisfies both

 ∥ˆm−E(M)∥HS≤2An and % ∥ˆm−E(M)∥∞≤2Bn.

Remark that is typically smaller than as expected in interesting large dimension situations.

### 4.3 Centered estimator

As already done in the case of the estimation of the mean of a random vector, we deduce in this section centered bounds from the uncentered bounds of the previous sections, using sample splitting.

Put and . Assume that we know finite constants such that

 supξ∈Sp,θ∈SqE(⟨ξ,¯¯¯¯¯¯Mθ⟩2)≤¯¯¯v<∞, supθ∈SqE(∥¯¯¯¯¯¯Mθ∥2)≤¯t<∞, supξ∈SpE(∥¯¯¯¯¯¯M⊤ξ∥2)≤¯¯¯u<∞,

When this is true, we can take for the previous uncentered constants

 v=¯¯¯v+∥m∥2∞,t=¯t+∥m∥2∞,u=¯¯¯u+∥m∥2∞,T=¯¯¯¯T+∥m∥2HS.

In view of this, it is suitable to assume that we also know some finite constants and such that

As we see that the Hilbert-Schmidt norm comes into play, we will use the combined preliminary estimate provided by Proposition 4.3.

Given an i.i.d. matrix sample , first use to build a preliminary estimator as described in Proposition 4.3. With probability at least ,

 ∥˜m−m∥HS ≤√Ak and ∥˜m−m∥∞≤√Bk, where A =4(√2(¯¯¯v+b)log(4/δ)+√¯T+c)2 and B =8(¯¯¯v+b)(2log(4/δ)+4max{¯t+¯¯¯u+2b¯¯¯v+b,(¯¯¯¯T+c¯¯¯v+b)1/2}).

Then use the sample to build an estimator based on the construction described in Proposition 4.2, at confidence level . It is such that with probability at least ,

If we choose for instance , we obtain that

 C