Consider independent random variables
having the same probability distribution of densitywith respect to Lebesgue’s measure.
The usual Parzen  - Rosenblatt  kernel estimator of is defined by
where and is a kernel. In 1969, Wolverton and Wagner introduced in  a variant of defined by
where and . Thanks to its recursive form, this type of estimator is well-suited to online treatment of data: by denoting ,
Thus, up-dating the estimator when new observations are available is easy and fast.
We can mention here that several variants or generalizations of the Wolverton and Wagner (WW) estimator have been proposed: see Yamato , Wegman and Davies , Hall and Patil . They were studied from almost sure convergence point of view, or asymptotic rates of convergence under fixed regularity assumptions. We choose to focus on Wolverton and Wagner estimator but our results and discussions may be applied to these.
Theoretical developments concerning either classical Parzen-Rosenblatt or WW recursive kernels estimators occurred recently following different and independent roads.
On the one hand, several recent works are dedicated to efficient and data-driven bandwidth selection, see Goldenshluger and Lespki  and several companion papers by these authors, or Lacour et al.  who proposed a modification of the method. The original Goldenshluger and Lepski (GL) method was difficult to implement because it turned out to be numerically consuming and with calibration difficulties, see Comte and Rebafka . This is why the improvement proposed in Lacour et al.  has both theoretical and practical interest.
On the other hand, the increase of computer speed and of data sets sizes made fast up-dating of estimators mandatory. The theoretical developments in this context are in the field of stochastic algorithms (see e.g. Mokkadem et al. ) or in view of specific applications (see Bercu et al. ).
Bandwidths have to be chosen for WW estimators as for Parzen-Rosenblatt ones, and this choice is crucial to obtain good performances. This is why we propose to extend to this context general risk study as described in Tsybakov  and the GL method as improved by Lacour et al. . More precisely, considering for instance for a parameter in formula (1), we study adaptive selection of . We prove risk bounds for the Mean Integrated Squares Error (MISE) of the resulting estimator where and .
Amiri  proved that for with regularity 2 and an adequate choice of the bandwidth, Parzen-Rosenblatt’s estimator had asymptotical smaller risk than the WW estimator. We propose an empirical finite sample study of this question, together with an interesting insight on the gain brought by higher order kernels.
Now, clearly, plugging in the estimator makes the recursivity fail. Therefore, a mixed strategy is required with initial estimation of on the first -sample and recursive up-dating relying on this "freezed" value on the following -sample, where should have the same order as . This is what is experimented in our final section, and empirically proved to be an appropriate strategy.
This paper provides in Section 2 controls of the MSE and of the MISE of the estimator under general regularity conditions on . Then, in Section 3, the well-known Goldenshluger-Lepski’s bandwidth selection method for Parzen-Rosenblatt’s estimator is extended to Wolverton-Wagner’s estimator. Lastly, an estimator in the spirit of Lacour et al.  is studied from both theoretical and practical point of view in Section 4. Concluding remarks present a mixed strategy in Section 5. Proofs are relegated in Section 6.
Consider . The space of -Hölder continuous functions from into itself is denoted by and equipped with the -Hölder semi-norm defined by
For and ,
For every square integrable functions ,
for every .
2. Bounds on the MSE and the MISE of Wolverton-Wagner’s estimator
Consider and . Throughout this section, the map fulfills the following assumption.
The map is integrable for every ,
Let us establish a control of the MSE of Wolverton-Wagner’s estimator under the following condition on .
The map belongs to .
Now, let us establish a control of the MISE of Wolverton-Wagner’s estimator under Nikolski’s condition on .
The map belongs to and there exists a constant such that
Remark. Assumptions 2.1, 2.2 and 2.4 are standard for density estimation, see Tsybakov (2009). Moreover, if we set , we recover the results stated in Section 1.2.1 for Proposition 2.3 and in Theorem 1.3 for Proposition 2.5 in Tsybakov (2009) (that is a squared bias term of order
and a variance term of order).
The estimator is consistent if the risk tends to zero when grows to infinity, that is if
Let us consider
with (otherwise or cannot tend to zero). Then
with the intermediate case
Indeed, if , then
On the other hand,
As a consequence, we have the following result:
Clearly, this is the optimal rate in the minimax sense, see Goldenshluger and Lepski  and the references therein.
Moreover, if and only if,
Therefore, makes the upper bound on the risk minimal. ∎
3. Goldenshluger-Lepski’s method for Wolverton-Wagner’s estimator
This section provides an extension of the well-known Goldenshluger-Lepski’s bandwidth selection method for Parzen-Rosenblatt’s estimator to Wolverton-Wagner’s estimator.
Throughout this section, assume that
where and the maps from into fulfill the following assumption.
For every ,
Moreover, is decreasing and one to one from into .
For instance, one can take as above for every and .
and the set , where and
A way to extend the Goldenshluger-Lepski bandwidth selection method to Wolverton-Wagner’s estimator is to solve the minimization problem
with not depending on and
In the sequel, the map fulfills the following assumption.
For every and ,
For every ,
Then, for any and ,
) has the order of the optimal rate, and is reached automatically by the data driven estimator. This result is more precise than the heuristics associated with cross-validation.
We mentioned previously that the optimal theoretical choice for under Assumptions 2.1 and 2.4 is . Here, the selected should be at nearest of this value, e.g. if is as in (6), distant from less than of the good choice. We may therefore consider that provides an estimate of and thus an estimate of the regularity of (at least for huge values of ).
4. The Lacour-Massart-Rivoirard (LMR) estimator
4.1. Estimator and main result
The Goldenshluger-Lepski method has been acknowledged as being difficult to implement, due to the square grid in required to compute intermediate versions of the criterion and to the lack of intuition in the choice of the constant which should be calibrated from preliminary simulation experiments. This is the reason why Lacour et al.  investigated and proposed a simplified criterion relying on deviation inequalities for -statistics due to Houdré and Reynaud-Bouret . This inequality applies in our more complicated context and Lacour-Massart-Rivoirard’s result can be extended here as follows.
Let us recall that for every and set
Let be the maximal proposal in and consider
Then, we define
In the sequel, , and fulfill the following assumption.
The kernel is symmetric, ,
Consider and . Under Assumption 4.1, there exists three deterministic constants , not depending on , and , such that with probability larger than ,
Remark. The term is negligible because it is a pure bias term for smallest bandwidth (e.g., under Assumption 2.4, it has order , see (3), and thus if is near of 1 and ). The terms following are of order and are always negligible compared to nonparametric rates in our setting. Therefore, the bound given in Proposition 4.2 says that the MISE of the adaptive estimator has the order of the best estimator of the collection, up to a multplicative factor larger than 1. This is the method we implement in the next section: it is faster than GL method and with no constant to calibrate in the penalty.
4.2. Simulation experiments
We consider basic densities with different types and orders of regularity:
, density ,
a mixed gaussian , density ,
, density ,
a mixed beta , density ,
, density ,
a mixed gamma , density ,
with , a Laplace density.
The densities and have infinite regularity, and should rather have regularity of order less than 2, and less than 4, and less than 1. This choice should allow to study the influence of the order of the kernel.
|LMR for WW||Original LMR|
Denoting by the density of a centered Gaussian random variable with variance equal to , we consider the following kernels:
a Gaussian kernel, which is of order 1,
a Gaussian-type kernel of order 3, ,
a Gaussian-type kernel of order 5, ,
a Gaussian-type kernel of order 7, .
With all these kernels, the penalty terms are computed analytically and without approximation. Indeed, for , it holds that
We compute the variable bandwidth estimator as described in Section 4 and select in a collection of equispaced values between 0 and 0.5 while the bandwidth associated with observation is . We also compute the original estimator of Lacour et al.  with bandwidth which does not depend on the observation and is selected among values in the set .
For comparison, we give the performance of the Matlab density estimator obtained from function (denoted by ks in Table 1), which entails a different bandwidth selection method and relies on a gaussian kernel.
We compute the integrated -risk associated with all the final estimators, evaluated at equispaced points in the range of the observations, averaged over repetitions:
where is the estimator computed for path .
Results are gathered in Table 1 and deserve some comments. As expected, when increasing from 250 to 1000, the resulting MSEs decrease and seem to be more improved in LMR methods of both types than for ks estimator. Increasing the order of the kernel systematically improves the results, except for the lowest regularity density , which is at best with , but it is interesting to note that taking higher order kernel is always a good strategy: if a loss occurs, it is negligible while the improvement, when it happens, is in all cases significant. Estimator ks fails for all mixed densities , and and provides rather bad results in these cases, for both sample sizes. For the other densities (, , , ), the results obtained with kernel and LMR method are better than with ks for (Gaussian case), and of comparable order in all other cases. Now if we compare the LMR and WW-LMR results both with kernel , we conclude that the WW-LMR method wins in 10 cases out of 14, but not significantly.
The first line of Figure 1 illustrates in the left picture the way Matlab estimator fails for mixed densities (here the mixed Gaussian ) by probably selecting a too large bandwidth, here . The two LMR estimators are almost confounded. The middle and right pictures present the estimators among which the LMR procedure makes the selection, for the same path: we observe that the collection of proposals are rather different. The second line of Figure 1 presents the same type of results for density , and sample size . Figure 2 shows beams of 30 final estimators for sample size , for the three estimators LMR-WW with , LMR with and ks, showing very similar behaviours.
A last remark corresponding to numerical results we do not report in detail is the following. For most densities, the value of selected by the LMR strategy decreases, and the value of increases, when the order of the kernel increases. Exceptions are densities with lower regularity (the beta , mixed beta and Laplace densities) for which the last value of selected with is less than the one selected with . This illustrates the fact that, asymptotically, if is the regularity index of the density and the order of the kernel, the optimal choice is for of order and for , .
5. Concluding remarks
Our study illustrates that bandwidth selection is an important step for kernel functional estimation, and recent methods are really powerful whatever the type of density to recover.
Our simulations show also that, even if it implies non necessarily nonnegative kernels and thus density estimators, increasing the order of the kernel improves the estimation both in the theory and in practice. Also, we proved that variable bandwidth for WW-type estimators can reach excellent rates, again both in theory and in practice, provided that adaptive choice of this variable bandwidth is performed. The orders of practical MISEs show that this WW-strategy provides results of the same order as the more classical bandwidth methods.
However, one may wonder how to keep these ideas compatible with recursive procedures and online updating of the kernel estimator. We believe that the adaptive bandwidth, whatever its type, can be selected on a preliminary sample and then, "freezed" to this selected value and plugged in the estimator. We tested this strategy on simulations: for each sample, another independent sample is generated, and a second estimator is computed based on the new sample, with the value of or selected for the first data set. Table 2 provides the MISEs obtained for the estimator with adaptive bandwidth with kernel for WW-estimator (column ) or for NW-estimator (column ), for comparison with MISEs of the estimator computed on an independent sample with the values based on the previous selections (columns "Fixed " and "Fixed "). Surprisingly, the results have exactly the same orders (we expected a slight deterioration): this proves that the proposal of preliminary selection is really fine.
6.1. Proof of Proposition 2.3
First, by the bias-variance decomposition,
Let us find controls for and .
On the one hand,
where On the other hand,
For every and , by Taylor-Lagrange’s formula there exists such that
Then, by Assumption 2.1,
Therefore, by Assumption 2.2,
where In conclusion, by Equation (8), setting , we get
6.2. Proof of Proposition 2.5
In order to prove Proposition 2.5, the two following generalizations of Minkowski’s inequality are required.
For any Borel function , if is integrable and
is a Borel function, then