A Matlab benchmarking toolbox for kernel adaptive filtering
We introduce a probabilistic approach to the LMS filter. By means of an efficient approximation, this approach provides an adaptable step-size LMS algorithm together with a measure of uncertainty about the estimation. In addition, the proposed approximation preserves the linear complexity of the standard LMS. Numerical results show the improved performance of the algorithm with respect to standard LMS and state-of-the-art algorithms with similar complexity. The goal of this work, therefore, is to open the door to bring some more Bayesian machine learning techniques to adaptive filtering.READ FULL TEXT VIEW PDF
The bilateral filter is a non-linear filter that uses a range filter alo...
Probabilistic solvers for ordinary differential equations (ODEs) assign ...
Recent work suggests improving the performance of Bloom filter by
There is a need to improve the capability of the adaptive filtering algo...
This paper exploits Geometric (Clifford) Algebra (GA) theory in order to...
This paper is concerned with an efficient numerical method for solving t...
Stochastic models that predict adaptive filtering algorithms performance...
A Matlab benchmarking toolbox for kernel adaptive filtering
A Matlab benchmarking toolbox for kernel adaptive filtering
Probabilistic models have proven to be very useful in a lot of applications in signal processing where signal estimation is needed [1, 2, 3]. Some of their advantages are that 1) they force the designer to specify all the assumptions of the model, 2) they provide a clear separation between the model and the algorithm used to solve it, and 3) they usually provide some measure of uncertainty about the estimation.
On the other hand, adaptive filtering is a standard approach in estimation problems when the input is received as a stream of data that is potentially non-stationary. This approach is widely understood and applied to several problems such as echo cancellation , noise cancellation , and channel equalization .
Although these two approaches share some underlying relations, there are very few connections in the literature. The first important attempt in the signal processing community to relate these two fields was the connection between a linear Gaussian state-space model (i.e. Kalman filter) and the RLS filter, by Sayed and Kailath and then by Haykin et al. . The RLS adaptive filtering algorithm emerges naturally when one defines a particular state-space model (SSM) and then performs exact inference in that model. This approach was later exploited in  to design a kernel RLS algorithm based on Gaussian processes.
A first attempt to approximate the LMS filter from a probabilistic perspective was presented in , focusing on a kernel-based implementation. The algorithm of  makes use of a Maximum a Posteriori (MAP) estimate as an approximation for the predictive step. However, this approximation does not preserve the estimate of the uncertainty in each step, therefore degrading the performance of the algorithm.
In this work, we provide a similar connection between state-space models and least-mean-squares (LMS). Our approach is based on approximating the posterior distribution with an isotropic Gaussian distribution. We show how the computation of this approximated posterior leads to a linear-complexity algorithm, comparable to the standard LMS. Similar approaches have already been developed for a variety of problems such as channel equalization using recurrent RBF neural networks, or Bayesian forecasting . Here, we show the usefulness of this probabilistic approach for adaptive filtering.
The probabilistic perspective we adopt throughout this work presents two main advantages. Firstly, a novel LMS algorithm with adaptable step size emerges naturally with this approach, making it suitable for both stationary and non-stationary environments. The proposed algorithm has less free parameters than previous LMS algorithms with variable step size [13, 14, 15]
, and its parameters are easier to be tuned w.r.t. these algorithms and standard LMS. Secondly, the use of a probabilistic model provides us with an estimate of the error variance, which is useful in many applications.
Experiments with simulated and real data show the advantages of the presented approach with respect to previous works. However, we remark that the main contribution of this paper is that it opens the door to introduce more Bayesian machine learning techniques, such as variational inference and Monte Carlo sampling methods , to adaptive filtering.
Throughout this work, we assume the observation model to be linear-Gaussian with the following distribution,
where is the variance of the observation noise,
is the regression vector andis the parameter vector to be sequentially estimated, both -dimensional column vectors.
In a non-stationary scenario, follows a dynamic process. In particular, we consider a diffusion process (random-walk model) with variance for this parameter vector:
denotes the identity matrix. In order to initiate the recursion, we assume the following prior distribution on
Given the described probabilistic SSM, we would like to infer the posterior probability distribution. Since all involved distributions are Gaussian, one can perform exact inference, leveraging the probability rules in a straightforward manner. The resulting probability distribution is
in which the mean vector is given by
where we have introduced the auxiliary variable
and the covariance matrix is obtained as
Note that the mode of
, i.e. the maximum-a-posteriori estimate (MAP), coincides with the RLS adaptive rule
This rule is similar to the one introduced in .
Finally, note that the covariance matrix is a measure of the uncertainty of the estimate conditioned on the observed data . Nevertheless, for many applications a single scalar summarizing the variance of the estimate could prove to be sufficiently useful. In the next section, we show how such a scalar is obtained naturally when is approximated with an isotropic Gaussian distribution. We also show that this approximation leads to an LMS-like estimation.
The proposed approach consists in approximating the posterior distribution , in general a multivariate Gaussian distribution with a full covariance matrix, by an isotropic spherical Gaussian distribution
In order to estimate the mean and covariance of the approximate distribution
, we propose to select those that minimize the Kullback-Leibler divergence with respect to the original distribution, i.e.,
The derivation of the corresponding minimization problem can be found in Appendix A. In particular, the optimal mean and the covariance are found as
We now show that by using (4) in the recursive predictive and filtering expressions we obtain an LMS-like adaptive rule. First, let us assume that we have an approximate posterior distribution at , . Since all involved distributions are Gaussian, the predictive distribution is obtained as
where the mean vector and covariance matrix are given by
From (6), the posterior distribution at time
can be computed using Bayes’ Theorem and standard Gaussian manipulations (see for instance[17, Ch. 4]). Then, we approximate the posterior with an isotropic Gaussian,
Note that, instead of a gain matrix as in Eq. (3), we now have a scalar gain that operates as a variable step size.
Finally, to obtain the posterior variance, which is our measure of uncertainty, we apply (5) and the trick ,
If MAP estimation is performed, we obtain an adaptable step-size LMS estimation
At this point, several interesting remarks can be made:
The adaptive rule (11) has linear complexity since it does not require us to compute the full matrix .
Finally, the proposed adaptable step-size LMS has only two parameters, and , (and only one, , in stationary scenarios) in contrast to other variable step-size algorithms [13, 14, 15]. More interestingly, both and have a clear underlying physical meaning, and they can be estimated in many cases. We will comment more about this in the next section.
We evaluate the performance of the proposed algorithm in both stationary and tracking experiments. In the first experiment, we estimate a fixed vector of dimension . The entries of the vector are independently and uniformly chosen in the range . Then, the vector is normalized so that . Regressors are zero-mean Gaussian vectors with identity covariance matrix. The additive noise variance is such that the SNR is dB. We compare our algorithm with standard RLS and three other LMS-based algorithms: LMS, NLMS , VSS-LMS .111The used parameters for each algorithm are: for RLS , ; for LMS ; for NLMS ; and for VSS-LMS , , . The probabilistic LMS algorithm in  is not simulated because it is not suitable for stationary environments.
In stationary environments, the proposed algorithm has only one parameter, . We simulate both the scenario where we have perfectly knowledge of the amount of noise (probLMS1) and the case where the value is times smaller than the actual value (probLMS2). The Mean-Square Deviation (), averaged out over independent simulations, is presented in Fig. 1.
The performance of probabilistic LMS is close to RLS (obviously at a much lower computational cost) and largely outperforms previous variable step-size LMS algorithms proposed in the literature. Note that, when the model is stationary, i.e. in (2), both the uncertainty , and the adaptive step size , vanish over time. This implies that the error tends to zero when goes to infinity. Fig. 1 also shows that the proposed approach is not very sensitive to a bad choice of its only parameter, as demonstrated by the good results of probLMS2, which uses a that is times smaller than the optimal value.
In a second experiment, we test the tracking capabilities of the proposed algorithm with real data of a wireless MISO channel acquired in a realistic indoor scenario. More details on the setup can be found in . Fig. 2 shows the real part of one of the channels, and the estimate of the proposed algorithm. The shaded area represents the estimated uncertainty for each prediction, i.e. . Since the experimental setup does not allow us to obtain the optimal values for the parameters, we fix these parameters to their values that optimize the steady-state mean square deviation (MSD). Table 1 shows this steady-state MSD of the estimate of the MISO channel with different methods. As can be seen, the best tracking performance is obtained by standard LMS and the proposed method.
We have presented a probabilistic interpretation of the least-mean-square filter. The resulting algorithm is an adaptable step-size LMS that performs well both in stationary and tracking scenarios. Moreover, it has fewer free parameters than previous approaches and these parameters have a clear physical meaning. Finally, as stated in the introduction, one of the advantages of having a probabilistic model is that it is easily extensible:
If, instead of using an isotropic Gaussian distribution in the approximation, we used a Gaussian with diagonal covariance matrix, we would obtain a similar algorithm with different step sizes and measures of uncertainty, for each component of . Although this model can be more descriptive, it needs more parameters to be tuned, and the parallelism with LMS vanishes.
Similarly, if we substitute the transition model of (2) by an Ornstein-Uhlenbeck process,
a similar algorithm is obtained but with a forgetting factor multiplying in (11). This algorithm may have improved performance under such a kind of autoregresive dynamics of , though, again, the connection with standard LMS becomes dimmer.
A similar approximation technique could be applied to more complex dynamical models, i.e. switching dynamical models . The derivation of efficient adaptive algorithms that explicitly take into account a switch in the dynamics of the parameters of interest is a non-trivial and open problem, though the proposed approach could be useful.
Finally, like standard LMS, this algorithm can be kernelized for its application in estimation under non-linear scenarios.
We want to approximate by . In order to do so, we have to compute the parameters of , and , that minimize the following Kullback-Leibler divergence,
Using symmetry arguments, we obtain
Then, (12) gets simplified into
The variance is computed in order to minimize this Kullback-Leibler divergence as
Deriving and making it equal zero leads to
Finally, since the divergence has a single extremum in ,
“A tutorial on hidden Markov models and selected applications in speech recognition,”Proc. IEEE, vol. 77, no. 2, pp. 257–286, 1989.
“Recurrent radial basis function networks for optimal symbol-by-symbol equalization,”Signal Process., vol. 40, no. 1, pp. 53–63, 1994.