Log In Sign Up

GMM based multi-stage Wiener filtering for low SNR speech enhancement

by   Wageesha Manamperi, et al.

This paper proposes a single-channel speech enhancement method to reduce the noise and enhance speech at low signal-to-noise ratio (SNR) levels and non-stationary noise conditions. Specifically, we focus on modeling the noise using a Gaussian mixture model (GMM) based on a multi-stage process with a parametric Wiener filter. The proposed noise model estimates a more accurate noise power spectral density (PSD), and allows for better generalization under various noise conditions compared to traditional Wiener filtering methods. Simulations show that the proposed approach can achieve better performance in terms of speech quality (PESQ) and intelligibility (STOI) at low SNR levels.


Dual-Stage Low-Complexity Reconfigurable Speech Enhancement

This paper proposes a dual-stage, low complexity, and reconfigurable tec...

PercepNet+: A Phase and SNR Aware PercepNet for Real-Time Speech Enhancement

PercepNet, a recent extension of the RNNoise, an efficient, high-quality...

Subjective Evaluation of Deep Neural Network Based Speech Enhancement Systems in Real-World Conditions

Subjective evaluation results for two low-latency deep neural networks (...

SNR-based teachers-student technique for speech enhancement

It is very challenging for speech enhancement methods to achieves robust...

Improving the Generalizability of Deep Neural Network Based Speech Enhancement

Enhancing noisy speech is an important task to restore its quality and t...

PROSE: Perceptual Risk Optimization for Speech Enhancement

The goal in speech enhancement is to obtain an estimate of clean speech ...

1 Introduction

Speech enhancement is useful in many applications such as automatic speaker recognition, voice communication, and hearing aid devices [wang2018supervised]. Notable contributions to speech enhancement were developed over the past few decades. However, the speech enhancement in low signal-to-noise ratio (SNR) conditions is still challenging. This paper addresses the problem of single-channel (i.e., one-microphone) speech enhancement using a Wiener filtering approach for low SNR levels and non-stationary noise conditions.

Single-channel speech enhancement methods can be classified mainly into statistical-based approaches and data-driven approaches. Statistical-based approaches include spectral subtraction

[loizou2007speech], Wiener filtering [scalart1996speech, srinivasan2005codebook, chehrehsa2016single, chehresa2011mmse, savoji2014speech], the minimum mean-square error (MMSE) short-time spectral amplitude estimation [ephraim1984speech], etc. In general, these methods tend to perform poorly in non-stationary noisy conditions and thus often assume stationarity to obtain good performance. Compared to data driven methods, statistical methods are computationally efficient, yet the overall quality of their speech enhancement performance is typically lesser.

Existing data-driven approaches for speech enhancement include non-negative matrix factorization (NMF) [mohammadiha2013supervised]

and neural networks (NNs)

[wang2014training, wang2005ideal, williamson2015complex, choi2018phase, pascual2017segan, rethage2018wavenet, luo2019conv, pandey2019tcnn, zeghidour2021wavesplit, saleem2019deep]

. They often use learning methods with multi-layers representations with large training sets of data to achieve high quality enhancement. Prior work on deep-learning based methods have recently attracted much attention. Supervised learning methods using deep neural networks (DNNs) can be classified into two categories mainly as Time-Frequency (TF) domain methods and time domain methods. Generally, the DNN-based TF algorithms estimate a real or complex value mask for each TF bin such as ideal binary mask (IBM) and ideal ratio mask (IRM) to obtain the clean speech spectra and show robustness to non-stationary noise sources

[wang2014training, wang2005ideal, williamson2015complex, choi2018phase]. The time domain DNN-based algorithms such as SEGAN [pascual2017segan], WaveNeT [rethage2018wavenet], Conv-TasNet [luo2019conv], TCNN [pandey2019tcnn] and Wavesplit [zeghidour2021wavesplit] have shown great improvements over TF methods. These techniques model its clean speech equivalent accurately in high ( dB) and moderate ( dB) SNR levels using large models with high complexity and long hours of training and validation data.

In our contribution, an extension of the Gaussian mixture model (GMM) based single-channel speech enhancement approach in [chehresa2011mmse]

for modeling noise power spectral densities (PSDs) is presented. We improve the performance through a multi-stage parametric Wiener filter at low SNR levels and non-stationary noise conditions. Motivated by the success of the Wiener filtering methods in noise reduction, we propose to model noise in a multi-stage process using a set of Gaussian components and a parametric Wiener filter. In this way, the spectral enhancement is achieved by considering the set of noise GMM mean vectors based on its energy level at each stage. The proposed method is a simple, efficient, and unsupervised approach that improves speech quality while preserving speech intelligibility in noisy environments. We will show that multi-stage speech enhancement can improve performance by providing more accurate estimates of the noise PSD. Simulations show that a better speech enhancement is achieved compared to the Wiener filtering method in


2 Problem Definition

We address the single-channel speech enhancement problem by focusing on the case of only two sources, speech and noise. Let the received noisy speech be

where and

are the Short Time Fourier Transforms (STFTs) of the desired speech and noise, respectively,

and denotes the time frame and frequency bin indices, respectively with and where and are the number of time frames and frequency bins, respectively.

Our problem is to extract the speech signal from the noisy speech signal under low SNR conditions.

3 Parametric Wiener filtering

Due to its simplicity, the Wiener filter has been used for noise reduction in the past. As one of the steps in our novel GMM based noise suppression framework, we propose a parametric Wiener filter in this section.

We can use the typical Wiener filtering to estimate the speech in the STFT domain as , where is the gain function of the Wiener filter


where and denote the power spectral density (PSD) of the speech and noise, respectively, at the frequency bin of frame .

Since a parametric Wiener filter allows a compromise between the amount of speech distortion and noise reduction and is known to perform better in the presence of non-stationary noise conditions [haykin2008adaptive]

, we introduce a parameter to the noise variance

and a power coefficient that imposes a constant on the maximum allowable distortion of the desired speech signal and provides the minimum mean squared error (MMSE) linear estimator of [lim1979enhancement] as


Here, is the over- or underestimation factor and controls the compromise between noise reduction and speech distortion. can be fixed or frequency dependent on the spectral properties of the desired speech. However, in this work, we mainly focus on fixed values of for more aggressive noise reduction.

4 GMM based Speech Enhancement

4.1 Modeling speech and noise PSDs with GMMs

We use the Gaussian Mixture Model method [vaseghi2008advanced]

to model the PSDs of the mixture, speech and noise using the expectation-maximization (EM) algorithm. A GMM for a mixture signal can be composed of

Gaussians, and is given by


is the probability density function (PDF) of the mixture signal when both speech and noise signals are present,

is the contribution of the GMM component in the mixture recording, and is the Gaussian mixture of the mixture recording for the GMM component and frequency. Also, and denote the mean and covariance matrix of the mixture for the GMM component and frequency, respectively. Likewise, we can write the GMM of clean speech as

and the GMM of noise as

where and denotes the number of GMM components for the speech and noise, respectively. The means and covariances are denoted by , and with the speech and noise as a suffix and and are trained offline in the training phase.

We define the GMM mean vector of speech as by arranging the mean values of each frequency for a single GMM component into a vector. Similarly, we can define the GMM mean vector for noise as .

4.2 Multi-stage parametric Wiener filtering

The proposed model (as summarized in Algorithm 1), has a training phase and an enhancement phase. During the training phase, we extract the GMM mean vectors of clean speech and noise PSDs which will be fed into the enhancement phase.

We propose a multi-stage parametric Wiener filter method to improve the speech quality in the enhancement phase. At each stage , we update a set of noise GMM mean vectors to obtain a better estimate of noise PSD and Wiener filtering gain. We determine the number of stages by choosing a set of based on their energy level at each stage . Basically, we consider the high energy noise GMM mean vectors in the first stage, and group the rest in the subsequent stages.

We write the estimated PSD of noise as


where denotes the number of noise GMMs in stage , and are GMM mean values and power coefficients for the noise at stage , respectively.

The estimated PSD of speech can be written as


where is the power coefficients corresponding to the GMM mean values for the speech.

Assuming the speech signal and noise are uncorrelated, we can write the PSD of the mixture as the weighted sum of the GMM means of both speech and noise


Writing (5) for each frequency bin , we have a set of linear equations in a matrix form for the time frame as


where , , and . We can solve (6) to find the coefficients of the current time frame based on the mixture PSD at stage


where is the Moore-Penrose inverse.

This estimation may cause some power coefficients to be negative due to the pseudo-inverse of the mean matrix . The negative power coefficients make PSDs negative which is not possible. To avoid that, we normalize the matrix to set all the values to be positive such that the negative power coefficients are set to zero and the total power of the matrix is the same as the original matrix. We can reconstruct an estimation of the speech-only PSD and noise-only PSD in the current time frame using the (4) and (3), respectively.

Using (2), we construct the transfer function of the parametric Wiener filter , and use it to filter the STFT of the mixture and obtain the enhanced speech at stage as

Using the inverse STFT, can be converted to as the enhanced speech at the stage. We continue to filter out at the next stage by considering a set of noise GMM mean vectors .

  1. [noitemsep]

  2. Training phase

    • [noitemsep]

    • STFT of the clean speech, and noise-only signal: , and .

    • Model , and with GMMs.

    • Extract and .

  3. Enhancement phase

    • [noitemsep]

    • STFT of the noisy signal .

    • Select a set of based on .

    • Update filters for stage :

      • use and , and estimate

      • Estimate using the parametric Wiener filtering with , and

    • iSTFT of the enhanced signal .

Algorithm 1 Multi-stage speech enhancement

5 Experiment Validation

5.1 Implementation Details

5.1.1 Experimental Setup

We generate the noisy speech by adding clean speech to noise with SNR levels of dB, dB, dB, dB, and dB. In the training phase, we use a clean speech dataset from the TIMIT database [garofolo1993timit] and a noise dataset from the NOISEX database [Noisex] to model the GMMs of speech and noise PSDs, respectively. Also, here we normalize the PSDs to make their energy equal to one. We use two different types of noise: multi-talker babble noise and destroyer engine noise from the NOISEX database. It should be noted that we use different speech data to construct the noisy signal for the enhancement phase.

5.1.2 Algorithm Parameters

The number of Gaussians for the speech and noise GMMs are chosen to be and following a trial and error approach proposed in [chehrehsa2016single],[chehresa2011mmse]. In order to obtain better denoising performance at low SNR levels, we use and . We set

and all signals are down sampled at 8 kHz and transformed to the frequency domain by a 512 point STFT using a 20 ms Hanning window with zero padding and 50% of overlap.

5.1.3 Evaluation Methods

We evaluate the quality of the enhanced speech using objective measures, (i) perceptual evaluation of speech quality (PESQ) [recommendation2001perceptual] and (ii) the short-time objective intelligibility score (STOI) [taal2011algorithm]. The range of PESQ score is from to , whereas STOI score is from to .

5.2 Results and Analysis

Figure 1: PESQ scores for (a) babble noise, and (b) destroyer engine noise.

Figure 2: STOI scores (in percent) for (a) babble noise, and (b) destroyer engine noise.

We compare our approach with Chehresa et al. [chehresa2011mmse]. Figure 1 and 2 show the speech enhancement performance over the two noise types. We observe that the proposed method can achieve higher PESQ and STOI scores at all input SNRs compared to [chehresa2011mmse] for both babble and engine noise. In the proposed algorithm, first stage does a considerable improvement while second stage does a minor improvement. In total it is a significant PESQ improvement compared to [chehresa2011mmse]. The proposed method leads to a significant improvement of predicted intelligibility, and a near-constant PESQ improvement of about 0.4 over the [chehresa2011mmse] and about 0.3 over the mixture signal for babble noise. Whereas PESQ improves about 0.2 over the [chehresa2011mmse] and about 0.4 over the mixture signal for destroyer engine noise. This suggests the proposed method improves speech quality without degrading intelligibility. Informal listening tests confirm these results. Furthermore, we evaluate the proposed method with DNN method (results from [saleem2019deep] which uses the same dataset [Noisex]) in terms of PESQ score for babble noise in Fig. 1 (a). The PESQ scores for babble noise are similar to the proposed method and DNN at low SNR levels until around dB. However, the DNN method has slightly improved PESQ scores at high SNR conditions. Note that we do not show the result of the STOI score of the DNN method due to its unavailability in the reference paper.

To further understand the staging effect in the proposed method, we examine the time varying spectrograms for the destroyer engine noise source at SNR level of dB in Fig. 3. Mixture signal is the clean speech contaminated by engine noise, and has PESQ of and STOI of scores. We notice that the destroyer engine noise has a higher harmonic structure and those GMM mean vectors lie in between Hz. For [chehresa2011mmse], we observe that the large residual noise is evident in Fig. 3(b). Whereas Fig. 3(c) and (d) show the noise reduction of the proposed dual staging process in the frequency region over to kHz. In order to highlight the noise suppression at each stage, we show a rectangle box on the plots (c) and (d). Fig. 3(d) shows the enhanced signal has PESQ of and STOI of scores. Note that we use only two stages () in the simulations to obtain the enhanced signal as the first stage performs a significant improvement compared to the second stage. However, the number of stages in the proposed method for an optimal output under a given noisy environment is left for future work.

Figure 3: Spectrogram plots of (a) the mixture, (b) Chehresa et al. [chehresa2011mmse], (c) stage I, and (d) stage II of the proposed method for destroyer engine noise at SNR level of dB.

Overall, the proposed method achieves noise suppression without compromising speech quality. Also, the dual staging method allows us to obtain a more precise estimate of the Wiener filter gain compared to the traditional Wiener filter. It also seems that [saleem2019deep] DNN method performance with babble noise is slightly similar to the proposed method at low SNR conditions.

6 Conclusion

In this paper, we have presented a novel single-channel speech enhancement approach using a GMM based multi-stage parametric Wiener filtering method at low SNR conditions. We show the proposed method facilitates dynamic noise adaptation and improves the speech quality while preserving speech intelligibility in a non-stationary noise environment with a simple multi-stage parametric Wiener filter. Evaluations verify that the proposed method achieves a better speech enhancement and noise suppression compared to the conventional Wiener filtering methods in terms of PESQ and STOI. For future work, we will investigate the applicability of the proposed method and its influence on objective speech quality at very low SNR levels.