. The denoising autoencoder (DAE), for example, is a typical variant of such networks, which is trained to directly convert a noisy speech spectrogram to a clean speech spectrogram with a supervised training. Alternatively, a DNN can be trained to predict time-frequency (TF) masks called ideal ratio masks (IRMs) that represent ratios of speech to input signals and are used for obtaining a speech spectrogram from a noisy spectrogram . Although it is necessary to prepare as training data a large amount of pairs of clean speech signals and their noisy versions, these supervised methods often deteriorate in unknown noisy environments. This calls for semi-supervised methods that are trained by using only clean speech data in advance and then adapt to unseen noisy environments.
Statistical source separation methods based on the additivity of speech and noise spectrograms have also been used for speech enhancement [7, 8, 9, 10, 11]. Non-negative matrix factorization (NMF) [12, 9]
, for example, regards a noisy speech spectrogram as a non-negative matrix and approximates it as the product of two non-negative matrices (a set of basis spectra and a set of the corresponding activations). If a partial set of basis spectra is trained in advance from clean speech spectrograms, the noisy spectrogram is decomposed into the sum of speech and noise spectrograms in a semi-supervised manner. Robust principal component analysis (RPCA)[13, 14] is another promising method that can decompose a noisy spectrogram into a sparse speech spectrogram and a low-rank noise spectrogram in an unsupervised manner. These conventional statistical methods, however, have a common problem that the linear representation or the sparseness assumption of speech spectrograms is not satisfied in reality and results in considerable signal distortion.
Recently, deep generative models such as generative adversarial networks (GANs) and variational autoencoders (VAEs) have gained a lot of attention for learning a probability distribution over complex data (e.g., images and audio signals) that cannot be represented by conventional linear models[15, 16, 17, 18, 19]. GANs and VAEs are both based on two kinds of DNNs having different roles. In GANs , a generator is trained to synthesize data that fool a discriminator from a latent space while the discriminator is trained to detect synthesized data in a minimax-game fashion. In VAEs [16, 17], on the other hand, an encoder that embeds observed data into a latent space and a decoder that generates data from the latent space are trained jointly such that the lower bound of the log marginal likelihood for the observed data is maximized. Although in general GANs can generate more realistic data, VAEs provide a principled scheme of inferring the latent representations of both given and new data.
In this paper we propose a unified probabilistic generative model of noisy speech spectra by combining a VAE-based generative model of speech spectra with an NMF-based generative model of noise spectra (Fig. 1
). The VAE is trained in advance from a sufficient amount of clean speech spectra and its decoder is used as a prior distribution on clean speech spectra included in noisy speech spectra. Given observed data, we can estimate both the latent representations of speech spectra as well as the basis spectra and their activations of noise spectra through Bayesian inference based on a Markov chain Monte Carlo (MCMC) initialized by the encoder of the VAE. Our Bayesian approach can adapt to both unseen speech and noise spectra by using prior knowledge of clean speech and the low-rankness assumption on noise instead of fixing all the parameters in advance.
2 Related work
This section overviews DNN-based speech enhancement and introduces the variational autoencoder (VAE).
2.1 DNN-based speech enhancement
Various network architectures and cost functions for enhancing speech signals have been reported [1, 2, 3, 4, 5, 6]. The popular approach of DNN-based speech enhancement is to train a DNN to directly represent clean speech . The DNN is trained using simulated noisy data constructed by adding noise to speech as input and clean speech as the target. There are several methods that combine a supervised NMF and a DNN [20, 21]
. A DNN is trained to estimate activation vectors of the pre-trained basis vectors corresponding to speech and noise. Bayesian WaveNet uses two networks: one, called a prior network, represents how likely a signal is speech and the other, called a likelihood network, represents how likely a signal is included in the observation. These two networks enhance the noisy speech signal with a maximum a posteriori (MAP) estimation. Another reported method uses two networks that are trained to represent how likely the input signal is speech or noise, respectively . The speech signal is enhanced by optimizing a cost function so that the estimated speech maximizes the speech-likelihood network and minimizes the noise-likelihood network. All the above mentioned methods are trained with datasets of both speech and noise signals. A DNN-based method using only training data of speech signals was reported . This method represents speech and noise spectra with two autoencoders (AEs). The AE for speech is pre-trained, whereas that for noise is trained at the inference for adapting to the observed noise signal. Since the inference of this framework is under-determined, the estimated speech is constrained to be represented by a pre-trained NMF model. It, thus, might have the same problem as the semi-supervised NMF.
2.2 Variational autoencoder
A VAE  is a framework for learning the probability distribution of a dataset.
In this subsection, we denote by a dataset that contains -dimensional samples ().
The VAE assumes that a -dimensional latent variable (denoted by )
follows a standard Gaussian distribution and each sampleis stochastically generated from a conditional distribution :
where represents a Gaussian distribution with mean parameter
and variance parameter.
is called a decoder and parameterized as a well-known probability density function whose parameters are given by nonlinear functions represented as neural networks. For example, Kingma et al. reported a VAE model that has the following Gaussian likelihood function:
where and are neural networks representing the mean and variance parameters, respectively.
The objective of VAE training is to find a likelihood function that maximizes the log marginal likelihood:
Since calculating this marginal likelihood is intractable, it is approximated with a variational Bayesian (VB) framework. The VAE first approximates the posterior distribution of with the following variational posterior distribution called an encoder:
where and are nonlinear functions representing the mean and variance parameters, respectively. These functions are formulated with DNNs. By using the variational posterior, the log marginal likelihood is lower-bounded as follows:
represents the Kullback-Leibler divergence. The VAE is trained so thatand maximize this variational lower bound. The first term of Eq. (9
) is analytically tractable and the second term can be approximated with a Monte-Carlo algorithm. The lower bound can be maximized by using a stochastic gradient descent (SGD).
3 Statistical Speech Enhancement
Based on Combination of VAE and NMF
This section describes the proposed probabilistic generative model called VAE-NMF, that combines a VAE-based speech model and a NMF-based noise model. We formulate the generative process of an observed complex spectrogram by formulating the process of a speech spectrogram and a noise spectrogram . The characteristics of speech and noise signals are represented by their priors based on VAE and NMF, respectively.
3.1 VAE-based speech model
In our speech model we assume a frame-wise -dimensional latent variable . Each time-frame of the latent variable is supposed to represent the characteristics of a speech spectrum such as fundamental frequency, spectral envelope, and type of phoneme. The specific representation of is obtained automatically by conducting the VAE training with a dataset of clean speech spectra. As in the conventional VAEs, we put the standard Gaussian prior on each element of :
Since the speech spectra are primarily characterized by its power spectral density (PSD), it follows a zero-mean complex Gaussian distribution whose variance parameter is formulated with (Fig. 2):
where is a complex Gaussian distribution with mean parameter and variance parameter . is a nonlinear function representing the relationship between and the speech signal . This function is formulated by using a DNN and obtained by the VAE training.
3.2 Generative model of mixture signals
In our Bayesian generative model, the input complex spectrogram is represented as the sum of a speech spectrogram and a noise spectrogram :
We put the VAE-based hierarchical prior model (Eqs. (10) and (11)) on the speech spectrogram . On the other hand, we assume that the PSD of the noise spectrogram is low-rank and put an NMF-based prior model on it. More specifically, the PSD of a noise spectrogram can be represented as the product of spectral basis vectors and their activation vectors . The zero-mean complex Gaussian distribution is put on each TF bin of the noise spectrogram as follows:
For mathematical convenience, we put conjugate prior distributions onand as follows:
is a gamma distribution with the shape parameterand the rate parameter ; , , , and
are hyperparameters that should be set in advance.
By marginalizing out the speech and noise complex spectrograms and , we obtain the following Gaussian likelihood:
Since this likelihood function is independent of the phase term of the input spectrogram , it is equivalent to the following exponential likelihood:
where is the power spectrogram of and
is the exponential distribution with a mean parameter. Maximization of the exponential likelihood on a power spectrogram corresponds to minimization of Itakura-Saito divergence, which is widely used in audio source separation [12, 26].
3.3 Pre-training of VAE-based speech model
The goal of the pre-training of the VAE-based speech model is to find that maximizes the following marginal likelihood from the dataset of clean speech signal (denoted by in this subsection):
As stated in Sec. 2.2, it is difficult to analytically calculate this marginal likelihood. We approximate it by using the Variational mean-field approximation. Let be the variational posterior distribution of . Since is independent from the phase term of the speech spectrogram , the variational posterior is defined by ignoring the phase term as follows:
where is the power spectrum of and and are nonlinear functions representing the mean and variance parameters of the Gaussian distribution. These two functions are defined with DNNs. The marginal likelihood is approximately calculated as follows:
The DNNs for , , and are optimized by using SGD so that this variational lower bound is maximized.
3.4 Bayesian inference of VAE-NMF
To enhance the speech signal in a noisy observed signal, we calculate the full posterior distribution of our model: . Since the true posterior is analytically intractable, we approximate it with a finite number of random samples by using a Markov chain Monte Carlo (MCMC) algorithm . MCMC alternatively and iteratively samples one of the latent variables (, , and ) according to their conditional posterior distributions.
where is the generalized inverse Gaussian distribution and and are auxiliary variables.
The latent variable of speech is updated by using a Metropolis method  because it is hard to analytically derive the conditional posterior . The latent variable is sampled at each time frame by using the following Gaussian proposal distribution whose mean is the previous sample :
where is a variance parameter of the proposal distribution. This candidate is randomly accepted with the following probability:
3.5 Reconstruction of complex speech spectrogram
In this paper we obtain the enhanced speech with Wiener filtering by maximizing the conditional posterior . Let be the speech spectrogram that maximizes the conditional posterior. It is given by the following equation:
We simply use the mean values of the sampled latent variables as , , and in Eq. (26).
4 Experimental Evaluation
This section reports experimental results with noisy speech signals whose noise signals were captured in actual environments.
4.1 Experimental settings
To compare VAE-NMF with a DNN-based supervised method, we used CHiME-3 dataset  and DEMAND noise database111http://parole.loria.fr/DEMAND/. The CHiME-3 dataset was used for both the training and evaluation. The DEMAND database was used for constructing another evaluation dataset for unseen noise conditions. The evaluation with the CHiME-3 was conducted by using its development set, which consists of 410 simulated noisy utterances in each of four different noisy environments: on a bus (BUS), in a cafe (CAF), in a pedestrian area (PED) and on a street junction (STR). The average signal-to-noise ratio (SNR) of the noisy speech signals was 5.8 dB. The evaluation with the DEMAND was conducted by using 20 simulated noisy speech signals in each of four different noisy environments: on a subway (SUB), in a cafe (CAF), at a town square (SQU), and in a living room (LIV). We generated these signals by mixing the clean speech signals of the CHiME-3 development set with the noise signals in the DEMAND database. The SNR of these noisy speech signals was set to be 5.0 dB. The sampling rate of these signals was 16 kHz. The enhancement performance was evaluated by using the signal-to-distortion ratio (SDR) .
To obtain the prior distribution of speech signals , we trained a VAE that had two networks of and as shown in Fig. 3. The dimension of the latent variables was set to be 10. The training data were about 15 hours of clean speech signals in the WSJ-0 corpus 
. Their spectrograms were obtained with a short-time Fourier transform (STFT) with a window length of 1024 samples and a shifting interval of 256 samples. To make the prior distribution robust against a scale of the speech power, we randomly changed the average power of the spectrogram between 0.0 and 10.0 at each parameter update.
The parameters for VAE-NMF were as follows. The number of bases was set to be . The hyperparameters , , , , and were set to be , , and , , and , respectively. The represents the empirical average power of the input noisy spectrogram. After drawing 100 samples for burn-in, we drew 50 samples to estimate the latent variables. These parameters had been determined empirically. The latent variables of noise and were randomly initialized. Since the latent variable of speech depends on the initial state, the initial sample was drawn from by setting the observation as the speech signal .
We compared VAE-NMF with a DNN-based supervised method and the unsupervised RPCA. We implemented a DNN that outputs IRMs (DNN-IRM). It had five hidden layers with ReLU activation functions. It takes as an input 11 frames of noisy 100-channel log-Mel-scale filterbank features and predicts one frame of IRMs222SDRs were evaluated by dropping 2048 samples (5 frames) at both ends.. We trained DNN-IRM with the training dataset of CHiME-3, which was generated by using the WSJ-0 speech utterances and noise signals. The noise signals were recorded in the same environments as those in the evaluated data.
4.2 Experimental results
The enhancement performance is shown in Tables 2 and 2. In the experiments using the CHiME-3 test set (Table 2), DNN-IRM, which was trained using the noisy data recorded in the same environments at the test data, yielded the highest average SDR. The proposed VAE-NMF achieved higher SDRs than RPCA in all conditions and even outperformed the supervised DNN-IRM in BUS condition without any prior training of noise signals. From the results obtained using the test set constructed with the DEMAND noise data, we can see that VAE-NMF outperformed the other methods in all the conditions. The noise data in DEMAND is unknown to DNN-IRM trained using the CHiME-3 training set, and its enhancement performance deteriorated significantly. These results clearly show the robustness of the proposed VAE-NMF against various types of noise conditions.
The SDR performance of VAE-NMF for the CAF condition in the DEMAND test set was lower than those for the other conditions. In this condition, the background noise contained conversational speech. Since VAE-NMF estimates speech component independently at each time frame, the background conversations were enhanced at the time frames where the power of the target speech was relatively small. This problem would be solved by making the VAE-based speech model to maintain time dependencies of a speech signal. The variational recurrent neural network would be useful for this extension.
We presented a semi-supervised speech enhancement method, called VAE-NMF, that involves a probabilistic generative model of speech based on a VAE and that of noise based on NMF. Only the speech model is trained in advance by using a sufficient amount of clean speech. Using the speech model as a prior distribution, we can obtain posterior estimates of clean speech by using an MCMC sampler while adapting the noise model to noisy environments. We experimentally confirmed that VAE-NMF outperformed the conventional supervised DNN-based method in unseen noisy environments.
One interesting future direction is to extend VAE-NMF to the multichannel scenario. Since complicated speech signals and a spatial mixing process can be represented by a VAE and a well-studied phase-aware linear model (e.g., [3, 2, 32]), respectively, it would be effective to integrate these models in a unified probabilistic framework. We also investigate GAN-based training of the speech model to accurately learn a probability distribution of speech.
-  X. Lu, Y. Tsao, S. Matsuda, and C. Hori, “Speech enhancement based on deep denoising autoencoder,” in Interspeech, 2013, pp. 436–440.
-  J. Heymann, L. Drude, and R. Haeb-Umbach, “Neural network based spectral mask estimation for acoustic beamforming,” in IEEE International Conference on Acoustics, Speech and Signal Processing, 2016, pp. 196–200.
-  A. A. Nugraha, A. Liutkus, and E. Vincent, “Multichannel audio source separation with deep neural networks,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 24, no. 9, pp. 1652–1664, 2016.
-  A. Narayanan and D. Wang, “Ideal ratio mask estimation using deep neural networks for robust speech recognition,” in IEEE International Conference on Acoustics, Speech and Signal Processing, 2013, pp. 7092–7096.
-  S. Pascual, A. Bonafonte, and J. Serrà, “SEGAN: Speech enhancement generative adversarial network,” Interspeech, pp. 3642–3646, 2017.
-  Z.-Q. Wang and D. Wang, “Recurrent deep stacking networks for supervised speech separation,” in IEEE International Conference on Acoustics, Speech and Signal Processing, 2017, pp. 71–75.
-  Y. Ephraim and D. Malah, “Speech enhancement using a minimum-mean square error short-time spectral amplitude estimator,” IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 32, no. 6, pp. 1109–1121, 1984.
-  P. C. Loizou, Speech enhancement: theory and practice. CRC press, 2013.
-  S. Mohammed and I. Tashev, “A statistical approach to semi-supervised speech enhancement with low-order non-negative matrix factorization,” in IEEE International Conference on Acoustics, Speech and Signal Processing, 2017, pp. 546–550.
-  P. Smaragdis, B. Raj, and M. Shashanka, “Supervised and semi-supervised separation of sounds from single-channel mixtures,” Independent Component Analysis and Signal Separation, pp. 414–421, 2007.
-  M. Sun, Y. Li, J. F. Gemmeke, and X. Zhang, “Speech enhancement under low SNR conditions via noise estimation using sparse and low-rank NMF with Kullback–Leibler divergence,” IEEE/ACM Transactions on Audio, Speech and Language Processing, vol. 23, no. 7, pp. 1233–1242, 2015.
-  C. Févotte, N. Bertin, and J. Durrieu, “Nonnegative matrix factorization with the Itakura-Saito divergence: With application to music analysis,” Neural computation, vol. 21, no. 3, pp. 793–830, 2009.
-  C. Sun, Q. Zhang, J. Wang, and J. Xie, “Noise reduction based on robust principal component analysis,” Journal of Computational Information Systems, vol. 10, no. 10, pp. 4403–4410, 2014.
-  P. S. Huang, S. D. Chen, P. Smaragdis, and M. Hasegawa-Johnson, “Singing-voice separation from monaural recordings using robust principal component analysis,” in IEEE International Conference on Acoustics, Speech and Signal Processing, 2012, pp. 57–60.
-  I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in neural information processing systems, 2014, pp. 2672–2680.
-  D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114, 2013.
-  O. Fabius and J. R. van Amersfoort, “Variational recurrent auto-encoders,” arXiv preprint arXiv:1412.6581, 2014.
-  W.-N. Hsu, Y. Zhang, and J. Glass, “Learning latent representations for speech generation and transformation,” in Interspeech, 2017, pp. 1273–1277.
-  M. Blaauw and J. Bonada, “Modeling and transforming speech using variational autoencoders,” in Interspeech, 2016, pp. 1770–1774.
-  T. G. Kang, K. Kwon, J. W. Shin, and N. S. Kim, “NMF-based target source separation using deep neural network,” IEEE Signal Processing Letters, vol. 22, no. 2, pp. 229–233, 2015.
-  T. T. Vu, B. Bigot, and E. S. Chng, “Combining non-negative matrix factorization and deep neural networks for speech enhancement and automatic speech recognition,” in IEEE International Conference on Acoustics, Speech and Signal Processing, 2016, pp. 499–503.
-  K. Qian, Y. Zhang, S. Chang, X. Yang, D. Florêncio, and M. Hasegawa-Johnson, “Speech enhancement using bayesian wavenet,” Interspeech, pp. 2013–2017, 2017.
-  E. M. Grais, M. U. Sen, and H. Erdogan, “Deep neural networks for single channel source separation,” in IEEE International Conference on Acoustics, Speech and Signal Processing, 2014, pp. 3734–3738.
-  M. Sun, X. Zhang, and T. F. Zheng, “Unseen noise estimation using separable deep auto encoder for speech enhancement,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 24, no. 1, pp. 93–104, 2016.
-  D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
-  A. T. Cemgil, “Bayesian inference for nonnegative matrix factorisation models,” Computational Intelligence and Neuroscience, vol. 2009, no. 785152, pp. 1–17, 2009.
-  C. M. Bishop, Pattern recognition and machine learning. Springer, 2006.
-  J. Barker, R. Marxer, E. Vincent, and S. Watanabe, “The third CHiME speech separation and recognition challenge: Dataset, task and baselines,” in IEEE Workshop on Automatic Speech Recognition and Understanding, 2015, pp. 504–511.
-  E. Vincent, R. Gribonval, and C. Févotte, “Performance measurement in blind audio source separation,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 14, no. 4, pp. 1462–1469, 2006.
-  J. Garofalo, D. Graff, D. Paul, and D. Pallett, “CSR-I (WSJ0) complete,” Linguistic Data Consortium, Philadelphia, 2007.
-  J. Chung, K. Kastner, L. Dinh, K. Goel, A. C. Courville, and Y. Bengio, “A recurrent latent variable model for sequential data,” in Advances in neural information processing systems, 2015, pp. 2980–2988.
-  A. Ozerov and C. Févotte, “Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 18, no. 3, pp. 550–563, 2010.