Log In Sign Up

Improving Embedding Extraction for Speaker Verification with Ladder Network

by   Fei Tao, et al.

Speaker verification is an established yet challenging task in speech processing and a very vibrant research area. Recent speaker verification (SV) systems rely on deep neural networks to extract high-level embeddings which are able to characterize the users' voices. Most of the studies have investigated on improving the discriminability of the networks to extract better embeddings for performances improvement. However, only few research focus on improving the generalization. In this paper, we propose to apply the ladder network framework in the SV systems, which combines the supervised and unsupervised learning fashions. The ladder network can make the system to have better high-level embedding by balancing the trade-off to keep/discard as much useful/useless information as possible. We evaluated the framework on two state-of-the-art SV systems, d-vector and x-vector, which can be used for different use cases. The experiments showed that the proposed approach relatively improved the performance by 10


page 1

page 2

page 3

page 4


Raw-x-vector: Multi-scale Time Domain Speaker Embedding Network

State-of-the-art text-independent speaker verification systems typically...

Deep Neural Network Embedding Learning with High-Order Statistics for Text-Independent Speaker Verification

The x-vector based deep neural network (DNN) embedding systems have demo...

Speaker Diarization with LSTM

For many years, i-vector based speaker embedding techniques were the dom...

Disentangled speaker and nuisance attribute embedding for robust speaker verification

Over the recent years, various deep learning-based embedding methods hav...

Improving Target Speaker Extraction with Sparse LDA-transformed Speaker Embeddings

As a practical alternative of speech separation, target speaker extracti...

Magnitude-aware Probabilistic Speaker Embeddings

Recently, hyperspherical embeddings have established themselves as a dom...

1 Introduction

Speaker verification (SV) aims to verify a speaker using the speech signal. As an open set recognition problem, speaker verification performs the task where testing labels are not seen during training, and most probably, there is large mismatch between testing and training data (different environments, languages, and devices) 

[1, 2, 3, 4, 5, 6, 7]. A standard pipeline is extracting a speaker embedding relying on some trained model and comparing the similarity between the registered and testing embeddings. One key problem in this task is extracting a good embedding which can capture characteristics of the input voice to discriminate speakers. Recently, the deep neural network (DNN) approaches are widely used since they were proved to be able to extract better high-level embeddings [4, 5, 6, 7]. The DNN approaches don’t have presumption of the data distribution, in contrast, they can learn the data distribution in a data-driven fashion in manifold space [8, 9, 10].

Research has also been done to improve the capability of DNN to output better embeddings. Previous work has either focused on improving the capability of modeling the hidden representation from the input, e.g.

convolutional neural network (CNN), time-delay neural network

(TDNN), or focused on designing more discriminative loss functions for classification task during the neural networking training, e.g.

siamese neural network

(SNN) and triplet loss. Only few efforts have investigated training strategies to maintain more original information in inputs. This is important, because supervised learning essentially discards input information regarding the targeting task within the DNN framework. We don’t know what information will affect the performance on testing data during training in the open set recognition problem, like SV, it is necessary to keep input information as much as possible when we train the embedding extraction model in supervised learning fashion. In this paper, we propose to apply ladder network framework to the SV system training. Ladder networks combine supervised and unsupervised learning fashions together. On one hand, supervised learning makes the network discard useless information regarding the classification task (capture the speaker’s characteristics); on the other hand, unsupervised learning makes the network keep useful information regarding the reconstruction task (improve the system generalization). Ladder network performs these two learning fashions simultaneously during training, so it guarantees to keep balance in the trade-off of keeping and discarding information.

The remaining of the paper is organized in this way: Section 2 reviews previous related work; Section 3 describes the details of our proposed SV system; Section 4.1 introduces the corpus we used and the features we extracted in this study; Section 4 introduce our experiments and the results; at last, we will draw conclusions in Section 5.

2 Related Work

SV systems involve parts of speaker embedding extraction and classification. For classification, vector similarity [4] or Probabilistic linear discriminant analysis (PLDA) [11]

is adopted as embedding classifier for verifying whether they are from same speaker or not. Choice of speaker embedding becomes the key problem mainly affecting the performance in the SV systems. Recently,

deep learning (DL) techniques have been emerging as a powerful model which is able to model complicated distribution in manifold space [8, 12]. Different types of DNN have been proposed for different purposes and proved to be successful in many areas [13, 14, 15]. The key advantage of the DNN is that it does not have any assumption on the input distribution, and model the input data in a data-driven fashion [16, 17].

Snyder et al. used a TDNN to model the input acoustic feature [5]. They did statistics over a fixed duration to capture the temporal information and represent the speaker’s embedding (named as x-vector), while Nagrani et al. [6] proposed to use CNN to capture high-level embedding from acoustic features. Chen and Salman also proposed a system which process frame by frame under SNN framework to improve the discriminability of DNN [18]. Varani et al. [4] adopted a small footprint DNN taking contextual input and outputting speaker ID. The embedding (named as d-vector) extracted before the classification layer was used for verification. The x-vector is the state-of-the-art speaker verification system, which was designed to deal with long duration input and normally deployed on server, while the d-vector is a light weight system, which was designed to deal with short duration input (shorter than 1 sec) and can be deployed on portable devices.

Most of the previous work were focusing on either better network structure or more discriminative loss for extracting better embedding. Recently, Li et al. [19] proposed a end-to-end fashion DNN-based SV system, which takes registered and testing audios as inputs and directly output whether the testing audio belongs to the registered speaker. Though the system does not have embedding extraction phase, the study is still following the idea that improve the network architecture for better performance.

Only few studies have been done on changing training strategy to maintain more original information in inputs. Previously, Denoising autoencoder (DAE) was used to pretrain DNN by reconstructing the clean version of the signal from the noisy version of the signal [8, 20]. DAE can keep most essential and robust information in the input in unsupervised learning fashion [21]. After pretraining, DNN was trained in supervised learning fashion. The trained DNN tends to discard some information in inputs, which the network learned to be useless with respect to the targeting task [22]. Therefore, the final system will bias to the training domain, which is not wanted in SV systems since it is open set recognition problem, even though DAE is used for pre-training. An ideal embedding extractor should discard information regarding the loss for speaker classification task, meanwhile it can keep those original information as much as possible, which does not affect the speaker classification task loss. Valpola [22] proposed ladder network framework to incorporate DAE training approach to DNN training at each layer during supervised learning. The parameters were updated from the cost of DAE reconstruction and targeting classification simultaneously [23].

3 Proposed Approach

3.1 Speaker Verification with DNN

We adopted d-vector and x-vector frameworks as our baseline in this paper, considering large scale use case and light weight use case.

The d-vector framework [4] consisted of 4 fully connected (FC) rectified linear units

(ReLUs) layers, with 512 neurons per layer; and 1 softmax layer on top of the FC layers. It is suitable to tackle with audios of short duration. We concatenate the historical and future feature frames to the current frame to construct the contextual input. We set 51 frames as the input window considering that it should be long enough to contain sufficient temporal information, and short enough to fit the application like wakeword detection. To perform quick training, the input window moves in non-overlap fashion. On top of the network, a classification layer is employed to perform

speaker identity (SID) task. The label is given corresponding to each input central frame (the current frame mentioned above).

The x-vector framework [5] was a TDNN consisting of 9 layers. The first 5 layers were time-delay layer at frame level. The 6th layer was the statistics pooling layer, which aggregated the frame-level inputs and compute the statistics over segments. The 2 layers following the statistics pooling layer were fully connected layer, which were segment-level layer, and the last layer was the softmax layer for SID classification task. We adopted the parameters setting in [5] and skip the description for simplicity. The x-vector is the state-of-the-art framework for speaker verification, which is suitable for audios of long duration. It can have better performance than d-vector since it will model the temporal information from the statistics over the input segment.

We extracted the sequence of embedding from aforementioned networks, which were trained for SID. Averaging the embedding over the utterance form the final utterance-based speaker embedding. Length normalization over the embedding is then performed. For the d-vector framework, we compute cosine similarity between the registered and testing embedding, while we use PLDA as backend to compute similarity for x-vector framework. The threshold of the computed similarity can be determined by finding the

equal error rate (EER) point.

3.2 Ladder Network

The ladder network is employed with both of the d-vector and x-vector frameworks. It is illustrated in Figure 1, which consists of classification network and corresponding DAE.

Figure 1: The diagram of the ladder network framework. The dashed line represent passing the hidden value at layer from encoder to decoder for denoising process.

The classification network for classification task is a normal network (e.g. DNN for d-vector and TDNN for x-vector). The DAE for reconstruction task has two parts, encoder and decoder. The encoder shares the same parameters with the classification network, while the decoder tries to reconstruct the clean version of the noised input. Unlike conventional DAE, the ladder network adds Gaussian noise and performs denoising at each layer. Equation 1 shows how to add the noise, where represents the noisy version of the hidden value, is the layer, and N is the Gaussian noise with zero mean and standard deviation (we set in this study). Equation 2 shows the denoising process relying on parameters of 3 and 4, where the 10 parameters ( to ) are learnable parameters,

is batch normalized preactivation value propagated from

layer in decoder, and is the reconstructed hidden value at the layer in decoder. The denoising process performs reconstruction depending on two things, one is the batch normalized preactivation value from higher layer in decoder, and the other is the noisy version of the hidden value from the counterpart at the current layer in encoder.


The cost function for reconstruction is defined in Equation 5, equal to the weighted summation of the square error between the reconstructed and original hidden values. The is a scalar weighting the reconstruction loss at layer. In this study, we set to 1000, to 10 and to 0.1. It can be noticed that if we set , the autoencoder is equivalent to a conventional DAE.


During training, the network will be updated with respect to the loss, which is the summation of the cross entropy loss of the SID task and the denoising cost. The proposed approach can therefore train a network distinguishing speakers’ acoustic characteristics and maintain (and denoise) original input information as much as possible at each layer.

For d-vector framework, the classification network has 4 FC layers and 1 softmax layers. The encoder shares the weights of the FC layers with the classification network. The decoder has the mirrored architecture of the encoder. The proposed architecture (d-vector with ladder network) is plotted in Figure 2. For x-vector framework, the classification network has 9 layers (5 frame-level time-delay layers, 1 statistics pooling layer, 2 FC layers and 1 softmax layer). The encoder only shares the weights of the first 5 frame-level layers (all the layers before the statistics pooling layer), while the decoder has the mirrored architecture of the encoder. We did not do the reconstruction on the layers after the statistics layer because the statistics layer breaks the frame details and makes it impossible to reconstruct the original input segment. The proposed architecture (x-vector with ladder network) is plotted in Figure 2.

After training, we only use the classification network for embedding extractor. For d-vector, we extracted the hidden value from the layer right before the softmax layer. For x-vector, we extracted the hidden value from the first segment-level layer, which was the first FC layer right after the statistics pooling layer. This proposed approach did not introduce parameters compared with the baselines.

Figure 2: (a) Diagram of the d-vector system with the ladder network. We name it as “d-ladder”. (b) Diagram of the x-vector system with the ladder network. The ladder framework is only applied on the frame-level layers. We name it as “x-ladder”. The dashed line between the DAE (denoising autoencoder) and the classification network (d-vector and x-vector in this case) represents the denoising cost. We did not plot the complete neural network for simplicity.

4 Experiments and Results

4.1 Data and Features

Though we targeted at two practical use cases, we used public corpus for training and testing for fair comparison. We used VoxCeleb corpus (including Phase 1 and 2) [6, 7], which is an audio-visual dataset consisting of short clips of human speech downloaded from YouTube. The human speech was mainly made during interviews, so the scenarios were natural and the background was not controlled. It includes 7363 speakers in total, where 39% are females and 61% are males. Each speaker spoke several segments of speeches. Duration of each segment varies from 3 to 20 seconds. The total duration is longer than 2000 hours. The corpus was originally separated into two sets ”develop” and ”test” in both phases. We used all the data from Phase 2 and the ”develop” set from Phase 1 to train the model, and used ”test” set from Phase 1 to evaluate the system, where there was 40 speakers. In this paper we focus on the speech based SV system, so we only used the speech part and ignored the videos.

We replicated the feature preparation described in [4] for d-vector framework and [5]

for x-vector framework. A minor modification is made for d-vector is the input window was set to 51 frames, including 25 historic and 25 future frames. All features extraction and

voice activity detection (VAD) were performed by Kaldi [24].

4.2 Experiment Settings

For baselines, we used aforementioned d-vector [4] and x-vector [5] frameworks without ladder network, and name them as “d-vector” and “x-vector” in the following. We used different training approaches for the two approaches. For d-vector training, the optimizer was Adam [25]

. Each network was trained by running 15 epochs. After the first 5 epochs, the learning rate was halved every 2 epochs. For x-vector training, we replicated the Kaldi recipe in Tensorflow

[26]. The training ran for 15 epochs so the results were comparable to the ones of the “d-vector” systems. For the proposed approach, we use the same settings and training approaches, except we name them as “d-ladder” and “x-ladder” correspondingly. Across all the experiments, we did not use data augmentation. For final testing with the ladder networks, we only use the classification networks so that all the parameters number are the same.

4.3 Experiment Results

Table 1 is the summary of the EER from the experiments, including the two baselines (“x-vector” and “d-vector”) and the proposed approach deployed on the baselines (“x-ladder” and “d-ladder”).

Approach EER [%]
d-vector 20.1
d-ladder 18.1
x-vector 3.86
x-ladder 3.78
Table 1: Summary of the experiment result. The “EER” is the equal-error-rate. The font in bold stands for the best performance in corresponding categories.

The results show that the “x-vector” framework can significantly outperform the “d-vector” framework. The gain was mainly from the long-term information on the segment level, because “d-vector” only took around 0.5 sec input, while “x-vector” was modeled on the segments longer than 2 secs.

The proposed approach can outperform the baseline in both cases. The “d-ladder” outperformed the “d-vector” by 10% relative difference, while the “x-ladder” outperformed the “x-vector” by 2% relative difference. The improvement in the “x-vector” framework was smaller, because the baseline, “x-vector” has already been well developed. There was limited space for improvement. It should be noticed that the improvement was from no extra data, no augmentation and no extra parameters.

The ladder network looks similar to multi-task learning with the secondary task of reconstructing the original input. To verify the capability of the ladder network further, we compared the proposed framework with the multi-task learning approach. The evaluating result is shown in Table 2, where the “x-multi” approach stands for the x-vector with multi-task learning. We performed the comparison under the x-vector framework, because this was the framework with the best performance in our study. The x-multi consisted of two tasks, speaker identification and original input reconstruction. The x-multi architecture was similar to the “x-ladder” (shown in Figure 2). The speaker identification task was performed on top the of the x-vector network, while the original input reconstruction was performed on the frame-level layers. We built a DAE for the frame-level layers, and kept its parameters number same as the DAE’s in x-ladder. The results show that the “x-ladder” can outperformed the x-multi, indicating the capability of the proposed approach. It should be noticed that the x-multi had worse performance than the x-vector, which means brutally adding reconstruction task to the network for speaker identification may not be helpful. It can be explained that the regularization from the reconstruction task was not imposed much on to the frame-level layers, because the error back-propagated from the reconstruction task may mainly apply on the tuning of the decoding side in DAE (the mirrored part of the frame-level layers). In contrast, the ladder network forced each layer in the classification network to be tuned regarding with the reconstruction errors, which was a strong regularization.

We also observe that the training process for the ladder network took longer. The training time of the “d-vector” system was about 40 hrs on the Nvidia RTX 2080 ti, while the “d-ladder” system took 45 hrs. The training time of the “x-vector” system was about 106 hrs, while the “x-ladder” system took 180 hrs. We analyzed the training process for the “x-vector” framework in details (because it was the best system in our study). We evaluated the performance from the intermediate models (models per 3 epochs) during training and plot the results in Figure 3. It can be seen that the performance curve of the “x-vector” system vibrated along the epochs, indicating it had over-fitting problem. But the curve of the “x-ladder” system was more stable and always went down. It implies the system had better generalization, since the results did not show any clue of over-fitting. We did not continue the training process, because we set 15 epochs as a global limit for all systems’ training. We could expect better performance when the training continued. Future work is needed to investigate how much improvement the “x-ladder” system can achieve. This proved that the ladder network can improve the system generalization in the speaker verification task.

Figure 3: The EER (evaluated on testing data) every 3 epochs from the intermediate models in the training of the “x-vector” and “x-ladder” systems. The performance at the 15th epoch was reported as the final result.
Approach EER [%]
x-vector 3.86
x-multi 4.25
x-ladder 3.78
Table 2: Comparison between ladder network and multi-task learning. “x-multi” stands for x-vector framework with multi-task learning; “x-ladder” stands for x-vector framework with the ladder network approach.

5 Conclusions

In this paper, we propose to apply a new approach to train the neural network for extracting better speaker embeddings. The proposed approach consisted of a modified DAE and a feed forward classification network. During training, the supervised and unsupervised learning were both applied to balance keeping and discarding information. The unsupervised learning helped the network not to bias to the task in the training data domain, therefore it is especially suitable for the speaker verification problem which is an open set problem. We evaluate our approach on Voxceleb corpus with d-vector and x-vector frameworks, which are the state-of-the-art approaches respectively tackling with short and long duration audios. The results showed that our approach can have at most 10% relative improvement without data augmentation and extra parameters, which proves the advantage of the proposed approach. We also further analyzed the ladder network under the x-vector framework. The analysis indicates the ladder network can help regularize and generalize the speaker identification network.


  • [1] N. Dehak, P. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, “Front-end factor analysis for speaker verification,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no. 4, pp. 788–798, May 2011.
  • [2] G. Liu, Q. Qian, Z. Wang, Q. Zhao, T. Wang, H. Li, J. Xue, S. Zhu, R. Jin, and T. Zhao, “The opensesame nist 2016 speaker recognition evaluation system.” in INTERSPEECH, Stockholm, Sweden, August 2017, pp. 2854–2858.
  • [3] D. Garcia-Romero and C. Espy-Wilson, “Analysis of i-vector length normalization in speaker recognition systems,” in Interspeech 2011, Florence, Italy, August 2011, pp. 249–252.
  • [4] E. Variani, X. Lei, E. McDermott, I. L. Moreno, and J. Gonzalez-Dominguez, “Deep neural networks for small footprint text-dependent speaker verification,” in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   Florence, Italy: IEEE, May 2014, pp. 4052–4056.
  • [5] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust dnn embeddings for speaker recognition,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   Calgary, Canada: IEEE, April 2018, pp. 5329–5333.
  • [6] A. Nagrani, J. S. Chung, and A. Zisserman, “Voxceleb: a large-scale speaker identification dataset,” in INTERSPEECH, Stockholm, Sweden, August 2017.
  • [7] J. S. Chung, A. Nagrani, and A. Zisserman, “Voxceleb2: Deep speaker recognition,” in INTERSPEECH, Hyderabad, India, September 2018.
  • [8] G. Hinton, S. Osindero, and Y. Teh, “A fast learning algorithm for deep belief nets,” Neural Computation, vol. 18, no. 7, pp. 1527–1554, July 2006.
  • [9] G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury, “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,” IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82–97, November 2012.
  • [10]

    F. Tao and C. Busso, “Gating neural network for large vocabulary audiovisual speech recognition,”

    IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 7, pp. 1286–1298, July 2018.
  • [11] S. Ioffe, “Probabilistic linear discriminant analysis,” in

    European Conference on Computer Vision

    .   Springer, 2006, pp. 531–542.
  • [12] Y. Bengio, “Learning deep architectures for AI,”

    Foundations and trends® in Machine Learning

    , vol. 2, no. 1, pp. 1–127, January 2009.
  • [13] F. Tao and C. Busso, “End-to-end audiovisual speech activity detection with bimodal recurrent neural models,” ArXiv e-prints (arXiv:1809.04553), pp. 1–11, September 2018.
  • [14] Y. LeCun and Y. Bengio, “Convolutional networks for images, speech, and time series,” in The handbook of brain theory and neural networks, M. Arbib, Ed.   The MIT Press, July 1998, pp. 255–258.
  • [15] X. Yu, J. Huang, S. Zhang, W. Yan, and D. N. Metaxas, “Pose-free facial landmark fitting via optimized part mixtures and cascaded deformable shape model,” in IEEE International Conference on Computer Vision (ICCV 2013), Sydney, NSW, Australia, December 2013, pp. 1944–1951.
  • [16] F. Tao and C. Busso, “Aligning audiovisual features for audiovisual speech recognition,” in IEEE International Conference on Multimedia and Expo (ICME 2018), San Diego, CA, USA, July 2018, pp. 1–6.
  • [17]

    ——, “Bimodal recurrent neural network for audiovisual voice activity detection,” in

    Interspeech 2017, Stockholm, Sweden, August 2017, pp. 1938–1942.
  • [18] K. Chen and A. Salman, “Extracting speaker-specific information with a regularized siamese deep network,” in Advances in Neural Information Processing Systems, Granada, Spain, December 2011, pp. 298–306.
  • [19] L. Wan, Q. Wang, A. Papir, and I. L. Moreno, “Generalized end-to-end loss for speaker verification,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   Calgary, Canada: IEEE, April 2018, pp. 4879–4883.
  • [20] Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, “Greedy layer-wise training of deep networks,” in Advances in neural information processing systems, Vancouver, Canada, December 2007, pp. 153–160.
  • [21] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, “Extracting and composing robust features with denoising autoencoders,” in Proceedings of the 25th international conference on Machine learning.   Helsinki, Finland: ACM, July 2008, pp. 1096–1103.
  • [22] H. Valpola, “From neural pca to deep unsupervised learning,” in

    Advances in Independent Component Analysis and Learning Machines

    .   Lille, France: Elsevier, July 2015, pp. 143–171.
  • [23]

    A. Rasmus, M. Berglund, M. Honkala, H. Valpola, and T. Raiko, “Semi-supervised learning with ladder networks,” in

    Advances in neural information processing systems, Monteral, Canada, December 2015, pp. 3546–3554.
  • [24] D. Povey, A. Ghosha, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, and J. Silovsky, “The Kaldi speech recognition toolkit,” in

    IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU)

    , Waikoloa, HI, USA, December 2011.
  • [25] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations, San Diego, CA, USA, May 2015, pp. 1–13.
  • [26] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard et al., “Tensorflow: A system for large-scale machine learning,” in 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), 2016, pp. 265–283.