Deep CNN based feature extractor for text-prompted speaker recognition

03/13/2018 ∙ by Sergey Novoselov, et al. ∙ Speech Technology Center 0

Deep learning is still not a very common tool in speaker verification field. We study deep convolutional neural network performance in the text-prompted speaker verification task. The prompted passphrase is segmented into word states - i.e. digits -to test each digit utterance separately. We train a single high-level feature extractor for all states and use cosine similarity metric for scoring. The key feature of our network is the Max-Feature-Map activation function, which acts as an embedded feature selector. By using multitask learning scheme to train the high-level feature extractor we were able to surpass the classic baseline systems in terms of quality and achieved impressive results for such a novice approach, getting 2.85 evaluation set. Fusion of the proposed and the baseline systems improves this result.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

I-vector-based systems are well known to be state-of-the-art solutions to the text-independent speaker verification problem

[1, 2, 3]. Nonetheless, this problem is gradually gaining attention from the deep learning perspective. Particularly, studies [2, 4] make use of the ASR deep neural network (ASR DNN) in order to divide acoustic space into senone classes, and the classic total variability (TV) model is applied to discriminate between speakers in that space afterwards [1].

In such phonetic discriminative DNN-based systems two major techniques can be distinguished. The first one uses DNN posteriors to calculate Baum-Welch statistics, and the second one uses bottleneck features in pair with speaker specific features (MFCC) for a full TV-UBM system training.

Recent publications [5, 6, 7, 8] suggest that substantial advancement of the state-of-the-art text-dependent verification systems is mainly based on progress in the text-independent speaker recognition task. Therefore, successful application of a phonetic discriminative DNN to the latter task justifies employment of a similar approach for text-dependent systems [9, 10, 11].

On the other hand, direct speaker discrimination is the most natural way of speaker verification. There are several profound studies on advantageous usage of deep end-to-end solutions for discriminating speakers directly in a text-dependent task [12, 13]. Paper [14] describes a DNN that extracts a small speaker footprint that is used to discriminate between speakers. Paper [15] presents a well performing implementation of a DNN extractor based on the speaker discriminative approach in the text-independent task.

This paper focuses on the text-prompted speaker verification scenario. It does not require user to remember a specific passphrase and reduces risk of replay attacks on the system. Previously, we have shown that by segmenting a passphrase into word states prior to supervector extraction we were able to construct more accurate statistical models of speech signals [7, 8]. The present study continues with this approach and suggest ways of further improvement of deep speaker verification systems.

This work exploits the deep convolutional neural network (CNN) with Max-Feature-Map activation function (MFM) from [16, 17], which is based on maxout activation function, for direct speaker discrimination in the text-dependent setting. We train a deep high-level feature extractor for prompted passwords. The experiments were mainly conducted on Part 3 of the RSR2015 database [18], which contains series of randomized digit sequences representing passwords. The results of the proposed and the baseline systems are compared in section 5.

In addition, we analyze performance of the proposed system on the training data extended with English (Wells Fargo Bank) and Russian (STC-Russian-digits) digits subcorpora.

2 Baseline systems

This section briefly describes three state-of-the-art text-dependent speaker verification systems, which are further referred as baseline systems. To do passphrase segmentation, we use hidden Markov model (HMM) based Viterbi alignment. The frontend of the baseline systems computes mel-frequency cepstral coefficients (MFCC) as well as their first and second time derivatives to yield a 39-dimensional vector per frame. Framing is done every 8 ms with a 16 ms window. A gaussian mixture model (GMM) based voice activity detector is used to find and remove non-speech frames. We also apply cepstral mean subtraction (CMS) and do not apply feature warping to cepstral coefficients.

2.1 Gmm-Svm

The baseline GMM-SVM system is implemented in accordance to [6]. The GMM mean supervector is obtained from a relevant maximum a posteriori (MAP) adaptation of the speaker-independent universal background model (UBM) of a passphrase. No segmentation is done by this system. We trained a passphrase UBM on the RSR2015

database development set. To compensate for inter-speaker variability, Nuisance Attribute Projection (NAP) is applied. Support Vector Machine (SVM) is used as a backend classifier. Finally, score s-normalization is done.

2.2 State-GMM-SVM

In this system HMM segmentation is used to split a passphrase into individual digits. For each digit a State-GMM mean supervector is extracted as described in [7]. Each state is associated with a unique speaker-independent UBM, which is trained on the RSR2015 database training set. The speaker-dependent GMM means of a state are obtained through MAP adaptation of the UBM means. The speaker-dependent GMM mean supervector is obtained with concatenation of the speaker state mean vectors over all states. NAP, SVM and score s-normalization are also utilized in this system.

We refer to this system as the StGMM-SVM system.

2.3 State-Plda

We have previously shown effectiveness of the State probabilistic linear discriminant analysis (State-PLDA) model when addressing the text-dependent speaker verification problem [8].

Log-likelihood ratio score for each digit in a tested utterance can be obtained with PLDA scoring procedure. The final score for the utterance is calculated as a sum of the llr scores over its states [8]. No score normalization is needed.

This system is called StPLDA.

3 State CNN

Encouraged by the success of our deep learning model in the replay attack spoofing detection task [17], we endeavour to improve over the state-of-the-art techniques for text-prompted speaker verification by employing a similar network architecture.

3.1 Features

Our CNN-based system also operates with distinct states, and the same HMM-based segmentation is used to split passphrase into separate digits. We use log mel power spectra extracted from the speech signal as input features. While the feature size along the frequency domain axis is fixed to bands, varying utterance time span must be adjusted to a fixed length of

frames, which is an estimation of the longest digit pronouncement time. This is done either by cropping the end of features along the time axis or wrap padding as it was done in


. Cepstral mean and variance normalization is done for each digit.

3.2 MFM CNN architecture

We make use of the Light CNN architecture [16]. MFM activation function is applied to the feature maps inferred from a convolutional layer:



being an MFM layer input tensor of shape

and its output tensor of shape . Here , indicates the frequency and time domains and is the channel index.

The network itself is composed of several layer groups, each of them is represented by a convolutional layer followed by an MFM layer and optionally a pooling layer. These groups are stacked together and followed by dense layers to produce embeddings for input log mel power spectra features. Details are covered in Table 2.

The choice of the MFM activation function is motivated by the fact that it not only provides ReLU-like activation with a varying threshold but is also an embedded feature selector


3.3 Training

There are two ways of training a deep neural network extractor in the text-prompted scenario. The first one is to train the network to discriminate between speakers (single task mode) and the second, more complex one, is to train the network to discriminate between speakers and digits simultaneously (multitask mode). The latter approach is used in [19]. We also exploit the multitask setup and assign an individual class to a speaker pronouncing a particular digit. Because of this total amount of classes increases by a factor of 10, i.e.

. The network is trained in multitask mode with multiclass cross entropy loss function.

3.4 Scoring

After training, the last fully-connected layer with its softmax activation is removed from the network in order to obtain an extractor of high-level representations for speaker pronounced digits. At the evaluation step embeddings for each individual digit utterance are obtained through the network forward pass. Cosine similarity metric is used afterwards to compare them with corresponding targets. Each test digit embedding is compared to the same digit embedding from the enrollment set, which is averaged over 3 enrollment sessions. Finally, average score of all digits in the passphrase is returned. Note that we do not use any discriminant analysis backend instead of cosine similarity.

4 Experimental setup

4.1 Databases description

We have conducted experiments on Part 3 of the RSR2015 database [18] and on the evaluation part of the STC-Russian-digits database (STCRus). These datasets contain prompted passphrases for speaker verification.

The training set is also extended with Wells Fargo Bank dataset (WF), described in [7], which contains short digit passphrase utterances, and the training part of STC-Russian-digits dataset.

STC-Russian-digits is a newly collected database. It contains utterances of random Russian digit sequences pronounced by native speakers. Each speaker has recorded 1 to 3 sessions, each of them consists of microphone records made with 3 different devices: an Android-based mobile phone, an iOS-based mobile phone, and a web-camera. Each record includes 5 pronunciations of 10 Russian digits in random order. STC-Russian-digits database is split into training and evaluation parts according to Table 1.

Set Males Females Total speakers Total records
Train 473 221 694 15 k
Eval 50 42 92 5 k
Table 1: STC-Russian-digits database

4.2 CNN parameters

Network parameters are described in Table 2

. The last dense layer is included only at the training stage. The number of its neurons corresponds to the number of classes in a particular setup, described in 



Filter / Stride

Output #Params
Conv1 / 6.4K
MaxPool1 /
Conv2a / 8.3K
Conv2b / 153.8K
MaxPool2 /
Conv3a / 18.6K
Conv3b / 614.7K
MaxPool3 /
Conv4a / 33K
Conv4b / 147.6K
Conv5a / 8.3K
Conv5b / 73.9K
MaxPool4 /
FC1 3147.8K
4366K +
Table 2: CNN architecture

The learning rate is decayed by a constant every 10 epochs. Batch size is set to 32.

5 Results and discussion

Tables 3, 4 and 5 present equal error rate (EER) and minimum detection cost function (minDCF) 111, and for various systems. The best performance among baseline systems is demonstrated by StGMM-SVM, yielding 3.11% EER for pooled male and female conditions. The speaker discriminative deep CNN trained in the single task mode () could not surpass baseline results, achieving only 7.83% EER. However, it showed significant performance improvement in multitask mode () with EER decreased to 5.12%. In contrast to GMM-SVM and StGMM-SVM, no score normalization is needed for StPLDA and StCNN based systems.

Remarkably, when new language corpora with foreign classes are presented in the training set, the system learns to discriminate embeddings between speakers for both languages, based on common speaker variability (see Table 4). Two bottom lines in Table 3 show that the extending of the training set for the CNN-based system leads to performance boost. The best performing system [+WF+STCRus] we have obtained had been trained on the pooled English (RSR2015 and WF) and Russian (STC-Russian-digits) subcorpora. Unlike the baseline systems, which benefit from training on in-domain data only, the deep CNN-based system improves from new out-of-domain data added during the learning phase in the multitask setup. Figure 1 illustrates embedding discriminative capability of the system: 5 randomly chosen speaker embeddings are projected on two principal axes with t-SNE [20].

System Male Female Pooled
4.01 (17,91) 2.58 (14.69) 3.37 (16.45)
3.65 (16.29) 2.13 (1.1.06) 3.11 (14.23)
3.37 (17.42) 3.04 (16.1) 4.44 (21.94)
7.06 (37.13) 8.4 (40.62) 7.83 (39.1)
4.88 (23.6) 5.19 (24.82) 5.12 (24.51)
4.16 (19.53) 4.23 (19.9) 4.27 (20.03)
2.68 (12.31) 3.13 (14.55) 2.85 (13.36)
Table 3: EER [%] (minDCF [%]) for 5-digit password verification. Here tr and ev indices stand for training and evaluation parts of the used data bases.
System TrainData TestData EER (minDCF)
5,12 (24,51)
2,85 (13,36)
5,86 (29,49)
4,24 (20,45)
Table 4: EER [%] (minDCF [%]) for pooled male-female case (5-digit password)

There are two advantages of using speaker and text discriminative embeddings, trained on English and Russian digits speech signals. First, it automatically validate correctness of the passphrase. Second, it can be used for both languages (see Table 4).

Figure 1: Visualizing speakers embeddings using t-SNE. Each speaker is marked by its color, text indicates pronounced digit

Table 5

shows evaluation metrics for the best

StCNN system fused with baseline systems. Fusion of all systems enabled us to reduce verification EER on the pooled male and female 5-digit passphrase test to 1.43%, which is 54% less than the best single baseline system. Fusion is done with BOSARIS toolkit [21].

System Male Female Pooled
2.05 (9.12) 1.77 (8.62) 2.09 (9.59)
1.61 (7.7) 1.4 (6.43) 1.63 (7.32)
1.48 (6.62) 1.56 (8.66) 1.57 (7.72)
All 1.26 (5.02) 1.31 (6.92) 1.43 (6.58)
Table 5: Fusion. EER [%] (minDCF [%]) for 5-digit password verification

6 Conclusion

This paper proposed a new deep CNN-based solution to the text-prompted speaker verification problem. A high level feature extractor was trained to discriminate between speakers and digits simultaneously, and embeddings can be measured simply with cosine similarity. By using multitask learning scheme we were able to surpass the classic baseline systems in terms of quality and achieved impressive results for such a novice approach, obtaining 2.85% EER on the RSR2015 evaluation set. Fusion of the proposed and the baseline systems led to almost 50% decrease in EER for all test sets.

Acknowledgements. This work was financially supported by the Ministry of Education and Science of the Russian Federation, Contract 14.578.21.0189
(ID RFMEFI57816X0189).


  • [1] Najim Dehak, Patrick J Kenny, Réda Dehak, Pierre Dumouchel, and Pierre Ouellet, “Front-end factor analysis for speaker verification,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no. 4, pp. 788–798, 2011.
  • [2] Yun Lei, Nicolas Scheffer, Luciana Ferrer, and Mitchell McLaren, “A novel scheme for speaker recognition using a phonetically-aware deep neural network,” in 2014 IEEE ICASSP, pp. 1695–1699.
  • [3] Sergey Novoselov, Timur Pekhovsky, Oleg Kudashev, Valentin S Mendelev, and Alexey Prudnikov, “Non-linear PLDA for i-vector speaker verification,” in Sixteenth Annual Conference of the International Speech Communication Association, 2015.
  • [4] Oleg Kudashev, Sergey Novoselov, Timur Pekhovsky, Konstantin Simonchik, and Galina Lavrentyeva, “Usage of DNN in speaker recognition: advantages and problems,” in ISNN. Springer, 2016, pp. 82–91.
  • [5] Anthony Larcher, Kong Aik Lee, Bin Ma, and Haizhou Li, “Text-dependent speaker verification: Classifiers, databases and RSR2015,” Speech Communication, vol. 60, pp. 56–77, 2014.
  • [6] Hagai Aronowitz, “Text dependent speaker verification using a small development set,” in Odyssey 2012-The Speaker and Language Recognition Workshop, 2012.
  • [7] Sergey Novoselov, Timur Pekhovsky, Andrey Shulipa, and Alexey Sholokhov, “Text-dependent GMM-JFA system for password based speaker verification,” in 2014 IEEE ICASSP. IEEE, 2014, pp. 729–737.
  • [8] Sergey Novoselov, Timur Pekhovsky, Andrey Shulipa, and Oleg Kudashev, “PLDA-based system for text-prompted password speaker verification,” in Advanced Video and Signal Based Surveillance (AVSS), 2015 12th IEEE International Conference on, pp. 1–5.
  • [9] Pavel Matějka, Ondřej Glembek, Ondřej Novotnỳ, Oldřich Plchot, František Grézl, Lukáš Burget, and Jan Honza Cernockỳ, “Analysis of DNN approaches to speaker identification,” in Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on. IEEE, 2016, pp. 5100–5104.
  • [10] Hossein Zeinali, Lukas Burget, Hossein Sameti, Ondrej Glembek, and Oldrich Plchot, “Deep neural networks and hidden Markov models in i-vector-based text-dependent speaker verification,” in Odyssey-The Speaker and Language Recognition Workshop, 2016, pp. 24–30.
  • [11] Gautam Bhattacharya, Jahangir Alam, Themos Stafylakis, and Patrick Kenny, “Deep neural network based text-dependent speaker recognition: Preliminary results,” 2016.
  • [12] Shi-Xiong Zhang, Zhuo Chen, Yong Zhao, Jinyu Li, and Yifan Gong, “End-to-end attention based text-dependent speaker verification,” in Spoken Language Technology Workshop (SLT), 2016 IEEE. IEEE, 2016, pp. 171–178.
  • [13] Georg Heigold, Ignacio Moreno, Samy Bengio, and Noam Shazeer, “End-to-end text-dependent speaker verification,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5115–5119.
  • [14] Ehsan Variani, Xin Lei, Erik McDermott, Ignacio Lopez Moreno, and Javier Gonzalez-Dominguez, “Deep neural networks for small footprint text-dependent speaker verification,” in 2014 IEEE ICASSP. IEEE, 2014, pp. 4052–4056.
  • [15] David Snyder, Daniel Garcia-Romero, Daniel Povey, and Sanjeev Khudanpur, “Deep neural network embeddings for text-independent speaker verification,” Proc. Interspeech 2017, pp. 999–1003, 2017.
  • [16] X. Wu X., R. He, Z. Sun, and T. Tan, “A Light CNN for Deep Face Representation with Noisy Labels,” IEEE Journal of Selected Topics in Signal Processing, 2015.
  • [17] Galina Lavrentyeva, Sergey Novoselov, Egor Malykh, Alexander Kozlov, Oleg Kudashev, and Vadim Shchemelinin, “Audio replay attack detection with deep learning frameworks,” Proc. Interspeech 2017, pp. 82–86, 2017.
  • [18] Anthony Larcher, Kong Aik Lee, Bin Ma, and Haizhou Li, “RSR2015: Database for text-dependent speaker verification using multiple pass-phrases,” in Thirteenth Annual Conference of the International Speech Communication Association, 2012.
  • [19] Nanxin Chen, Yanmin Qian, and Kai Yu, “Multi-task learning for text-dependent speaker verification,” in INTERSPPECH, 2015, pp. 185–189.
  • [20] Laurens van der Maaten and Geoffrey Hinton, “Visualizing data using t-SNE,”

    Journal of Machine Learning Research

    , vol. 9, no. Nov, pp. 2579–2605, 2008.
  • [21] N. Brümmer and E. de Villiers, “The BOSARIS Toolkit: Theory, Algorithms and Code for Surviving the New DCF,” arXiv:1304.2865, Apr. 2013.