Sequence to Multi-Sequence Learning via Conditional Chain Mapping for Mixture Signals

06/25/2020 ∙ by Jing Shi, et al. ∙ hitachi IEEE Johns Hopkins University Search Home 首页»»正文 0

Neural sequence-to-sequence models are well established for applications which can be cast as mapping a single input sequence into a single output sequence. In this work, we focus on one-to-many sequence transduction problems, such as extracting multiple sequential sources from a mixture sequence. We extend the standard sequence-to-sequence model to a conditional multi-sequence model, which explicitly models the relevance between multiple output sequences with the probabilistic chain rule. Based on this extension, our model can conditionally infer output sequences one-by-one by making use of both input and previously-estimated contextual output sequences. This model additionally has a simple and efficient stop criterion for the end of the transduction, making it able to infer the variable number of output sequences. We take speech data as a primary test field to evaluate our methods since the observed speech data is often composed of multiple sources due to the nature of the superposition principle of sound waves. Experiments on several different tasks including speech separation and multi-speaker speech recognition show that our conditional multi-sequence models lead to consistent improvements over the conventional non-conditional models.



There are no comments yet.


page 6

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Many machine learning tasks can be formulated as a sequence transduction problem, where a system provides an output sequence given the corresponding input sequence. Examples of such tasks include machine translation, which maps text from one language to another, automatic speech recognition (ASR), which receives a speech waveform and produces a transcription, and video captioning, which generates the descriptions of given video scenes. In recent years, the development of neural sequence-to-sequence (seq2seq) models

Cho et al. [2014], Sutskever et al. [2014] with attention mechanisms has led to significant progress in such tasks Bahdanau et al. [2015], Yao et al. [2015], Venugopalan et al. [2015], Wu et al. [2016], Chorowski et al. [2015], Chan et al. [2016].

In reality, the observed data contains various entangled components, making the one-to-many sequence transduction for mixture signals a common problem in machine learning Shun-Ichi Amari and Cardoso [1997], Jung et al. [1998], Roweis [2001]. This problem often happens in audio and speech processing due to the sequential properties and superposition principle of sound waves. For example, given the overlapped speech signal, speech separation is a problem of extracting individual speech sources, and multi-speaker speech recognition is a problem of decoding transcriptions of individual speakers. This type of problem is called the cocktail party problem Cherry [1953], Bregman [1994]. The existing methods to tackle this common sequence-to-multi-sequence problem can be roughly divided into two categories according to the correlation strength of multiple output sequences: serial mapping and parallel mapping.

Figure 1: Sequence-to-multi-sequence mapping approaches. (a) and (b) shows the serial mapping and parallel mapping used by existing methods respectively, and (c) refers to our Conditional Chain mapping strategy.

Serial mapping aims to learn the mappings through a forward pipeline, as shown in Figure 1(a). With the serial form, the output sequence of the first seq2seq model is fed into the following seq2seq model to output another sequence. This method is quite common when the logic and relationship between different output sequences are straightforward. For example, a cross-lingual speech translation system contains two components: speech recognition and machine translation. Serial mapping first recognizes the speech into the source language text and then use another model to translate it into the target language text. However, serial mapping methods usually suffer from some drawbacks. First, many of them need to be trained separately for different components, without taking advantage of the raw input information in the latter components. And the error accumulation through the pipeline will make the system suboptimal. The other category is parallel mapping, as shown in Figure 1(b), which simultaneously outputs multiple sequences. This method is often used when the outputs are from the same domain. Speech separation and multi-speaker ASR are typical examples following this paradigm. Similar to serial mapping, parallel mapping could not effectively model the inherent relationship that exists between different outputs, and usually assumes the number of the output sequence is fixed (e.g., the fixed number of speakers in speech separation tasks), which limits its application scenarios.

In this paper, we propose a new unified framework aiming at the sequence-to-multi-sequence (seq2Mseq) transduction task, which can address the disadvantages of both the serial mapping and parallel mapping methods. For clarity, we refer to our methods as Conditional Chain (CondChain) model, combining both the serial mapping and parallel mapping with the probabilistic chain rule. Simultaneous modeling for these two methods not only makes the framework more flexible but also encourages the model to automatically learn the efficient relationship between multiple outputs.

To instantiate the idea, as shown in Figure 1(c), we assume that the input sequence can be mapped into different sequences . We take sequence as the primary input for every output sequence. Meanwhile, the outputs will be generated one-by-one with the previous output sequence as a conditional input. We consider that the multiple outputs from the same input have some relevance at the information level. By combining both the serial and parallel connection, our model learns the mapping from the input to each output sequence as well as the relationship between the output sequences.

In this paper, we introduce the general framework in Section 2, and present a specific implementation for the tasks of speech separation and recognition in Section 3. We discuss some related work in Section 4 and describe our experiments in Section 5, and finally conclude in Section 6. Our source code will be available on our webpage:

2 General framework

We assume that the input sequence with length of can be mapped into different sequences , where the output index represents a particular domain . All the output sequences form a set

. The basic formulation of our strategy is to estimate the joint probability of multiple output sequences, i.e.,

. The joint probability is factorized into the product of conditional probabilities by using the probabilistic chain rule with/without the conditional independence assumption (denoted by ), as follows:


where, we also present the formulation of serial mapping and parallel mapping for comparison. As shown in Eq. 1, the serial mapping methods adopt a fixed order and the conditional distributions are only constructed with sequence from the previous step, i.e.,

. As a contrast, parallel mapping simplifies the joint distribution by the conditional independence assumption, which means all the output sequences are only conditioned on the raw input, i.e.,

. For our conditional chain mapping, we manage to explicitly model the inherent relevance from the data, even if it seems very independent intuitively. To achieve this, we depart from the conditional independence assumption in parallel mapping or the Markov assumption in serial mapping. Instead, with the probabilistic chain rule, our method models the joint distribution of output sequences over an input sequence as a product of conditional distributions. We can also apply the same methodology to the non-probabilistic regression output (e.g., speech separation).

In our model, each distribution in Eq. 1 is represented with a conditional encoder-decoder structure. Different from the conventional one-to-one sequence transduction for learning the mapping , additional module in our model preserves the information from previous target sequences and takes it as a condition for the following targets. This process is formulated as follows:


where, all the symbols are the number of dimensions for the features, and , , represent the size of temporal dimension. In the above equations, and refer to the specific networks designed for learning the mapping for the reference sequence . Note that the and

here may also consist of linear layers, attention mechanism or other neural networks besides the standard RNN layer, so the lengths of the hidden embeddings

and the estimation sequence may vary from the input, i.e., , , may not equal the . For the -th output, the in Encoder gets a length of while the should get the same length with the reference , where is the length of the sequence from domain . Different from the conventional seq2seq model, we utilize a conditional chain ( in Eq. 3) to store the information from the previous sequences and regard them as conditions. This conditional chain is analogous to the design of memory cell in the LSTM model and the key component to realize Figure 1(c). Similarly, the conditional chain in Eq. 3 does not serve a specific target domain alone, it models some unified information for multi-sequence outputs. In other words, the encoder-decoder is specialized for each target sequence, but the conditional chain is shared by all the transduction steps .

For most situations, when the logic and relationship between different output sequences is straightforward, we could set a fixed ordering of the outputted sequence, like the cross-lingual speech translation showed in Figure 1(a). Differently, for the outputs from the same domain, i.e., , the and for each step could be shared with the same architecture and parameters, which yields less model parameters and better efficiency for training.

3 Implementation for speech processing

This section describes our implementation of the proposed conditional chain model by using specific multi-speaker speech separation / recognition tasks as examples. Both of them are typical examples of seq2Mseq tasks with input from mixture signals.

3.1 Basic model

Multi-speaker speech separation / recognition aims at isolating individual speaker’s voices from a recording with overlapped speech. Figure 2 shows the network structure under our conditional chain mapping for these two tasks. For both of them, the input sequence is from the speech domain. Let us first assume this to be an audio waveform . Another common feature for these two tasks lies in that the output sequences are from the same domain (), which means we could use a shared model at each step, i.e., in Eq. 2 and in Eq. 4 are respectively the same networks with different .

For speech separation, the target sequences are all from the same domain as the input, i.e., , and with the same length of the input mixture. Thus, we could use an identical basic Encoder (Enc in Figure 2(a)) to extract acoustic features from the input waveform and predicted waveform . As a contrast, multi-speaker speech recognition outputs a predicted token sequence with token vocabulary . We introduce an additional embedding layer, , to map these predicted tokens as continuous representations, which is used for conditional representation.

Figure 2: Sequence-to-multi-sequence mapping with conditional model for multi speaker speech separation or recognition. In each sub-figure, the block with same name are all shared.

In speech separation, as illustrated in Figure 2(a), both the mixed audio and the predicted source will go through an Encoder to extract some basic auditory features. For the mixture waveform , another separator (

) will also be used as the function to learn some hidden representation which is suitable for separation. And both the

and the form the process in Eq. 2. For the block, due to the same lengths from input and output, a simple concatenation operation is used to stack the feature dimension for each frame. For the in Eq. 3, we use a unidirectional LSTM. At each source step , the Decoder () is used to map the hidden state into the final separated speech source. Multi-speaker ASR is also performed in a similar form, as illustrated in Figure 2(b). Note that we use connectionist temporal classification (CTC) Graves et al. [2006] as a multi-speaker ASR network, since CTC is simple but yet powerful end-to-end ASR, and also the CTC output tokens without removing blank and repetition symbols can have the same length with the auditory feature sequence. Thus, we can realize the processing with a simple concatenation operation, similarly to speech separation.

3.2 Stop criterion

One benefit of the conventional seq2seq model is the ability to output a variable-length sequence by predicting the end of the sequence ( symbol) as a stop criterion. This advantage is inherited in our model to tackle the variable numbers of multiple sequences. For example, current speech separation or recognition models are heavily depending on a fixed number of speakers Kolbaek et al. [2017] or require extra clustering steps Hershey et al. [2016]. Thanks to the introduction of the above stop criterion, we can utilize the mixture data with various numbers of speakers during training, and can be applied to the case of unknown numbers of speakers during inference.

In our implementation, when we have the total number of output sequences as , we attach an extra sequence to reach the stop condition during training. The target of this last sequence prediction for both speech separation and recognition tasks must be the silence, and we use the silent waveform and silent symbol (an entire label sequence in CTC), respectively.

3.3 Training strategy with teacher-forcing and ordering

Like the conventional seq2seq approach Bahdanau et al. [2015], we use a popular teacher-forcing Williams and Zipser [1989] technique by exploiting the ground-truth reference as a conditional source . Teacher-forcing provides proper guidance and makes training more efficient, especially at the beginning of the training, when the model is not good enough to produce reasonable estimation. Considering the unordered nature of multiple sources in multi-speaker speech separation or recognition, we adopt a greedy search method to choose the appropriate permutation of the reference sequences. This method achieves good performance in practice while maintaining high efficiency. More details about teacher-forcing and reference permutation search could be found in Section B in the Appendix.

4 Related Work

Speech Separation   As the core part of the cocktail party problem Cherry [1953], speech separation draws much attention recently Hershey et al. [2016], Isik et al. [2016], Yu et al. [2017b], Kolbaek et al. [2017], Luo et al. [2018], Luo and Mesgarani [2018a, b], Luo et al. [2020]. The common design of this task is to disentangle overlapped speech signals from a mixture speech with a fixed number of speakers, which is a typical example of the sequence-to-multi-sequence problem. Most existing approaches in this area follow the Parallel-mapping paradigm mentioned in Section 1, trained with permutation invariant training (PIT) technique Yu et al. [2017b]. This design should know the number of speakers in advance and could only tackle the data with the same number of speakers Shi et al. [2018]. These constraints limit their application to real scenes, while our proposed structure can provide a solution to the variable and unknown speaker number issues. This study is inspired by recurrent selective attention networks (RSAN) Kinoshita et al. [2018], which has been proposed to tackle the above variable number of speakers in speech separation by iteratively subtracting a source spectrogram from a residual spectrogram. Similar ideas have also been proposed in time-domain speech separation Takahashi et al. [2019] and speaker diarization  Fujita et al. [2020]

. However, the RSAN is based on the strong assumption of acoustic spectral subtraction in the time-frequency domain, and its application is quite limited. On the other hand, our conditional chain model reformulates it as a general sequence to multi-sequence transduction problem based on the probabilistic chain rule, which is applicable to the other problems including multi-speaker ASR than time-frequency domain speech separation. In addition, the relevance between current estimation and the former is learned by a conditional chain network in Eq. 

3, which is more flexible and even applied to time-domain speech separation, making it totally end-to-end transduction from waveform to waveforms.

Multi-speaker speech recognition   Multi-speaker speech recognition Weng et al. [2015], Yu et al. [2017a], Seki et al. [2018], Settle et al. [2018], which aims to directly recognize the texts of each individual speaker from the mixture speech, has recently become a hot topic. Similar to the speech separation task, most of the previous methods follow the parallel mapping paradigm mentioned in Section 1. These methods could only tackle the data with the fixed number of speakers and require external speaker counting modules (e.g., speaker diarization Tranter and Reynolds [2006], Anguera et al. [2012], Sell et al. [2018]), which lose an end-to-end transduction function, unlike our proposed method.

5 Experiments

We tested the effectiveness of our framework with speech data as our primary testing ground, where the sequence mapping problem is quite common and important. To be specific, the following sections describe the performance of our conditional chain model towards multi-speaker speech separation and speech recognition tasks, compared to other baselines. Furthermore, we also evaluated a joint model of speech separation and recognition, using multiple conditions from both waveform and text domains. In the Section A of Appendix, we provide the implementation details about all our experiments, and we also extend our model to one iterative speech denoising task in Section 5.5.

5.1 Datasets

For the speech mixtures, i.e., the input for our tasks, with different numbers of speakers, data from the Wall Street Journal (WSJ) corpus is used by us. In the two-speaker scenario, we use the common benchmark called WSJ0-2mix dataset introduced by Hershey et al. [2016]. The 30 h training set and the 10 h validation set contains two-speaker mixtures generated by randomly selecting speakers and utterances from the WSJ0 training set si_tr_s

, and mixing them at various signal-to-noise ratios (SNRs) uniformly chosen between 0 dB and 10 dB. The 5 h test set was similarly generated using utterances from 18 speakers from the WSJ0 validation set

si_dt_05 and evaluation set si_et_05. For three-speaker experiments, similar methods are adopted except the number of speakers is three. The WSJ0-2mix and WSJ0-3mix datasets have become the de-facto benchmarks for multi-speaker source separation, and we compare our results to alternative methods. Besides the separation task, we also instantiate our conditional chain model on multi-speaker speech recognition with the same WSJ0-2mix dataset.

5.2 Multi-speaker speech separation

Figure 3: Visualization of two examples (mix1&mix2) with 4-speaker mixture from our WSJ0-4mix testset. For the mixture, both the waveform and spectrogram are showed. More examples and audios are available on our webpage:
Eval (OC) SI-SNRi
in WSJ0-Nmix
Methods Training Data
Training with
variable number
of speakers
Single architecture
multiple architectures
2mix 3mix 4mix 5mix
RSAN Kinoshita et al. [2018] WSJ0-2mix single 8.8 - - -
OR-PIT Takahashi et al. [2019] WSJ0-2&3mix single 14.8 12.6 10.2 -
TasNet Luo and Mesgarani [2018b] WSJ0-mix multiple 14.6 11.6 - -
Our implemented TasNet Xu WSJ0-mix multiple 15.4 12.8 - -
ConvTasNet Nachmani et al. [2020] WSJ0-mix multiple 15.3 12.7 8.5 6.8
WSJ0-2mix single 15.6 - - -
WSJ0-3mix single 12.7 13.3 - -
WSJ0-2&3mix single 16.3 13.4 - -
WSJ0 2-5 mix single 16.7 14.2 12.5 11.7
DPRNN Nachmani et al. [2020] WSJ0-mix multiple 18.8 14.7 10.4 8.4
Voice Separation Nachmani et al. [2020] WSJ0-mix multiple 20.1 16.9 12.9 10.6
Table 1: Performance for speech separation on WSJ0-mix dataset, compared with the same base model and SOTA methods. The means the same base model with identical settings and hyper-parameters. The in WSJ0-mix means the dataset with fixed speakers. All the "multiple architectures" based methods are trained specifically towards each specific .
RefEst 2 3 4 5 6 Sum Acc (%)
2 2961 39 0 0 0 3000 98.7
3 12 2884 104 0 0 3000 96.1
4 0 42 2658 300 0 3000 88.6
5 0 0 125 2875 0 3000 95.2
Table 2: The estimation counting results on WSJ0-mix test set with variable number of speakers (2 to 5 in our experiments). The overall accuracy of the counting is 94.8%. Here we set the threshold of the energy value per frame in the estimated speech as to judge whether to stop the iteration.

First, we investigate the application of our conditional chain model to speech separation benchmarks. We take TasNet as the main base model in Figure 2(a), which is a simple but powerful de-facto-standard method in speech separation. Table 1

reports the results with different training settings (the number of speakers), compared with the same base model. By following the convention of this benchmark, we use the downsampled 8 kHz WSJ0-2mix set to reduce the memory consumption of separation. We retrained the TasNet with the same settings and hyper-parameters from an open source implementation

Xu . It should be noticed that the TasNet and most speech separation methods could only be trained and used in the same number of speakers, while our conditional chain model removes this limitation. In terms of the architecture of the model, we only added a single layer of LSTM compared with the base model, resulting in a negligible increase in the number of parameters.

The scale-invariant source-to-noise ratio improvement (SI-SNRi) results from Table 1 show that our conditional strategy achieves better performance than the base model (TasNet) under the same configuration. For the fixed number of speakers (2 or 3), our model improves on the original results, especially when there are more speakers (0.2 dB gains in WSJ0-2mix while 0.5 dB in WSJ0-3mix). Moreover, thanks to the chain strategy in our model, the WSJ0-2&3 mix datasets could be concurrently taken to train the model, and the performance is better than the training with each dataset. Compared with the RSAN Kinoshita et al. [2018] and OR-PIT Takahashi et al. [2019] as mentioned in Section 4, which also could handle variable number of sources, our model achieves significant performance improvements as a result of our end-to-end speech transduction in time domain. To further verify the upper limit that our model can reach with more speakers in speech separation task, we remixed two datasets, which we refer to as WSJ0-4mix and WSJ0-5mix, by simply concatenating the given mixture list from the standard WSJ0-2&3mix. As we expect, without adding any additional speech sources, the performance trained with WSJ0-2to5mix gains further improvement in WSJ0-2&3 mix and get reasonable SI-SNRi results in even 4 and 5 speaker mixtures.

Besides the baseline models related to our methods, we also report the results from two strong works, i.e., DPRNNLuo et al. [2020] and Voice Separation Nachmani et al. [2020], which two upgrade the model architecture from the TasNet. Especially, Voice Separation Nachmani et al. [2020] achieves the SOTA results in speech separation. However, their methods require multiple models (or multiple submodels in Nachmani et al. [2020]) for each number of speakers in advance, which import additional model complexities or training procedures, and also cannot be applied to more speakers than the training setup. Even though, with the suboptimal base model (TasNet), our conditional chain model gets better results in 5 speaker mixtures. And we could clearly observe that as the number of speakers increases, our model achieves less performance degradation. We expect to realize further gains with our conditional chain model by improving the base models to DPRNN or Vocice Separation, which will be a part of our future work.

Furthermore, Table 2 reports the estimation counting accuracy with the trained WSJ0 2-5mix datasets. Here, we set the threshold of the energy value per frame in the estimated speech as to judge whether to stop the iteration, resulting in the overall accuracy of 94.8%, which is significantly better than the Voice Separation Nachmani et al. [2020] (). It is worth mentioning that the upper bound of speaker number was set as 5 in our trained model, so there is a strong tendency to predict silence at the 6-th step (similar with the observation from RSAN Kinoshita et al. [2018]), leading to higher accuracy for 5-speaker mixtures than the 4-speaker ones. We also visualize two examples from the WSJ0-4mix test set in Figure 3, where we observe clear spectrograms with the estimated sources from pretty tangled mixtures.

5.3 Multi-speaker speech recognition

Methods System Training Data WER
WSJ0-2mix WSJ0-3mix
(1) DPCL + GMM-HMM Isik et al. [2016] HMM WSJ0-2mix 30.8% -
(2) PIT-DNN-HMM Qian et al. [2018] HMM WSJ0-2mix 28.2% -
(3) DPCL + DNN-HMM Menne et al. [2019] HMM WSJ0-2mix 16.5% -
(4) PIT-RNN Chang et al. [2019]
Attention-based WSJ0-2mix 25.4% -
(5) PIT-Transformer
Attention-based WSJ0-2mix 17.7% -
(6) PIT-Transformer
CTC WSJ0-2mix 31.2% -
(7) Conditional-Chain-Transformer
CTC WSJ0-2mix 24.7% -
(8) Conditional-Chain-Transformer
CTC WSJ0-1&2&3mix 14.9% 37.9%
Table 3: WER (%) for multi-speaker speech recognition on WSJ0-mix-16 kHz dataset.

Second, we evaluate our proposed method on the multi-speaker speech recognition task with the same WSJ0-mix corpus. We use the Transformer Vaswani et al. [2017] as the basic architecture block in Figure 2(b) to build speech recognition models, optimized with the CTC criterion Graves et al. [2006]. Unlike the separation experiment in Section 5.2, which used 8 kHz, this section used the sampling rate as 16 kHz, which is the default setup for ASR to achieve better performance like most other previous works. An LSTM-based word-level language model with shallow fusion is used during decoding Hori et al. [2018]. We compare our conditional chain Transformer-based CTC with the other systems including the HMM systems (1): deep clustering (DPCL)+GMM-HMM Isik et al. [2016], (2): PIT-DNN-HMM Qian et al. [2018] , and (3): DPCL+DNN-HMM Menne et al. [2019], and the attention-based systems (4): PIT-RNN Chang et al. [2019] and (5): PIT-Transformer. Note that all the PIT based methods correspond to the parallel mapping method in Figure 1(b), and they cannot deal with variable numbers of speakers.

We compare the effectiveness of the proposed conditional chain models with the same CTC architecture based on PIT in Table 3. Our conditional-chain Transformer (7) is significantly better than the corresponding PIT based system (6). Furthermore, the proposed conditional chain model can straightforwardly utilize the mixture speech data that has a variable number of speakers. To show this benefit, we train our model using the combination of the single and multi-speaker mixture WSJ0 training data. It can be seen that the conditional chain model trained with the combination of -speaker speech (8) achieves the best WERs, , on the 2-speaker mixture evaluation set among all the other systems including (1) – (7). Also, the proposed method can be applied to the 3-speaker mixture evaluation set and achieves reasonable performance () in this very challenging condition. This result is consistent with the observation in the speech separation experiment in Section 5.2. Interestingly, by analyzing the hypothesis generation process, we found that the speaker with the longest text is predicted first generally, described in Section C in the Appendix.

5.4 Cross-domain condition in joint separation and recognition

Figure 4: Conditional Chain Model for multi-speaker joint speech separation and recognition.
Methods Finetune Part Condition Training Data WSJ0-2mix
Conditional TasNet + Pre-trained ASR - Wave WSJ0-2mix 15.2 dB 25.1%
 + With Multiple Condition Separation Wave + CTC WSJ0-2mix 15.5 dB 17.2%
 + With Joint Training Separation, ASR Wave WSJ0-2mix 12.0 dB 15.3%
  + With Multiple Condition Separation, ASR Wave + CTC WSJ0-2mix 10.3 dB 14.4%
Table 4: SI-SNRi (dB) and WER (%) for multi-speaker joint training on WSJ0-mix-8 kHz dataset.

In previous experiments of separation and multi-speaker speech recognition tasks, we explore the effectiveness of our conditional chain model with output sequences from the same domain, as depicted in Figure 2. In contrast, in this subsection, we evaluate a combination of cross-domain conditions by using the ASR output to guide separation learning. This direction is motivated by the so-called informational masking effect in the cocktail party problem Brungart [2001], Kidd and Colburn [2017], where the linguistic clue may exert a strong influence on the outcome of speech-to-speech perception.

Specifically, as illustrated in Figure 4, in each step of our conditional chain model, both the separated waveform and the CTC alignment from the ASR model are utilized as the conditions for the next step, encouraging in Eq. 3 to jointly capture the information spanning multiple sources. The baseline conditional TasNet is trained using utterance-level waveform instead of chunk-level as in Section 5.2. In addition, unlike Section 5.3, we use a pre-trained ASR based on a single speaker Transformer-based CTC ASR system trained on the WSJ training set SI284 without overlap with WSJ0-2mix test set to stabilize our joint model training. We also use the downsampled 8 kHz data for the reduction of the memory consumption in separation and non-truncation of the speech mixture for appropriate ASR evaluation. Details of framework are explained in the Appendix A.3.

The results are shown in Table 4. Note that the numbers listed in this experiment cannot be strictly compared with those in Sections 5.2 and 5.3

due to the above different configuration requirements between separation and recognition. When directly feeding the separated waveform from the baseline conditional TasNet to the pre-trained ASR, we get 25.1% WER. Finetuning of the TasNet with both waveform and CTC alignment conditions achieved 0.3dB improvement of SI-SNRi. The improvement shows that the semantic condition, such as CTC alignment, provides a good guide to the separation learning. Finetuning of both TasNet and ASR models with both waveform and CTC alignment conditions yields the best WER of 14.4%, while waveform-only conditioning obtains the WER of 15.3%. Note that this joint training severely degraded the SI-SNRi result, but the WER result gets a significant improvement. This intriguing phenomenon demonstrates that the separation evaluation metric (SI-SNRi) does not always give a good indication of ASR performance, as studied in 

Subramanian et al. [2019, 2020].

5.5 Implementation for iterative speech denoising

Former experiments on multi-speaker speech separation and recognition show the effectiveness of our conditional chain to disentangle the input mixture signals into several components. For our proposed tasks above, actually, there is a mutually exclusive relationship between the outputs. However, the seq2Mseq tasks also cover some instances that the output sequences get a positive correlation. In this section, we manage to verify the ability to model this positive correlation, besides the proposed tasks above.

In the speech domain, the problem of a positive correlation between multiple outputs is also reflected in some problems. In this section, we take the speech denoising task as an example to verify the effectiveness of our conditional chain model in the case of positive correlation between iterative steps. The iterative estimation of some signals is an effective technique in speech processing, which could be used in speech enhancement Gannot et al. [1998]

, i-vector estimation

Medennikov et al. [2020], speech separation Kolbaek et al. [2017]. Similar to this technique for speech denoising, we implement our conditional chain model with two iteration in the chain to denoise the noisy input speech. This formulation is very similar to the iterative re-estimation of the clean signal. That is to say that our conditional chain method is trained with two identical references as objects, i.e., . And the output of the second step is conditioned on the estimation from the first step, similar to the structure shown in Figure 2(a).

To evaluate this, we conduct the speech denoising task based on a recently published dataset from the DNS-Challenge 2020 Reddy et al. [2020], which consists of 60,000 no-reverberant noisy clips in training and 300 in evaluation set.

Methods SDR SDRi
Official baselineXia et al. [2020] 13.4 4.2
Our implementation for Xia et al. [2020] 14.6 5.5
TasNet 17.3 8.2
Conditional TasNet (1st step) 17.8 8.7
Conditional TasNet (2nd step) 18.0 8.9
Table 5: The SDR performance on no-reverberant testset in DNS-challenge 2020.

Here, we compare the results with the official baseline model Xia et al. [2020] and our implemented baseline based on TasNet. And, we also report the performance of our conditional chain model with the same architecture and hyper-parameters with the base TasNet model.

From the results in Table 5, we could see that, with the iterative estimation of the clean speech signal in our conditional chain model, the performance gets obvious improvement over the same base model (TasNet). And, the estimation of the second step is better than the first step. These results show that our conditional chain learns to refine the condition from former steps, which further proves that our model has good adaptability and generalization performance when learning the relationship between multiple output sequences.

6 Conclusions

In this work, we introduced conditional chain model, a unified method to tackle a one-to-many sequence transduction problem for mixture signals. With the probabilistic chain rule, the standard sequence-to-sequence model is extended to output multiple sequences with explicit modeling of the relevance between multiple output sequences. Our experiments on speech separation and multi-speaker speech recognition show that our model led to consistent improvements with negligible increase in the model size compared with the conventional non-conditional methods,

In terms of the application scope, although we verify the proposed method with two specific tasks, speech separation and recognition, this conditional chain model can be flexibly extended to other sequence-to-multi-sequence problems for mixture signals, as a general machine learning framework. Therefore, as a future work, it might be interesting to adopt this method to other sequential tasks including natural language processing and video analysis. Another exciting direction would be introducing the attention mechanism into the fusion and conditional chain part, which could flexibly capture the implicit relationship between input and output from variable domains.


  • Anguera et al. [2012] Xavier Anguera, Simon Bozonnet, Nicholas Evans, Corinne Fredouille, Gerald Friedland, and Oriol Vinyals. Speaker diarization: a review of recent research. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 20(2):356–370, 2012.
  • Bahdanau et al. [2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proc. ICLR, 2015.
  • Bregman [1994] Albert S Bregman. Auditory scene analysis: the perceptual organization of sound. MIT press, 1994.
  • Brungart [2001] Douglas S Brungart. Informational and energetic masking effects in the perception of two simultaneous talkers. The Journal of the Acoustical Society of America, 109(3):1101–1109, 2001.
  • Chan et al. [2016] William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. Listen, attend and spell: a neural network for large vocabulary conversational speech recognition. In Proc. ICASSP, pages 4960–4964. IEEE, 2016.
  • Chang et al. [2019] Xuankai Chang, Yanmin Qian, Kai Yu, and Shinji Watanabe. End-to-end monaural multi-speaker asr system without pretraining. In Proc. ICASSP, pages 6256–6260. IEEE, 2019.
  • Cherry [1953] E. Colin Cherry. Some experiments on the recognition of speech, with one and with two ears. Journal of the Acoustical Society of America, 25(5):975–979, 1953.
  • Cho et al. [2014] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proc. EMNLP, pages 1724–1734. ACL, 2014.
  • Chorowski et al. [2015] Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. In Proc. NeurIPS, pages 577–585, 2015.
  • Fujita et al. [2020] Yusuke Fujita, Shinji Watanabe, Shota Horiguchi, Yawen Xue, Jing Shi, and Kenji Nagamatsu. Neural speaker diarization with speaker-wise chain rule. arXiv preprint arXiv:2006.01796, 2020.
  • Gannot et al. [1998] Sharon Gannot, David Burshtein, and Ehud Weinstein.

    Iterative and sequential kalman filter-based speech enhancement algorithms.

    IEEE/ACM Transactions on Audio, Speech, and Language Processing, 6(4):373–385, 1998.
  • Graves et al. [2006] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber.

    Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks.

    In Proc. ICML, pages 369–376., 2006.
  • Hershey et al. [2016] John R Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: discriminative embeddings for segmentation and separation. In Proc. ICASSP, pages 31–35. IEEE, 2016.
  • Hori et al. [2018] Takaaki Hori, Jaejin Cho, and Shinji Watanabe. End-to-end speech recognition with word-based rnn language models. In Proc. SLT, pages 389–396, 2018.
  • Isik et al. [2016] Yusuf Isik, Jonathan Le Roux, Zhuo Chen, Shinji Watanabe, and John R. Hershey. Single-channel multi-speaker separation using deep clustering. In Proc. INTERSPEECH, 2016.
  • Jung et al. [1998] Tzyy-Ping Jung, Colin Humphries, Te-Won Lee, Scott Makeig, Martin J McKeown, Vicente Iragui, and Terrence J Sejnowski. Extended ICA removes artifacts from electroencephalographic recordings. In Proc. NeurIPS, pages 894–900, 1998.
  • Kidd and Colburn [2017] Gerald Kidd and H Steven Colburn. Informational masking in speech recognition. In The Auditory System at the Cocktail Party, pages 75–109. Springer, 2017.
  • Kinoshita et al. [2018] Keisuke Kinoshita, Lukas Drude, Marc Delcroix, and Tomohiro Nakatani. Listening to each speaker one by one with recurrent selective hearing networks. In Proc. ICASSP, pages 5064–5068. IEEE, 2018.
  • Kolbaek et al. [2017] Morten Kolbaek, Dong Yu, Zheng Hua Tan, Jesper Jensen, Morten Kolbaek, Dong Yu, Zheng Hua Tan, and Jesper Jensen. Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(10):1901–1913, 2017.
  • Luo and Mesgarani [2018a] Yi Luo and Nima Mesgarani. Real-time single-channel dereverberation and separation with time-domain audio separation network. In Proc. INTERSPEECH, pages 342–346, 2018a.
  • Luo and Mesgarani [2018b] Yi Luo and Nima Mesgarani. Tasnet: time-domain audio separation network for real-time, single-channel speech separation. In Proc. ICASSP, pages 696–700. IEEE, 2018b.
  • Luo and Mesgarani [2019] Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(8):1256–1266, 2019.
  • Luo et al. [2018] Yi Luo, Zhuo Chen, and Nima Mesgarani. Speaker-independent speech separation with deep attractor network. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(4):787–796, 2018.
  • Luo et al. [2020] Yi Luo, Zhuo Chen, and Takuya Yoshioka. Dual-path rnn: efficient long sequence modeling for time-domain single-channel speech separation. In Proc. ICASSP, pages 46–50. IEEE, 2020.
  • Medennikov et al. [2020] Ivan Medennikov, Maxim Korenevsky, Tatiana Prisyach, Yuri Khokhlov, Mariya Korenevskaya, Ivan Sorokin, Tatiana Timofeeva, Anton Mitrofanov, Andrei Andrusenko, Ivan Podluzhny, et al. Target-speaker voice activity detection: a novel approach for multi-speaker diarization in a dinner party scenario. arXiv preprint arXiv:2005.07272, 2020.
  • Menne et al. [2019] Tobias Menne, Ilya Sklyar, Ralf Schlüter, and Hermann Ney. Analysis of deep clustering as preprocessing for automatic speech recognition of sparsely overlapping speech. In Proc. INTERSPEECH, pages 2638–2642, 2019.
  • Nachmani et al. [2020] Eliya Nachmani, Yossi Adi, and Lior Wolf. Voice separation with an unknown number of multiple speakers. arXiv preprint arXiv:2003.01531, 2020.
  • Qian et al. [2018] Yanmin Qian, Xuankai Chang, and Dong Yu. Single-channel multi-talker speech recognition with permutation invariant training. Speech Communication, 104:1–11, 2018.
  • Reddy et al. [2020] Chandan KA Reddy, Ebrahim Beyrami, Harishchandra Dubey, Vishak Gopal, Roger Cheng, Ross Cutler, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, et al. The interspeech 2020 deep noise suppression challenge: Datasets, subjective speech quality and testing framework. arXiv preprint arXiv:2001.08662, 2020.
  • Roweis [2001] Sam T Roweis. One microphone source separation. In Proc. NeurIPS, pages 793–799, 2001.
  • Seki et al. [2018] Hiroshi Seki, Takaaki Hori, Shinji Watanabe, Jonathan Le Roux, and John R Hershey. A purely end-to-end system for multi-speaker speech recognition. In Proc. ACL, pages 2620–2630. ACL, 2018.
  • Sell et al. [2018] Gregory Sell, David Snyder, Alan McCree, Daniel Garcia-Romero, Jesús Villalba, Matthew Maciejewski, Vimal Manohar, Najim Dehak, Daniel Povey, Shinji Watanabe, and Sanjeev Khudanpur. Diarization is hard: some experiences and lessons learned for the JHU team in the inaugural DIHARD challenge. In Proc. INTERSPEECH, pages 2808–2812, 2018.
  • Settle et al. [2018] Shane Settle, Jonathan Le Roux, Takaaki Hori, Shinji Watanabe, and John R Hershey. End-to-end multi-speaker speech recognition. In Proc. ICASSP, pages 4819–4823. IEEE, 2018.
  • Shi et al. [2018] Jing Shi, Jiaming Xu, Guangcan Liu, and Bo Xu. Listen, think and listen again: capturing top-down auditory attention for speaker-independent speech separation. In Proc. IJCAI, 2018.
  • Shun-Ichi Amari and Cardoso [1997] Shun-Ichi Amari and J.-F. Cardoso. Blind source separation-semiparametric statistical approach. IEEE Transactions on Signal Processing, 45(11):2692–2700, 1997.
  • Subramanian et al. [2019] Aswin Shanmugam Subramanian, Xiaofei Wang, Murali Karthick Baskar, Shinji Watanabe, Toru Taniguchi, Dung Tran, and Yuya Fujita. Speech enhancement using end-to-end speech recognition objectives. In Proc. WASPAA, pages 234–238. IEEE, 2019.
  • Subramanian et al. [2020] Aswin Shanmugam Subramanian, Chao Weng, Meng Yu, Shi-Xiong Zhang, Yong Xu, Shinji Watanabe, and Dong Yu. Far-field location guided target speech extraction using end-to-end speech recognition objectives. In Proc. ICASSP, pages 7299–7303. IEEE, 2020.
  • Sutskever et al. [2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In Proc. NeurIPS, pages 3104–3112, 2014.
  • Takahashi et al. [2019] Naoya Takahashi, Sudarsanam Parthasaarathy, Nabarun Goswami, and Yuki Mitsufuji. Recursive Speech Separation for Unknown Number of Speakers. In Proc. INTERSPEECH, pages 1348–1352, 2019.
  • Tranter and Reynolds [2006] Sue E Tranter and Douglas A Reynolds. An overview of automatic speaker diarization systems. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 14(5):1557–1565, 2006.
  • Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. NeurIPS, pages 5998–6008, 2017.
  • Venugopalan et al. [2015] Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence-video to text. In Proc. ICCV, pages 4534–4542. IEEE, 2015.
  • Watanabe et al. [2018] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al. Espnet: end-to-end speech processing toolkit. In Proc. INTERSPEECH, 2018.
  • Weng et al. [2015] Chao Weng, Dong Yu, Michael L Seltzer, and Jasha Droppo. Deep neural networks for single-channel multi-talker speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(10):1670–1679, 2015.
  • Williams and Zipser [1989] Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):270–280, 1989.
  • Wu et al. [2016] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.
  • Xia et al. [2020] Yangyang Xia, Sebastian Braun, Chandan K. A. Reddy, Harishchandra Dubey, Ross Cutler, and Ivan Tashev. Weighted speech distortion losses for neural-network-based real-time speech enhancement. In Proc. ICASSP, pages 871–875. IEEE, 2020.
  • [48] Kaituo Xu. Project webpage.
  • Yao et al. [2015] Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, and Aaron Courville. Describing videos by exploiting temporal structure. In Proc. ICCV, pages 4507–4515. IEEE, 2015.
  • Yu et al. [2017a] Dong Yu, Xuankai Chang, and Yanmin Qian. Recognizing multi-talker speech with permutation invariant training. In Proc. INTERSPEECH, pages 2456–2460, 2017a.
  • Yu et al. [2017b] Dong Yu, Morten Kolbæk, Zheng-Hua Tan, and Jesper Jensen. Permutation invariant training of deep models for speaker-independent multi-talker speech separation. In Proc. ICASSP, pages 241–245, 2017b.

Appendix A Implementation Details

a.1 Speech separation experiments

For our conditional TasNet, we used the original configure from TasNet Luo and Mesgarani [2018b] with . More specifically, TasNet contains three parts: (1) a linear 1-D convolutional encoder that encapsulates the input mixture waveform into an adaptive 2-D front-end representation, (2) a separator that estimates a fixed number of masking matrices, and (3) a linear 1-D transposed convolutional decoder that converts the masked 2-D representations back to waveforms. We use the same encoder and decoder design as in Luo and Mesgarani [2018b], referring to the and in Figure 2(a) respectively. For separator, we set the channel number of the last as one, making it outputs single speech at each step. And we move the last into the part. For the and LSTM, is concatenated with the conditional state from the previous step at the feature’s dimension. Then, a single layer of is carried out to mapping the fused feature into dimension at each frame.

Also, we noticed the update of the base model could further improve the performance like the same tendency in Luo and Mesgarani [2019], Luo et al. [2020]. In this paper, we mainly focus on the relative performance over the original TasNet. For separation-related tasks, all the speeches are re-sampled to 8 kHz to make a fair comparison with other works.

For the training loss calculation, the negative SI-SNR metric is widely used in Xu , Luo and Mesgarani [2018b, 2019], Luo et al. [2020], Nachmani et al. [2020] and achieves satisfying performance. However, the SI-SNR will lead the predicted sources as a scaled signal compared with the ground-truth, making it totally mismatch in inference phase. To address this problem, we change the training loss from negative SI-SNR to negative SDR, forcing the prediction of speech signals as similar as possible with the ground-truth.

For the training strategy, we set the initial learning rate of

, which is multiplying by 0.9 every 8 epochs. In practice, we find the original ground-truth signals used as condition result in faster training speed, but a decrease in generalization ability. Therefore, to make the separation more robust, we add Gaussian noise with a standard deviation of 0.25 on these ground-truth source vectors (waveforms) during training.

a.2 Multi-speaker speech recognition experiments

We basically use the open source package ESPnet Watanabe et al. [2018] for the implementation of our ASR model. In the conditional Transformer-based CTC ASR model, there is a total of 16 Transformer layers, 8 before and 8 after the conditional chain LSTM. For the baseline Transformer-based CTC with PIT, there is a total of 12 Transformer layers in the acoustic model. The configuration of each Transformer layer is as follows: the dimension of attention is , the dimension of feed-forward is , number of heads is . Before feeding the input to the Transformers, the log mel-filterbank features are encoded by two CNN blocks. The CNN layers have a kernel size of and the number of feature maps is 64 in the first block and 128 in the second block.

a.3 Multi-speaker joint speech separation and recognition

In the joint training experiments, we first pre-train two models for both separation and ASR tasks. The parameters of conditional TasNet are the same as our previous setup. For the pre-trained ASR, we train a single speaker Transformer-based CTC ASR model on the WSJ training set, which is a clean close-mic read speech corpus with about 80h. When jointly finetuning these two parts, we use the separated wave of TasNet as the input of ASR model and feedback the predicted CTC alignments of ASR into TasNet as an additional condition. Figure 4 shows an overview of our joint model.

Besides, we also introduce two extra teacher-forcing hyperparameters to control the optimization part of the joint model, which are

and . The parameter is the probability of feeding separated wave to ASR, while is the probability of inputting predicted CTC alignments as the condition. When aiming to optimize TasNet with multiple conditions, we fixed the parameters of ASR and set and , which means we only feed ground-truth wave to ASR and use the generated CTC alignments to guide the separation learning. When joint training both part to improve the performance of ASR, we set and . All experiments only have access to the predicted wave and CTC alignments during inference.

Appendix B Training strategy with teacher-forcing and ordering

Figure 5: The procedure of selecting and sorting the target sequences using a greedy algorithm in our proposed methods. At each step , the ground-truth sequence is selected from all the available references. In this example, the final order is .

In Eq. 3, the neural network accepts the hidden state of the previous sequence that is estimated at the previous iteration. However, the estimation error at the previous iteration hurts the performance at the next iteration. To reduce the error, we use the teacher-forcing Williams and Zipser [1989] technique, which boosts the performance by exploiting ground-truth reference. During training, Eq. 3 is replaced with as follows:


Here, is a ground-truth sequence of index . The teach-forcing technique is commonly used in conventional seq2seq methods. However, target sequence in seq2seq is determined and has an immutable order, so the previous approaches also generate the sequence through a fixed order, either from the beginning to the end, or the reverse order. But for many seq2MSeq problems, the multiple reference sequences are unordered. There arises a problem about how to select the from the to process the next iteration, which is also how to determine the best order of target sequences.

One most straightforward method is to use the permutation invariant training (PIT) strategy to traverse all the permutations and select the optimal one to update the parameters. However, with the teacher-forcing technique attending each step of the output, we must go through the whole feedforward process for each permutation, which takes too much computational complexity. To alleviate this problem, we examine one simple greedy search strategy. As shown in Figure 5, for each output iteration , the optimal target index is selected by minimizing the difference (distance) with among a set of the remaining target set, and the selected ground-truth sequence is fed into the next decoding iteration. With this greedy strategy, the repetitive computation only occurs at the calculation of distance, and there is no need to re-run the feedforward process. In addition, the total number of repetitive computation is , compared with the in PIT based strategy.

Appendix C Analysis on Speech Recognition Outputs

Our method is trained in a greedy fashion, which does not address the label permutation problem from the formulation as other methods do, such as permutation invariant training (PIT) or deep clustering (DPCL), etc. We further look into the speech recognition results in terms of the order following which our model generates the hypotheses. In Table C

, we show the confusion matrix of the prediction order and the text length ranking for the two-speaker scenario. We observe that the order is somehow correlated to the length of the text. In

of evaluation samples, the generation order is consistent with the length ranking. By considering two texts sequences that may have very close lengths or some words are much simpler than the others, we loosen the constraint for the length ranking. If we simply accept those cases where the reference text of the first hypothesis is shorter than the second, but within a range, the pattern is more obvious. For example, the result is shown in Table C when the range is . In evaluation samples, the generation order is consistent with the length ranking. In the three-speaker scenario, we find a similar pattern: the first hypothesis is significantly longer than the other two, when the range is . We also have the same conclusion on the Transformer-based CTC ASR system trained with PIT in two speaker scenario. It is a Parallel-mapping framework without such conditional dependency. However, we found that the output of each head is highly dependent on the lengths. In our model, of output from one head is longer than that from the other. If we further consider the range of , the ratio becomes

. Perhaps this can be the heuristic information to address the label permutation problem.

Table 6: Confusion matrix between the hypothesis (Hyp.) generation order and the order of reference (Ref.) text length in 2-speaker case. Hyp.Ref. long short 1st output 2627 373 2nd output 373 2627 Table 7: Confusion matrix between the hypothesis (Hyp.) generation order and the order of reference (Ref.) text length in 2-speaker case with loosing range 5. Hyp.Ref. long short 1st output 2965 35 2nd output 35 2965