Deep learning-based source separation models have improved the single-channel speech denoising performance, nearing the ideal ratio masking results (IRM) [18, 26], or sometimes exceeding them . However, as the model size gets bigger, challenges grow for deploying such large models into resource-constrained devices, even just for feedforward inferences. Solving this issue is critical in real life scenarios for devices that require speaker separation and noise cancellation for quality speech communication.
There is an ongoing research in considering the tradeoff between complexity and accuracy, which is a priority for mobile and embedded applications. Optimization of convolutional operations have been explored to build small, low latency models [9, 27]. Pruning less active weights  and filters  is another popular approach to reducing the network complexity, too. The other dimension of network compression is to reduce the number of bits to represent the network parameters, sometimes down to just one bit [20, 23] , one of its kind has shown promising performances in speech denoising .
In this paper we take another route to source separation by redefining the problem as a -nearest neighborhood (
NN) search task: for a given test mixture, the separation is done by finding the nearest mixture spectra in the training set, and consequently their corresponding ideal binary mask vectors. However, the complexity of the search process linearly increases with the size of the training data. We expedite this tedious process by converting the query and database spectra into a hash code to exploit bitwise matching operations. To this end, we start from locality sensitive hashing (LSH), which is to construct hash functions such that similar data points are more probable to collide in the hashed space, or, in other words, more similar in terms of Hamming distance[10, 4]. While simple and effective, the random projection-based nature of the LSH process is not trainable, thus limiting its performance when one uses it for a specific problem.
We propose a learnable, but still projection-based hash function, Boosted LSH (BLSH), so that the separation is done in the binary space learned in a data-driven way . BLSH reduce the redundancy in the randomly generated LSH codes by relaxing the independence assumption among the projection vectors and learn them sequentially in a boosting manner such that they complement one another, an idea shown in search applications [15, 29]
. BLSH learns a set of linear classifiers (i.e. perceptrons), whose binary classification results serve as a hash code. To learn the sequence of binary classifiers we employ the adaptive boosting (AdaBoost) technique, while we redefine the original classification-based AdaBoost algorithm so that it works on our hashing problem for separation. Since the binary representation is to improve the quality of the hash code-based NN search during the separation, the objective of our training algorithm is to maximize the representativeness of the hash codes by minimizing the loss between the two self-similarity matrices (SSM) constructed from the original spectra and from the hash codes.
We evaluate BLSH on the single-channel denoising task and empirically show that with respect to the efficiency, our system compares favorably to deep learning architectures and generalizes well over unseen speakers and noises. Since binary codes can be cheaply stored and the NN search is expedited with bitwise operations, we believe this to be a good alternative for the speaker enhancement task where efficiency matters.
BLSH can be seen as an embedding technique with a strong constraint that the embedding has to be binary. Finding embeddings that preserve the semantic similarity is a popular goal in many disciplines. In natural language processing, Word2Vec or GloVe  methods use pairwise metric learning to retrieve a distributed contextual representation that retains complex syntactic and semantic relationships within documents. Another model that trains on similarity information is the Siamese networks [12, 2] which learn to discriminate a pair of examples. Utilizing similarity information has also been explored in the source separation community by posing denoising as a segmentation problem in the time-frequency plane with an assumption that the affinities between time-frequency regions could condense complex auditory features together . Inspired by studies of perceptual grouping , in 
local affinity matrices were constructed out of cues specific to that of speech. Then, spectral clustering segments the weighted combination of similarity matrices to unmix speech mixtures. On the other hand, deep clustering learned a neural network encoder that produces discriminant spectrogram embeddings, whose objective is to approximate the ideal pairwise affinity matrix induced from ideal binary masks (IBM). ChimeraNet extended the work by utilizing deep clustering as a regularizer for TF-mask approximation .
2 The NN Search-Based Source Separation
2.1 Baseline 1: Direct Spectral Matching
Suppose a masking-based source separation method by maintaining a large dictionary of training examples and searching for only NN to infer the mask. We assume that if the mixture frames are similar, so are the sources in the mixture as well as their IBMs.
Let be the normalized feature vectors from frames of training mixture examples, e.g., noisy speech spectra.
can be a potentially very large number as it exponentially grows with the number of sources. Out of many potential choices, we are interested in short-time Fourier transform (STFT) and mel-spectra as the feature vectors. For example, ifis from STFT on the training mixture signals, corresponds to the number of subbands in each spectrum, while for mel-spectra . is normalized with the L2 norm. We also prepare their corresponding IBM matrix, , whose dimension matches that of STFT. For a test mixture spectrum out of STFT, , our goal is to estimate a denoising mask, , to recover the source by masking, . While masking is applied to the complex STFT spectrum , the NN search can be done in the -dimensional feature space , e.g., for mel-spectra.
Algorithm 1 describes the NN source separation procedure. We use notation
as the affinity function, e.g., the cosine similarity function. For each framein the mixture signal, we find the closest frames in the dictionary (line 4 to 7), which forms the set of indices of NN, . Using them, we find the corresponding IBM vectors from and take their average (line 8).
Complexity: The search procedure requires a linear scan of all real-valued feature vectors in , giving , where stands for the floating-point precision (e.g., for double precision). This procedure is restrictive since needs to be large for quality source separation.
2.2 Baseline 2: LSH with Random Projections
We can reduce the storage overhead and expedite Algorithm 1 using hashed spectra and the Hamming similarity between them. We define random projection vectors as , where the -th bits in the codes is in the form of
with as a bias term. Applying the same onto and , we obtain bipolar binary and , respectively. Now, we tweak Algorithm 1, by having them take the binary feature vectors. Moreover, Hamming similarity also replaces the similarity function, which counts the number of matching bits in the pair of binary feature vectors: , where . Otherwise, the algorithm is the same. Fig. 1 overviews the separation process of the baseline 2.
Randomness in the projections, however, dampens the quality of the hash bits, necessitating for longer codes. This is detrimental in terms of query time, computational cost of projecting the query to hash codes, and storage overhead from the large number of projections. The lackluster quality of the codes originates from the data-blind nature of random projections [21, 28]. BLSH addresses this issue by learning each projection as a weak binary classifier.
Complexity: Since the same Algorithm 1 is used, the dependency to remains the same, while we can reduce the complexity from to if . The procedure can be significantly accelerated with supporting hardware as the Hamming similarity calculation is done through bitwise AND and pop counting.
3 The proposed Boosted LSH training algorithm for Source Separation
We set up our optimization problem with an objective to enrich the quality of the hash codes. Since the code vector is a collection of the binary classification results, during training the projection vectors are directed to minimize the discrepancies between the pairwise affinity relationships among the original spectra in and those we construct from their corresponding hash codes .
Hence, given hash string from the
-th projection, we express the loss function in terms of the dissimilarity between the self-similarity matrices (SSM) as follows:
for a given distance metric , such as element-wise cross-entropy. We scale the bipolar binary SSM to ranges of the ground truth’s such that . With this objective the learned binary hash codes can be more compact and representative than the ones from a random projection. There can be potentially many different solutions to this optimization problem, such as solving this optimization directly for the set of projection vectors or spectral hashing that learns the hash codes directly with no assumed projection process . The proposed BLSH algorithm employs an adaptive boosting mechnism to learn the projection vectors one by one.
We reformulate AdaBoost , an adaptive boosting strategy for classification, which is to learn efficient weak learners in the form of linear classifiers that complement those learned in previous iterations. It is an adaptive basis function model whose weak learners form a weighted sum to achieve the final prediction, in our case an approximation of the original self-similarity that minimizes the total error as follows:
where is the weight for the -th weak learner.
In AdaBoost, the -th projection is trained to focus more on the previously misclassified examples by assigning an exponentially larger weight to them. We redesign this part by making the sample weights exponentially large for the too different pairs of SSM elements as in (2). The SSM weights are initialized with uniform values over all pairs, and then updated after adding every projection, such that each pairwise location in the SSM is weighted element-wise based on the exponentiated discrepancy made by the current weak learner as follows:
The effect of this update rule is to tune the weights with respect to the distances between the bitwise and original SSMs on a given metric. Weights of the elements with the largest distance computed from the -th projection are amplified, and vice versa on close pairs. The -th weak learner can then use these weights to concentrate on harder examples. This approach pursues a complementary -th projection; thus, overall rapidly increasing the approximation quality in (3) with relatively smaller projections. The final boosted objective for the -th projection is formulated as
Under this objective, the first few projections index the majority of the original features, thereby dramatically reducing the overall storage of projections, computation from projecting elements, and the length of hashed bit strings. Given the learned -th projection and sample weights, we obtain the weights over the projections as
Algorithm 2 summarizes the procedure. Fig. 2 shows the complementary nature of the projections and their convergence behavior. After learning the projection matrix , the rest of the test-time source separation process is the same with Algorithm 1.
Complexity: BLSH does not change the run-time complexity of baseline 2. However, we expect that BLSH can outperform LSH with smaller thanks to the boosting mechanism.
4.1 Experimental Setup
To investigate the effectiveness of BLSH for the speech enhancement job, we evaluate our model on the TIMIT dataset. A training set consisting of 10 hours of noisy utterances was generated by randomly selecting 160 speakers from the train set, and by mixing each utterance with different non-stationary noise signals with 0 dB signal-to-noise ratio (SNR), namelybirds, casino, cicadas, computer keyboard, eating chips, frogs, jungle, machine guns, motorcycles, ocean . For both subsamples, we select half of the speakers as male and the other half as female. There are 10 short utterances per speaker recorded with a 16kHz sampling rate. The projections are learned on the training set and the output hash codes saved as the dictionary. 2 hours of cross validation set were generated similarly from the training set, which is used to evaluate the source separation performance of the closed speaker and noise experiments (closed mixture set). One hour of evaluation data set was mixed from 110 unseen speakers in the TIMIT test set and unseen noise sources from the DEMAND database, which consists of domestic, office, indoor public places, transportation, nature, and street (open mixture set) . The test set mixtures are with a 0 dB SNR and gender-balanced. We apply a Short-Time Fourier Transform (STFT) with a Hann window of 1024 and hop size of 256. During projection training, the mixture speech was segmented with the length of 1000 frames as a minibatch, such that the self-similarity matrices are of reasonable size. For evaluation, calculate the SDR, SIR, and SAR from BSS_Eval toolbox .
Fig. 3 shows the improvements in performance over the number of projections on the open mixture set. For our evaluation, all NN-based approaches were performed with to narrow the focus on BLSH behavior with respect to number of projections. We compare two sub-sampled dictionaries, 10% and 1%, respectively, to expedite the NN search process and to reduce the spatial requirement of the hashed dictionary. We repeat the test ten times on different randomly sampled subsets. Fig. 3
shows the average and respective standard deviations (shades). First of all, we see that larger dictionaries with 10% of data generally outperforms smaller dictionaries. Enlarging the length of the hash codealso generally improves the performance until it saturates. More importantly, BLSH consistently outperforms the random projection method even with less data utilization (BLSH learned from 1% of data always outperforms LSH with 10% subset), showcasing the merit of the boosting mechanism. Also, random projections show a noticeable sensitivity to the randomness in the sub-sampling procedure: the standard deviation is significantly larger especially in the 1% case, whereas that of BLSH remains tighter.
For comparison, we show the results in Table 1 at in the 10% case. In the table, the proposed BLSH method consistently outperforms the random projection approach and also NN on the raw STFT magnitudes in the open mixture case. Although the oracle IRM was higher in the open case, all systems perform worse than in the closed case. It suggests that the unseen DEMAND noise sources are actually easier to separate, yet difficult to generalize for models trained on mixtures with ten noise types. In particular, we found that for unseen noise types, such as public cafeteria sounds, the NN systems on the raw STFT magnitudes generalize poorly. Regardless, the proposed BLSH shows more robust generalization for the open mixture set than both the random projection method and the direct NN on the raw STFT case. It demonstrates the better representativeness of the learned hash codes than the raw Fourier coefficients or the basic LSH codes.
We compare the BLSH system with some deep neural network models as well, a 3-layer fully connected network (FC) and a bi-directional long short-term memory (LSTM) network with two layers. Both networks are with 1024 hidden units per layer. Unsurprisingly, both networks outperform BLSH. However, considering the significantly the faster bitwise inference process of BLSH, more than 10dB SDR improvement is still a promising performance.
Given these preliminary results, we believe further improvements can come from several areas. For example, by combining each values with the computed Hamming similarity from the -th hash codes, the separation could take into account the relative contributions of the learned projections. In addition, our approach does not employ information on the time axis since each projections are applied on a frame-by-frame basis. This could be enhanced with using projections on multiple frames and windowing. Finally, enriching the dictionary to include more disparate noise types as well as optimizing training using kernel methods are potential candidates for future work.
In this paper, we proposed an adaptive boosted hashing algorithm called Boosted LSH for the source separation problem using nearest neighbor search. The model trains linear classifiers sequentially under a boosting paradigm, culminating to a better approximation of the original self-similarity matrix with shorter hash strings. With learned projections, the -nearest matching frames in the hashed dictionary to the test frame codes are found with efficient bitwise operations, and a denoising mask estimated by the average of associated IBMs. We showed through the experiments that our proposed framework achieves comparable performance against deep learning models in terms of denoising quality and complexity111 The code is open-sourced on
The code is open-sourced onhttps://github.com/sunwookimiub/BLSH.222Audio examples can be found: https://saige.sice.indiana.edu/research-projects/BWSS-BLSH.
-  (2006) Learning spectral clustering, with application to speech separation. Journal of Machine Learning Research 7 (Oct), pp. 1963–2001. Cited by: §1.
-  (1994) Signature verification using a “siamese” time delay neural network. In Advances in Neural Information Processing Systems (NIPS), pp. 737–744. Cited by: §1.
-  (2001) The auditory organization of speech and other sources in listeners and computational models. Speech communication 35 (3-4), pp. 141–177. Cited by: §1.
-  (2004) Locality-sensitive hashing scheme based on p-stable distributions. In Proceedings of the twentieth annual symposium on Computational geometry, pp. 253–262. Cited by: §1.
-  (2012) Online PLCA for real-time semi-supervised source separation. In International Conference on Latent Variable Analysis and Signal Separation, pp. 34–41. Cited by: §4.1.
-  (1996) Experiments with a new boosting algorithm. In icml, Vol. 96, pp. 148–156. Cited by: §1, §3.
-  (2016) Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. In Proceedings of the International Conference on Learning Representations (ICLR), Cited by: §1.
-  (2016-03) Deep clustering: discriminative embeddings for segmentation and separation. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Cited by: §1.
Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861. Cited by: §1.
Approximate nearest neighbor – towards removing the curse of dimensionality. In
Proceedings of the Annual ACM Symposium on Theory of Computing (STOC), pp. 604–613. Cited by: §1.
-  (2018) Bitwise neural networks for efficient single-channel source separation. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Cited by: §1.
-  (2015) Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, Vol. 2. Cited by: §1.
-  (2016) Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710. Cited by: §1.
-  (2011) Learning to search efficiently in high dimensions. In Advances in Neural Information Processing Systems, pp. 1710–1718. Cited by: §1.
Boosting complementary hash tables for fast nearest neighbor search.
Thirty-First AAAI Conference on Artificial Intelligence, Cited by: §1.
-  (2017-03) Deep clustering and conventional networks for music separation: stronger together. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Vol. , pp. 61–65. Cited by: §1.
-  (2013) Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. Cited by: §1.
-  (2013) Ideal ratio mask estimation using deep neural networks for robust speech recognition. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 7092–7096. Cited by: §1.
-  (2014) GloVe: global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543. External Links: Cited by: §1.
-  (2016) XNOR-net: imagenet classification using binary convolutional neural networks. arXiv preprint arXiv:1603.05279. Cited by: §1.
-  (2009) Semantic hashing. International Journal of Approximate Reasoning 50 (7), pp. 969–978. Cited by: §2.2.
-  (1997) Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing 45 (11), pp. 2673–2681. Cited by: §4.2.
Expectation backpropagation: parameter-free training of multilayer neural networks with continuous or discrete weights. In Advances in Neural Information Processing Systems (NIPS), Cited by: §1.
-  (2013) The diverse environments multi-channel acoustic noise database (demand): a database of multichannel environmental noise recordings. In Proceedings of Meetings on Acoustics ICA2013, pp. 035081. Cited by: §4.1.
-  (2006) Performance measurement in blind audio source separation. IEEE transactions on audio, speech, and language processing 14 (4), pp. 1462–1469. Cited by: §4.1.
-  (2018) Supervised speech separation based on deep learning: an overview. IEEE/ACM Transactions on Audio, Speech, and Language Processing 26 (10), pp. 1702–1726. Cited by: §1.
Factorized convolutional neural networks.
Proceedings of the IEEE International Conference on Computer Vision, pp. 545–553. Cited by: §1.
-  (2009) Spectral hashing. In Advances in neural information processing systems, pp. 1753–1760. Cited by: §2.2, §3.
-  (2011) Complementary hashing for approximate nearest neighbor search. In 2011 International Conference on Computer Vision, pp. 1631–1638. Cited by: §1.
-  (2019) Efficient context aggregation for end-to-end speech enhancement using a densely connected convolutional and recurrent network. arXiv preprint arXiv:1908.06468. Cited by: §1.