Exploratory Analysis of a Large Flamenco Corpus using an Ensemble of Convolutional Neural Networks as a Structural Annotation Backend

06/29/2018
by   Nadine Kroher, et al.
0

We present computational tools that we developed for the analysis of a large corpus of flamenco music recordings, along with the related exploratory findings. The proposed computational backend is based on a set of Convolutional Neural Networks that provide the structural annotation of each music recording with respect to the presence of vocals, guitar and hand-clapping ("palmas"). The resulting, automatically extracted annotations, allowed for the visualization of music recordings in structurally meaningful ways, the extraction of global statistics related to the instrumentation of flamenco music, the detection of a cappella and instrumental recordings for which no such information existed, the investigation of differences in structure and instrumentation across styles and the study of tonality across instrumentation and styles. The reported findings show that it is feasible to perform a large scale analysis of flamenco music with state-of-the-art classification technology and produce automatically extracted descriptors that are both musicologically valid and useful, in the sense that they can enhance conventional metadata schemes and assist bridging the semantic gap between audio recordings and high-level musicological concepts.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 5

page 12

page 24

01/31/2021

Structure-Aware Audio-to-Score Alignment using Progressively Dilated Convolutional Neural Networks

The identification of structural differences between a music performance...
02/18/2021

Low Resource Audio-to-Lyrics Alignment From Polyphonic Music Recordings

Lyrics alignment in long music recordings can be memory exhaustive when ...
12/05/2019

Love Me, Love Me, Say (and Write!) that You Love Me: Enriching the WASABI Song Corpus with Lyrics Annotations

We present the WASABI Song Corpus, a large corpus of songs enriched with...
08/05/2020

Learning to Denoise Historical Music

We propose an audio-to-audio neural network model that learns to denoise...
10/16/2019

A Study of Annotation and Alignment Accuracy for Performance Comparison in Complex Orchestral Music

Quantitative analysis of commonalities and differences between recorded ...
08/26/2020

The Freesound Loop Dataset and Annotation Tool

Music loops are essential ingredients in electronic music production, an...
06/04/2021

Classification of Audio Segments in Call Center Recordings using Convolutional Recurrent Neural Networks

Detailed statistical analysis of call center recordings is critical in t...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

The growing availability of digital music collections, in the form of symbolic music representations and audio recordings, along with recent advances in symbolic and audio-based Music Information Retrieval (MIR), have enabled researchers to scale descriptive music analysis [Whi94, Ber03] from small collections of a few manually annotated items to large corpora of music content. In the case of non-Western music traditions, which often lack formal documentation, exploratory studies of music corpora are a powerful means to reveal distinctive musical characteristics, validate musicological research hypotheses and identify meaningful features for the automatic organization of digital collections. It is now acknowledged that the development of computational analysis tools can contribute significantly to the preservation of cultural heritage, assist the understanding of the past and help bridge the semantic gap between sparsely documented content and concepts at various levels of music abstraction.

Data-driven studies have already shown significant potential in discovering knowledge and hidden correlations within and among music genres. The work in [PBBD17], which was based on a learned dictionary of vocal pitch contour elements, showed that singing styles with geographical proximity, exhibit similar characteristics. In [Con10], a comparison of melodic pattern occurrences between genre-specific corpora and an anti-corpus, revealed distinctive patterns for different folk song traditions. Using Subgroup Discovery techniques, the method in [THM09] generated a set of interpretable rules which can associate folk song transcriptions with their origin. An analysis of manually annotated harmonic progressions in rock music [dCT11] revealed a number of frequently occurring chords and chord sequences, as well as evolutionary trends over several decades. In [VdH13], a large-scale corpus study of ragtime scores led to the empirical validation of certain hypotheses that are related to the occurrence of rhythmic patterns.

In this paper, we present, for the first time, an exploratory study of a large flamenco corpus [KDBMG16], from a computational analysis perspective. In particular, we focus on the musical structure with respect to instrumentation and its relation to style and tonality. Flamenco is a rich oral music tradition from Southern Spain, which, under the influence of diverse cultures, has evolved from its folkloric origin to an elaborate art form [GMDBGM14]. As an oral tradition with a highly improvisational character, flamenco is not bound to scores, but it is based on style-specific, melodic, rhythmic and harmonic templates, which set the basis for expressive performance. These underlying rules are, to a large extent, implicitly passed from generation to generation, and, despite recent efforts to formalize flamenco music theory [Man86, Fer04], many aspects remain undocumented.

Given the absence of scores, our large-scale analysis of flamenco music relies on automatic annotations that are directly extracted from audio recordings. More specifically, we developed an automatic annotation system which segments each music recording into sections that are consistent with respect to instrumentation. This is feasible, because in classical flamenco recordings, instrumentation is in most cases limited to singing with guitar accompaniment and rhythmic hand-clapping (referred to as palmas). Some styles are traditionally performed a capella, and, in rare cases, performances can be instrumental.

To detect segments with consistent instrumentation, we employ an ensemble of Convolutional Neural Networks (CNNs), where each CNN solves a binary classification problem. CNNs were selected as a backend, because, over the past few years, they have been successfully applied to various analysis tasks, ranging from face detection

[LLS15, GD04] and object recognition [KSH12], to MIR tasks including boundary detection [USG14], music recommendation [VdODS13] and instrument recognition [HKL17].

The three CNNs proceed to the segmentation of an audio recording as follows:

  • The first classifier detects parts of the recording where the singing voice is present.

  • In the remaining solo guitar sections, a second classifier distinguishes between two guitar playing techniques, strumming and picking.

  • The third classifier detects parts of the recording where the palmas are present, irrespective of the outcome of the other classifiers.

Figure 1 depicts schematically the outcome of the annotation procedure.

Figure 1: Schematic illustration of the segmentation of a flamenco recording with respect to instrumentation.

This type of automatic annotation sets the basis for a large-scale analysis of a music corpus containing more than commercial flamenco recordings [KDBMG16]. In particular:

  • We detect instrumental and a cappella recordings based on the estimated percentage of vocal and non-vocal frames.

  • We explore intra and inter-style similarity based on two types of features: song-level instrumentation statistics and a time series representation of the temporal structure. We also investigate the usefulness of these features for automatic style recognition tasks.

  • We verify music theoretical assumptions on tonality with respect to style and instrumentation, by correlating pitch class profiles with mode templates.

The results of this study can be further exploited by several potential applications for automatic indexing and content-based retrieval in digital flamenco collections, can provide valuable clues towards the unresolved problem of automatic style detection [KDB16], and can reveal interesting musicological observations.

The rest of the paper is structured as follows: Section 2 provides a detailed description and evaluation of the structural segmentation backend. The exploratory analysis study, its results and a detailed description of the corpus, are presented in Section 3. Finally, conclusions are drawn in Section 4.

2 Structural annotation backend

In this section, we describe the segmentation backend, the purpose of which is to break a flamenco music recording into segments of consistent instrumentation, thus yielding a structural annotation of the track. An overview of the proposed multi-stage system architecture is shown in Figure 2.

At a first stage, given a raw audio recording, silent sections are detected based on the short-term energy envelop of the signal. Silence usually occurs at the endpoints of a recording and between successive sung sections in a cappella styles. The detected silent parts are excluded from subsequent classification stages to improve classification accuracy.

Subsequently, we exploit the fact that in a flamenco stereo recording, the vocals are usually dominant in one channel and the guitar in the other. It is worth noting that this is not only the case with live recordings, where the resulting stereo panorama corresponds to the physical location of singer and guitarist on stage, but it is also common practice in multi-track flamenco studio productions, as a means to separate voice and guitar in the artificially created panorama. Here, we apply the heuristic that was proposed in

[KG16] to identify the channel in which the vocals are dominant. In the sequel, we will refer to this channel as the vocal channel and to the other one as the guitar channel. Furthermore, we create a mix of the two channels by computing the mean signal, to which we will later refer as the mono mix. If the recording is monophonic, the channel selection stage is skipped, in which case, the vocal and guitar channels coincide with the mono mix signal.

At the next stage, the vocal channel is passed as input to the first CNN, which acts as a vocal detector and estimates, on a frame-level basis, if the singing voice is present or not. Successive frames that have been classified to the vocal class are considered to form a vocal segment and are annotated accordingly. Vocal detection in flamenco music has been previously studied in the context of automatic transcription [KG16] and in an unsupervised scenario [PKKDB16]. However, the CNN-based method that is proposed in this paper is shown to outperform previous approaches. The remaining, non-vocal segments are taken from the guitar channel and are passed as input to the next CNN, which discriminates, again on a frame-level basis, between two guitar playing techniques, namely, picking and strumming. The third CNN operates independently, takes the mono mix signal as input and detects, for each frame, if palmas are present or not.

Figure 2: Schematic overview of the segmentation backend.

Below, we describe in detail, from a technical perspective, each stage of the processing chain and justify the respective design choices.

2.1 Pre-processing

After normalizing the audio recording and re-sampling it to , we detect silent frames by analyzing the signal energy over non-overlapping short-term windows, samples long. A frame is considered to be silent, if its energy value is at least lower than the highest energy value in the recording.

In order to identify the channel in which the vocals are dominant, we apply the heuristic that was proposed in [KG16], which evolves around the observation that the presence of the singing voice increases the energy in the frequency range between and . Although this criterion is not sufficient to accurately detect the presence of vocals at the frame-level, it has however shown to reliably point to the channel in which the vocals are dominant. For each channel, we compute, for every short-term frame, the ratio of the sum of magnitudes of the DFT coefficients in the frequency range from to , to the respective sum in a lower frequency band, ranging from to . The channel with the higher average ratio is selected as the vocal channel and the other one as the guitar channel.

2.2 Feature extraction

Before a signal is fed to a CNN, a short-term feature extraction stage extracts a sequence of feature vectors from its time-domain representation. Specifically, the signal is first parsed with a non-overlapping moving window,

samples long (

ms). At each window, the Discrete Fourier Transform (DFT) is computed and it is given as input to a mel filter-bank that yields

mel-band energy coefficients. Each mel-filter computes a weighted sum of the DFT coefficients that lie inside its frequency range ([RJ93]).

After the short-term feature extraction stage has been completed, a subsequence of successive feature vectors is aggregated each time to form a image representation, that will serve to assign a classification label to the time instant corresponding to the middle of the subsequence. As a result, the short-term feature sequence generates a sequence of 2-D images. Each image reflects the evolution of spectral content (image height) over a neighborhood of frames (image width) around the time instant for which a classification decision will be made. Given the short-term window step of samples and the sampling frequency of , an image width of frames corresponds to of audio, i.e., to s of audio around the time instant to which the classification label will be assigned. In order to reduce the computational burden, the image slides by a hop size of frames at a time. Equivalently, a classification decision is taken approximately every s. To ensure robustness to signal intensity variation, each image is normalized by dividing each one of its elements by the maximum absolute value of the image. The feature extraction process is depicted in Figure 3.

This approach is inspired by the two-dimensional pixel matrices that are used in image classification systems and has shown to give promising results in various audio analysis tasks (see for example [USG14, HKL17, HCE17]). For the CNN that serves as the palmas detector, the logarithm of the mel-band energy values is computed, to capture the characteristic sudden energy increases that are produced by the percussive impulses.

Figure 3: Illustration of the feature extraction stage.

2.3 CNN architecture

The three CNNs follow a common architecture, shown in Figure 4. Specifically, each input matrix, for which a classification decision will be taken, is passed through two convolutional layers. Each layer contains convolutional masks (feature detectors) of size . The output of each convolutional operation is passed through a relu function and the resulting feature matrix is subsequently subsampled by a max pooling scheme. The resulting representation at the output of the second layer is unfolded (flattened), yielding a

one-dimensional representation, which is subsequently fed as input to a standard, fully connected feed forward neural network. This network has a hidden layer of

units and a softmax output layer of two units due to the binary nature of the classification tasks under study. Figure 4 presents the data flow across the adopted neural network architecture, including the dimensionality of matrices at each processing stage.

Figure 4: Illustration of the CNN architecture.

2.4 Post-processing

As it was previously described, when an input image is given, the predicted class is interpreted as the label of the time instant corresponding to the middle of the respective frame subsequence. Furthermore, assuming that instrumentation remains consistent over short time periods, we apply a median-filter to the sequence of binary classification decisions to suppress noisiness. In particular, for the vocal detection and guitar playing technique classification tasks, a median filter, long, is used. For the palmas detection module, we use a longer median filter, long, as we expect that the palmas events unfold over longer time periods.

2.5 Classifier training

The singing voice classifier is trained and evaluated on the cante100 dataset, which has been previously used in the context of vocal detection in flamenco music [PKKDB16]. The collection is a subset of 100 commercial recordings taken from the corpusCOFLA collection of flamenco recordings [KDBMG16]. For the classifier of guitar playing techniques, we manually annotated strummed and picked guitar sections on the same dataset. It has to be noted that, with respect to the presence of palmas, the cante100 collection proves to be rather unbalanced - only about 10 % of the tracks contain hand-clapping. We therefore assembled a balanced subset of recordings from the corpusCOFLA collection, where half the recordings contain hand clapping.

All three classifiers are trained and evaluated using a -fold cross-validation scheme. For the singing voice detection task, artist-filtered folds are used, i.e., all songs of the same singer are included in the same fold. For the other two tasks, folds are song-filtered, ensuring that all training images from a common recording are located in the same fold. These standard measures are taken to reduce the risk that the classifiers will overfit to the timbral characteristics of a particular singer or recording. After the feature extraction stage is carried out on the aforementioned datasets, each classifier is trained for epochs using the Adam [KB14] algorithm for optimization of the cross-entropy function over the training set. During each training epoch, the images are shuffled and are grouped to form mini-batches ( images per mini-batch). An early-stopping criterion is also used, which terminates the training procedure if the loss-value does not decrease significantly during a patience-period of epochs (at least by a value of ).

2.6 Baseline methods

The performance of each binary classifier is compared against at least one baseline system. More specifically, the vocal detection classifier is compared against:

  • The method in [SLY13]

    which is based on a Gaussian Mixture Model (GMM) per class, trained on frame-level mel-frequency cepstral coefficients

    (GMM-BL-VOC).

  • The vocal detection module of the automatic transcription system cante [KG16], where vocal sections are first detected depending on the presence of the predominant melody and are subsequently filtered based on low-level audio descriptors (CANTE).

  • The unsupervised method in [PKKDB16], which learns a dictionary of atoms from the bark band representation of the signal and detects the singing voice based on the activation patterns of the computed dictionary atoms (UNSUPERVISED).

As a baseline method for the performance of the guitar playing technique and palmas detection classifiers, we implemented a GMM-based method for each task. Specifically, we extract the mel-frequency cepstrum coefficients and their first and second derivatives over time in long windows, with a hop size of and average the extracted short-term sequence with a s long moving window. We then train a GMM with

components for each class. Each frame is assigned to the class whose GMM yields the highest probability.

All baseline methods and the CNN approach are trained and evaluated using the same -fold cross validation scheme. Furthermore, the same pre- and post-processing stages are applied as for the proposed system.

2.7 Experimental results

The results of the vocal detection experiments are shown in Table 1. It can be seen that the proposed CNN-based method achieves the highest F-measure () among the four algorithms. A manual analysis showed that errors were mainly caused by shouts from the guitarist and audience during performance, which were mistakenly classified as singing voice segments. Although limited in duration, this is a typical phenomenon in flamenco performances.

Tables 3 and 2 show the results of the guitar playing technique and palmas classification experiments. For both tasks we observe that the proposed method outperforms the GMM-based baseline approach. For the guitar playing technique classification, the CNN yields an F-measure of and for the palmas detection task an F-measure of . The performance of the GMM-baseline method is significantly lower, yielding an f-measure of for the guitar playing technique classification task and for the palmas detection method. In the former, picked guitar was considered as the positive class.

We furthermore observe, that the performance of each CNN-based classifier is balanced with respect to precision and recall, indicating that there is no notable bias towards one of the target classes. In the vocal detection task, the

CANTE method reaches the highest recall value of , however, at the cost of a significantly lower recall value of , compared to the proposed method ().

Method Precision Recall F-measure
GMM-BL-VOC 0.92 0.85 0.88
CANTE 0.97 0.75 0.85
UNSUPERVISED 0.94 0.77 0.85
PROPOSED 0.94 0.95 0.95
Table 1: Experimental results: vocal detection.
Method Precision Recall F-measure
GMM-BL-PAL 0.66 0.65 0.66
PROPOSED 0.94 0.95 0.95
Table 2: Experimental results: palmas detection.
Method Precision Recall F-measure
GMM-BL-GIT 0.69 0.80 0.74
PROPOSED 0.91 0.92 0.91
Table 3: Experimental results: guitar playing technique classification.

2.8 What did the networks learn?

We are now making an attempt to analyze in detail how the networks solve the tasks at hand, which is always a challenging issue when convolutional architectures are employed. To this end, based on selected recordings, we examine the relu activations of the convolutional filter output of the first and second layer (Figure 6) for each target class. Furthermore, we visualize the output of selected filters from both layers for the input frames which yield the highest and lowest output probabilities for each target class (Figure 5).

To begin with, an examination of the two input images that yield the highest and lowest posterior probability for the singing voice class (Figure

5), reveals that the presence of the singing voice causes the spectral energy to be centered in a higher frequency range, compared to the case of guitar.

Figure 5: Visualisation of input features and first and second layer convolutional filter outputs: (a) vocal detection; (b) palmas detection; (c) guitar playing technique classification.

We can furthermore observe a fluctuation of the dominant frequency bins over time, which is not the case with the horizontal spectral lines of the guitar image (which do not exhibit fluctuation). Comparing images with and without palmas, we can see, that hand clapping produces sudden short-duration increases in the spectral magnitude, that become more apparent in high frequencies. Filter no. 4 seems to detect these sudden bursts because of the vertical stripes visible in the output its mask. Filter no. 8 yields a high activation in areas with low spectral magnitude. Consequently, this filter has a high activation for frames without palmas, that exhibit less energy in the high frequency range. The feature matrices representative for strummed and picked guitar differ mainly in the number of temporally simultaneous peaks and the long-term pitch fluctuation. The strummed guitar produces multiple sustained peaks, visible as horizontal lines, whereas the picked guitar results in single peaks which quickly decay.

Figure 6: Average activation of first and second layer convolutional filters: (a) vocal detection; (b) palmas detection; (c) guitar playing technique classification.

Figure 6 (a) shows the average activation of the convolutional filters of the first and second layer, for images which are classified as containing / not containing vocals.

It can be seen that each filter tends to have a higher activation for one of the two classes. This difference in average activation is more apparent in the second layer. A similar behavior can be observed for the palmas detection (Figure 6 (b)) and guitar playing style classification (Figure 6 (c)) tasks. This observation is furthermore reflected in the output of selected filters for frames representative of the output classes (Figure 5). Consequently, each filter seems to specialize in one of the two output classes.

3 Exploratory corpus study

In this section, we employ the previously described structural segmentation backend to explore a large corpus of flamenco music recordings. More specifically, we demonstrate how the automatically extracted annotations can be used to (a) augment editorial meta-data with meaningful content-based descriptors and visualizations, (b) discover genre-specific characteristics and evaluate their potential for music information retrieval tasks, and (c) illustrate music theoretical concepts in a large-scale data-driven context. Our goal is to focus on computationally feasible analysis experiments and tasks that would require immense effort when conducted manually.

First, we show that the visualization of the automatically extracted structure of a recording yields a concise overview of its content and can reveal interesting musical properties. Then, we compute global statistics over the corpus with respect to instrumentation and describe how the backend can be used to automatically identify a cappella and instrumental recordings. At a next stage, we examine how styles differ with respect to instrumentation and respective structure and investigate the potential use of the segmentation backend in automatic style classification systems. In addition, we illustrate the relationship between style, instrumentation and tonality via an example of three popular styles with distinct harmonic characteristics.

3.1 Corpus

We focus on Version 1.1 of corpusCOFLA111http://www.cofla-project.com/?page_id=170 [KDBMG16], a large collection of commercial recordings. This dataset was recently assembled in order to create a representative set of classical flamenco recordings that will enable the realization of computational analysis tasks. It encompasses a number of renown anthologies, together with their editorial meta-data. The annotated styles, which exhibit varying levels of detail and sub-style distinction, were classified into categories and artist names were assigned unique identifiers. An overview of the statistics of the corpus is given in Table 4 and the ten most frequently occurring styles are shown in Table 5.

no. recordings 1594
no. styles 86
tracks without style information 109
tracks containing multiple styles 104
no. singers 364
no. anthologies 10
Table 4: Statistics of the corpusCOFLA.
style no. of recordings
Fandangos 203
Soleares 150
Siguiriyas 124
Bulerías 120
Malagueñas 73
Tangos 59
Alegrías 43
Granaínas 39
Tarantas 37
Tientos 34
Table 5: 10 most frequently occurring styles in the corpusCOFLA.

3.2 Visualization of the structural annotations

Before proceeding with the analysis of the corpus as a whole, we show on the basis of two examples, how the output of the classifiers can provide a compact, content-based visualization of a flamenco recording.

Figure 7 (a) presents the automatic segmentation of the song Me fuí detras de los míos, performed by Manuel Sordera. This particular recording, which contains two styles, starts with the tientos style and ends with a tango. It is not uncommon that these two styles are encountered in a single performance, in this particular order. As a matter of fact, they are closely related styles because they follow a common harmonic progression and rhythmic structure. However, a tango is performed at a faster tempo with stronger rhythmic accentuation.

The visualization illustrates some further interesting differences between these two styles in this recording. The tientos part, which lasts from the beginning until approximately second , is characterized by long vocal segments, with guitar interludes alternating between picked and strummed sections. With the onset of the tangos part, the palmas set in to emphasize the underlying rhythmic pattern and the vocal sections become shorter. The guitar accompaniment is limited to strumming, which, due to its percussive nature, further increases the accentuation of the rhythm. Figure 7 (b) shows the same form of representation for a recording containing only tientos. Since in this recording, there is no modulation to tangos, there are no palmas and the guitar moves between picked and strummed sections until the end of the recording.

Figure 7: Example of visualizing structure: (a) tientos - tangos and (b) tientos.

The proposed visualization can be combined with other content-based descriptors that cover different musical aspects. For example, in Figure 8, structural annotations are combined with descriptors related to melodic repetition. Specifically, the arcs connecting vocal segments correspond to the five highest similarity values among all vocal sections. The thickness of the arc is proportional to the respective similarity value, which is computed using the alignment method in [KPMDB15]. It can be seen that vocal sections , , , and are of similar melodic content and that section appears to be a repetition of section .

Figure 8: Example of visualizing structure and vocal section similarity.

3.3 Global statistics

The extraction of the automatic annotations for all the recordings of the collection enables the computation of global statistics related to the presence of different instrumentation scenarios. Figure 9 shows the distribution of vocals, palmas and picked and strummed guitar, computed over all recordings of the collection. It can be seen, that only of the analyzed frames are estimated to contain vocals. This is an interesting observation, because flamenco is known to be heavily centered around the singing voice, the core element and origin of the flamenco tradition. However, at the same time, this is a meaningful result, because many performances contain long, solo guitar introductions and interludes. The remaining of solo guitar sections can be further divided to of strummed and of picked guitar. This observation can be explained by the fact that the reported distribution refers to solo guitar sections only, where guitarists are given the opportunity to perform the so-called falsetas, i.e., long melodic guitar introductions or interludes with an autonomous compositional identity. Falsetas ([Gre73, MK17]) do usually contain dominant melodic lines, and are mainly picked, rather than strummed. Furthermore, palmas are detected in of all frames. Even though there are no strict rules about the presence of palmas, they are unlikely to be found in some of the styles played in slow tempo, including fandangos, soleares, siguiriyas and malagueñas, which comprise a large part of the corpus.

Figure 9: Global corpus statistics.

3.4 Discovery of instrumental and a cappella recordings

Although a flamenco music performance usually contains both singing voice and guitar accompaniment, certain styles, which belong to the tonás family, are traditionally performed a cappella. These styles are of particular interest to musicological studies, since they are assumed to represent the origin from which flamenco evolved to its present form. Consequently, a number of musicological and computational studies have targeted their particular characteristics [MGG08, Tho85, KGG14]. However, assembling a representative corpus of a cappella flamenco recordings is not a trivial task, because the editorial metadata of commercial recordings do not contain information on the existence or absence of guitar accompaniment and style annotations are most often missing in online repositories and audio sharing platforms. As a support to this claim, so far, there only exists one such dataset of manually assembled recordings of the tonás family222https://www.upf.edu/web/mtg/tonas.

Here, we aim to discover automatically a cappella recordings within the corpusCOFLA collection. To this end, we begin by assuming that, due to the absence of the guitar, the frames of an a cappella recording will be mostly classified to the vocal class, taking into account a of classification error. Therefore, to retrieve the a cappella recordings, we first set a high threshold for the percentage of vocal frames that are detected in a recording and we evaluate the results. Then we lower the threshold by each time and we repeat the retrieval operation. Each time, we evaluate the results by reporting the retrieval precision, which indicates the fraction of relevant instances among all retrieved recordings. We also note the total number of retrieved songs.

Figure 10: Discovery of a cappella recordings: Retrieval precision and total number of retrieved songs for varying thresholds.

The results in Figure 10 show that, when the threshold value is at least , none of the retrieved recordings contain instrumental accompaniment and are therefore true hits. In this way, we identify a cappella recordings. When the threshold is further decreased, some more a cappella recordings are retrieved to the expense of an increasing number of false positives. A manual inspection showed, that the false positives are performances which contain a guitar background in the presence of vocals, i.e., not in solo guitar interludes, hence the high percentage of vocal frames. This is also due to the setup of our vocal classifier, according to which, a frame is classified in the vocal class as long as it contains vocals, i.e, even when there exists simultaneous guitar accompaniment. For a threshold value of , we retrieve a total of recordings, out of which are a cappella.

Similarly, we approach the discovery of instrumental recordings by analyzing songs with a low percentage of vocal frames. As in the previous experiment, we compute the retrieval precision for varying threshold values. In the case of instrumental recordings, the threshold refers to the maximum allowable percentage of vocal frames, i.e., the percentage below which a recording is considered to be instrumental. The results in Figure 11 (black curves) show that for threshold values below , around of the retrieved recordings are instrumental. However, the precision drops rapidly when the threshold increases further. A closer inspection revealed, that the large majority of false positives originate from the anthology Historia del flamenco. This collection contains mainly historic field recordings of poor quality. In some of these tracks, the classifier failed to detect most of the singing voice sections, yielding a low percentage of vocal frames. Based on this observation, we repeated the experiment, excluding this particular anthology from the search space. In this new scenario (Figure 11, red curve), all retrieved recordings are instrumental performances for threshold values up to . In this way, we were able to identify instrumental recordings in the corpus. As mentioned in Section 1, instrumental performances are rare in classical flamenco and it can be stated that solo guitar pieces have only recently gained popularity [Spe10].

Figure 11: Discovery of instrumental recordings: Retrieval precision and total number of retrieved songs for varying thresholds.

These two experiments have shown that the structural annotations can be used for the semi-automatic (if perfect precision is required) and automatic (if a percentage of false alarms can be tolerated) identification of a cappella and instrumental recordings in large flamenco collections. The resulting metadata represent an important augmentation to the commonly provided editorial meta-data scheme, which usually lacks this type of information. Furthermore, the creation of collections of a cappella recordings with minimal manual intervention is of great importance for scaling existing computational approaches. Furthermore, the automatic tagging of instrumental recordings is important for genre-specific recommender systems, because solo guitar flamenco recordings may appeal to a different audience than classical flamenco, which mainly evolves around the singing voice.

3.5 Differences in structure and instrumentation across styles

Flamenco styles are distinguishable by a complex set of harmonic, melodic and rhythmic characteristics. Computational approaches to automatic style classification have so far been limited to a small number of substyles which can be identified based on common melodic templates [KDB17, MGGDB16]. Consequently, the problem of automatically detecting the style of a flamenco recording remains largely unresolved [KDB16].

The occurrence of different instrumentations and the existence of prototypical structures across styles have not yet been formally studied. Therefore, in order to gain a better insight into style-specific characteristics and evaluate the potential use of the proposed structural segmentation backend for the purposes of automatic style classification, we investigate if (a) certain types of instrumentation are more likely to occur in particular styles and (b) songs belonging to the same style are similar in structure.

To this end, each song is represented as a feature vector, where each feature dimension stands for the percentage of vocals, palmas and strummed and picked guitar. This representation is used to assess differences across styles on a multi-dimensional plane. In addition, we investigate instrumentation as a temporal structure, to explore intra- and inter-style structural similarity. Specifically, the evolution of instrumentation throughout a recording is represented by a sequence of feature vectors, where each vector encodes the classification decisions of the backend at the short-term level. Both representations are applied on the ten most frequently occurring styles of the corpus (Table 5) and recordings with multiple styles are excluded.

3.5.1 Pair-wise visualization of song-level descriptors

In a first experiment, we visualize pairs of styles with respect to the song-level percentages of palmas, vocals and picked guitar, and, for some cases, we observe that pairs of these features exhibit style-specific behavior. Two examples are shown in Figure 12. The left scatter plot shows the percentage of picked guitar vs. the percentage of palmas for the styles of bulerías and malagueñas. It can be seen that the two styles are almost linearly separable on the respective two-dimensional plane. While most bulerías exhibit a high percentage of palmas, the palmas are largely absent in the malagueñas style. Regarding the guitar playing technique, picking is dominant in malagueñas, whereas the bulerías mainly contain strummed guitar.

Using our prior knowledge on a cappella styles, we now compare recordings of bulerías to the tonas family, which encompasses most a cappella styles of flamenco music. Figure 12

(right) presents the joint distribution of the percentage of vocals and

palmas for the two classes. As expected, the a cappella recordings contain a high percentage of vocal frames, mostly above and do not contain palmas

. Some outliers can be explained by the fact that one of the a cappella styles, the

martinete, is usually accompanied by stokes of a hammer on a anvil. A manual inspection revealed that this accompaniment was often mistakenly classified as either palmas or picked guitar (due to the harmonic nature of the produced sounds). The combination of the low percentage of vocals and the presence of palmas in the bulerías leads again to a practically linearly separable visualization of the two styles.

Figure 12: left: % of picked guitar frames (x-axis) and % of palmas frames (y-axis) for bulerías and malagueñas; right: % of vocal frames (x-axis) and % of palmas frames (y-axis) for bulerías and tonás.

3.5.2 Two-dimensional distance matrix visualization

Based on these observations, we now aim to understand how well this feature representation of a single vector per recording can discriminate the ten most frequent styles in the corpus. Therefore, we compute pair-wise Euclidean distances among all involved recordings and visualize the two-dimensional ForceAtlas2-layout [JVHB14] of the resulting distance matrix. The ForceAtlas2 algorithm models distances as forces, causing repulsion between instances. Consequently, the resulting layout reflects the similarity between instances via the proximity in the designed two-dimensional space.

The resulting layout is shown in Figure 13. It can be seen that two clusters are formed, one of which contains mainly recordings belonging to the styles of bulerías and tangos. The remaining styles are located in a second cluster. Within this group, we observe a slight tendency of the malagueñas being clustered together and the fandangos being closer to the second cluster. The algerías appear to spread over both groups.

Figure 13: Force Atlas 2 layout of pairwise distances between song-level descriptors.

3.5.3 Structural similarity across styles

At a next step, we investigate intra-style and inter-style structural similarity, i.e., we extend the previous experiment by taking into account the aforementioned temporal representation of the extracted annotations. Each recording is now represented as a -dimensional sequence, where each dimension corresponds to the frame-wise binary decisions of the respective CNN. For each pair of sequences, a dynamic time warping (DTW) [RK04] cost is computed. To avoid excessive time stretching during the DTW operation, the global path constraint in [Ita75] is employed. Consequently, the distance between two sequences is equal to the warping cost of their alignment, that may be infinite due to the adopted global constraint. In order to reduce the computational burden of the DTW computations, each sequence is sub-sampled prior to the application of the alignment operation. In particular, one -d vector is preserved every second. Finally, as it was the case with the previous experiment, the ForceAtlas2 layout is generated.

Figure 14: Force Atlas 2 layout of pairwise structural distances: (a) 10 most frequently occurring styles; (b) bulerías and siguiriyas.

The resulting visualization is now shown in Figure 14 (a). The structure of the layout is similar to the results of the previous experiment (Figure 13). Again, bulerías and tangos appear separately from the other styles. However, the internal structure of the second, large cluster, differs slightly from the previous experiment. In particular, the soleares and seguiriyas, which were previously spread across the cluster, now tend to group together, indicating temporal structural similarity.

Even though the experiments of Sections 3.5.2 and 3.5.3 show, that the (dis)similarity of instrumentation-related representations is not sufficient to discriminate among all ten investigated styles, they do however demonstrate the potential of distinguishing between pairs of styles, as it was shown in Figure 12. This is also supported by the example of Figure 14 (b), where bulerías and siguiriyas form clearly separated clusters, when the DTW cost is computed for all pairs of recordings of these two styles.

3.5.4 Distance-based style retrieval

Figure 15: MRR for the distance-based retrieval of the ten most frequent styles, when using song-level descriptors vs. structural annotations.

In order to enhance our understanding of the discriminative capabilities of the computed distance values, we now formulate a distance-based retrieval task. Namely, for each recording, we retrieve the ten most similar recordings and analyze their style with respect to the style of the query. The success of the retrieved results is quantified, for each style, via its mean reciprocal rank, , which is defined as follows:

(1)

where denotes the number of recordings belonging to the style and is the highest position in the list of returned results of the result that matches the style of the query, when the recording of the style is used as a query, assuming that the top result is found in position . The MRR values have been computed for both types of representations, i.e., the single-vector representation per recording and the feature sequence one. The results in Figure 15 indicate that the DTW cost yields, for most styles, a higher MRR. Interestingly, in both cases, the highest MRR is observed for the style of bulerías. This result is in line with the observations drawn in previous sections, where it was shown that this particular style exhibits certain characteristics that distinguish it from most other styles. It should also be noted that the reported results are influenced by the imbalance of styles in the corpus. For example, the fandangos style accounts for nearly half of the search space. This explains its relatively high MRR, despite the rather low separability of this style, as it was shown in the previous sections via the ForceAtlas2 approach.

In conclusion, this exploratory investigation has shown, that song-level and local descriptors related to instrumentation hold valuable information with respect to style identity. Even though these features alone are not sufficient to distinguish between the styles investigated in this study, they do however show potential for their application, together with features related to other musical dimensions, in automatic style detection systems. Furthermore, they allow us to formalize typical characteristics of particular styles in a data-driven manner.

3.6 Tonality across instrumentation and styles

As a last experiment, we illustrate flamenco-specific music theoretical concepts about the relationship between style, instrumentation and tonality in a large-scale data analysis environment. In flamenco, we encounter, apart from major and minor mode, a third mode, often referred to as the flamenco mode. Most styles are restricted to on of these three modes. For example, alegrías, are set in major mode, and seguiriyas are set in flamenco mode. The fandangos are an exception, because they are described in the literature as bimodal, in the sense that guitar solo sections are performed in flamenco mode, while singing voice sections are performed in major [Fer12]. However, previous computational studies [KDB16] on a small dataset have indicated a weaker major mode identity during singing voice sections for this style, compared to other styles. Here, we study this phenomenon in a data-driven approach on a larger scale. Specifically, we analyze the estimated tonality for three styles, i.e., the fandangos, seguiriyas and alegrías, in vocal, strummed and picked guitar sections.

A common method to determine the tonality of a piece is to extract its pitch class profile and compare it to tonality-specific pitch class templates [TM08]. Pitch class profiles quantify the occurrence of scale degrees throughout a composition. The statistical occurrence of scale degrees is closely related to the perceived tonality [Che00] and pitch class templates are prototypical scale degree occurrence patterns, specific to a particular tonality. These are either extracted from large annotated corpora or are estimated through listening experiments [KS79, TM08]. If the pitch class profile of a given piece exhibits a high correlation with the pitch class template, it is likely that the piece will be perceived as belonging to the tonality associated with the template.

In line with this approach, we estimate the tonality of a flamenco recording by correlating its pitch class profile with tonality-specific templates. To this end, we first extract harmonic pitch class profiles (HPCP) [Góm06] on a frame-level basis (non-overlapping frames of length samples), and then average over vocal, strummed and picked guitar segments. In this way, we create three pitch class profiles for each recording.

For the major mode, we use the template from [TM08]. Furthermore, we extract a flamenco mode template from all recordings in the corpus belonging to the soleares style. This process requires key normalization per recording, in order to account for possible transposition among performances. Therefore, we normalize each pitch class histogram to a randomly selected reference example, by computing the correlation between circularly shifted versions of the histogram and the reference. For each recording, the pitch shift yielding the highest correlation is applied to all HPCP vectors.

We then compute the correlation between the pitch class profiles of the recordings under study and both templates. Here, we again normalize each profile to the key of the respective template. The results, broken down into styles and instrumentation, are shown in the form of kernel density plots in Figure 16. High values on the x-axis correspond to a high correlation with the major mode template and consequently indicate a strong major mode identity. Similarly, high y-values correspond to a strong flamenco mode identity.

It can be seen, that across instrumentation, the algerías show a higher correlation with the major mode template, and the siguiriyas with the flamenco mode template. This observation is in line with theoretical studies [Fer04]. For the style of siguiriyas

, we observe less variance for picked guitar sections, compared to vocal and strummed guitar sections. In the plots displaying the

fandango recordings, we can see the bimodality of the style. Specifically, both picked and strummed guitar sections exhibit a higher correlation with the flamenco mode template. In contrast, the vocal sections yield a higher correlation with the major mode template. However, similar to the findings in [KDB16], we observe a weaker correlation and a larger variance, compared to the alegrías.

Figure 16: Kernel density plots displaying the correlation of pitch class profiles with pitch class templates across styles and instrumentation.

This experiment demonstrates the theoretically formalized bimodality of the fandango style, and, at the same time, reveals two interesting aspects related to tonality in flamenco. First, we observe a stronger tonal identity in picked compared to strummed guitar sections in seguiriyas. Secondly, we confirm the observations from [KDB16] on a larger scale, which indicate that the vocal sections of the fandangos exhibit a weaker major mode identity compared to the alegrías.

4 Conclusion

The first goal of this paper was to develop state-of-the-art classifiers that are capable of providing reliable, content-based instrumentation-related annotations of flamenco recordings. The second goal was to use the extracted annotations to enable the realization of musicological findings that would otherwise require time-consuming manual procedures, and verify, at a large scale, via computational means and a data-driven approach, existing musicological observations. The paper showed that the developed annotation backend is reliable enough to achieve the goals that were set and this was demonstrated via a number of related experiments that investigated the discriminative capability among genres of higher-level features drawn from the extracted low-level descriptors of instrumentation. The reported results focused both on pairwise style relationships and multi-style associations, via a variety of complementary approaches, ranging from visualizations of varying complexity to a style-based retrieval engine. In all experiments, emphasis was given on explaining the findings from a computational and musicological perspective. Our computational study is the first of its kind in the context of flamenco music. The developed backend is provided in the form of pre-trained models for the sake of reproducibility of results and as a tool for studies in related ethnomusicological tasks.

Acknowledgements

  • We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.

  • This research has been partly funded by the Junta de Andalucía and the FEDER funds of the European Union, project COFLAII under the grant identifier P12-TIC-1362 and the international mobility grant of the University of Seville (plan propio de investigación y transferencia 2017-1.3-A).

References

  • [Ber03] Jan Beran. Statistics in musicology. CRC Press, 2003.
  • [Che00] Elaine Chew. Towards a mathematical model of tonality. PhD thesis, Massachusetts Institute of Technology, 2000.
  • [Con10] Darrell Conklin. Discovery of distinctive patterns in music. Intelligent Data Analysis, 14(5):547–554, 2010.
  • [dCT11] Trevor de Clercq and David Temperley. A corpus analysis of rock harmony. Popular Music, 30:47–70, 2011.
  • [Fer04] Lola Fernández. Flamenco Music Theory: Rhythm, Harmony, Melody, Form. Mel Bay Publications, 2004.
  • [Fer12] Lola Fernández. La bimodalidad en las formas del fandango y en los cantes de levante: origen y evolución. Revista de Investigación sobre Flamenco La Madrugá, 5(1):37–53, 2012.
  • [GD04] Christophe Garcia and Manolis Delakis. Convolutional face finder: A neural architecture for fast and robust face detection. IEEE Transactions on pattern analysis and machine intelligence, 26(11):1408–1423, 2004.
  • [GMDBGM14] Francisco Gómez-Martín, José-Miguel Díaz-Báñez, Emilia Gómez, and Joaquin Mora. Flamenco music and its computational study. In BRIDGES: Mathematical Connections in Art, Music, and Science, pages 119–126, Seoul, Korea, 14/08/2014 2014.
  • [Góm06] Emilia Gómez. Tonal description of polyphonic audio for music content processing. INFORMS Journal on Computing, 18(3):294–304, 2006.
  • [Gre73] Juan D Grecos. The flamenco guitar. S. Fox Pub. Co., 1973.
  • [HCE17] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 131–135, New Orleans, USA, 2017.
  • [HKL17] Yoonchang Han, Jaehun Kim, Kyogu Lee, Yoonchang Han, Jaehun Kim, and Kyogu Lee. Deep convolutional neural networks for predominant instrument recognition in polyphonic music. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 25(1):208–221, 2017.
  • [Ita75] Fumitada Itakura. Minimum prediction residual principle applied to speech recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 23(1):67–72, 1975.
  • [JVHB14] Mathieu Jacomy, Tommaso Venturini, Sebastien Heymann, and Mathieu Bastian. Forceatlas2, a continuous graph layout algorithm for handy network visualization designed for the gephi software. PLOS ONE, 9(6):e98679, 2014.
  • [KB14] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint:1412.6980, 2014.
  • [KDB16] Nadine Kroher and José-Miguel Díaz-Báñez. Towards flamenco style recognition: the challenge of modelling the aficionado. In Proceedings of the 6th International Workshop on Folk Music Analysis, pages 61–63, Dublin, Ireland, 2016.
  • [KDB17] Nadine Kroher and José-Miguel Díaz-Báñez. Audio-based melody categorisation: exploring signal representations and evaluation strategies. Computer Music Journal, 41(4):1–19, 2017.
  • [KDBMG16] Nadine Kroher, José-Miguel Díaz-Báñez, Joaquin Mora, and Emilia Gómez. Corpus cofla: a research corpus for the computational study of flamenco music. Journal on Computing and Cultural Heritage (JOCCH), 9(2):10:1–10:21, 2016.
  • [KG16] Nadine Kroher and Emilia Gómez. Automatic transcription of flamenco singing from polyphonic music recordings. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(5):901–913, 2016.
  • [KGG14] Nadine Kroher, Emilia Gómez, Catherine Guastavino, Francisco Gómez, and Jordi Bonada. Computational models for perceived melodic similarity in a cappella flamenco singing. In Proceedings of the International Society of Music Information Retrieval Conference (ISMIR), pages 65–70, Taipei, Taiwan, 2014.
  • [KPMDB15] Nadine Kroher, Aggelos Pikrakis, Jesús Moreno, and José-Miguel Díaz-Báñez. Discovery of repeated vocal patterns in polyphonic audio: A case study on flamenco music. In 23rd European Signal Processing Conference (EUSIPCO), pages 41–45, Nice, France, 2015.
  • [KS79] Carol L Krumhansl and Roger N Shepard. Quantification of the hierarchy of tonal functions within a diatonic context. Journal of experimental psychology: Human Perception and Performance, 5(4):579–594, 1979.
  • [KSH12] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (NIPS), pages 1097–1105, Lake Tahoe, USA, 2012.
  • [LLS15] Haoxiang Li, Zhe Lin, Xiaohui Shen, Jonathan Brandt, and Gang Hua. A convolutional neural network cascade for face detection. In

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    , pages 5325–5334, Boston, USA, 2015.
  • [Man86] Peter Manuel. Evolution and structure in flamenco harmony. Current Musicology, (42):46–57, 1986.
  • [MGG08] Joaquín Mora, Francisco Gómez, Emilia Gómez, Francisco Escobar-Borrego, José-Miguel Díaz-Báñez, Juan Jesús Cabrera, Jordi Bonada, and Francisco Gómez Martín. Comparative analysis of a cappella flamenco cantes. In Fourth Conference on Interdisciplinary Musicology (CIM08), Thessaloniki, Greece, 2008.
  • [MGGDB16] Joaquín Mora, Francisco Gómez, Emilia Gómez, and José-Miguel Díaz-Báñez. Melodic contour and mid-level global features applied to the analysis of flamenco cantes. Journal of New Music Research, 45(2):145–159, 2016.
  • [MK17] Inmaculada Morales and Nadine Kroher. Melodic pattern cross-occurrences between guitar falsetas and singing voice in flamenco music. In Proceedings of the International Workshop on Folk Music Analysis (FMA), pages 119–120, Málaga, Spain, 2017.
  • [PBBD17] Maria Panteli, Rachel Bittner, Juan Pablo Bello, and Simon Dixon. Towards the characterization of singing styles in world music. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 636–640, New Orleans, USA, 2017.
  • [PKKDB16] Aggelos Pikrakis, Yannis Kopsinis, Nadine Kroher, and José-Miguel Díaz-Báñez. Unsupervised singing voice detection using dictionary learning. In 24th European Signal Processing Conference (EUSIPCO), pages 1212–1216, Budapest, Hungary, 2016.
  • [RJ93] Lawrence R Rabiner and Biing-Hwang Juang. Fundamentals of speech recognition. PTR Prentice Hall, 1993.
  • [RK04] Chotirat Ann Ratanamahatana and Eamonn Keogh. Everything you know about dynamic time warping is wrong. In Third Workshop on Mining Temporal and Sequential Data, pages 22–25, Seattle, USA, 2004.
  • [SLY13] Liming Song, Ming Li, and Yonghong Yan. Automatic vocal segments detection in popular music. In 9th International Conference on Computational Intelligence and Security (CIS), pages 349–352, Girne, Cyprus, 2013.
  • [Spe10] Claire Christine Spera. Flamenco nuevo: Tradition, evolution and innovation. University of Southern California, 2010.
  • [THM09] Jonatan Taminau, Ruben Hillewaere, Stijn Meganck, Darrell Conklin, Ann Nowé, and Bernard Manderick. Descriptive subgroup mining of folk music. In

    2nd International Workshop on Machine Learning and Music (MML)

    , pages 1–6, Bled, Slovenia, 2009.
  • [Tho85] Barbara Thompson. Flamenco: A tradition in évolution. The World of Music, 27(3):67–80, 1985.
  • [TM08] David Temperley and Elizabeth West Marvin. Pitch-class distribution and the identification of key. Music Perception: An Interdisciplinary Journal, 25(3):193–212, 2008.
  • [USG14] Karen Ullrich, Jan Schlüter, and Thomas Grill. Boundary detection in music structure analysis using convolutional neural networks. In Proceedings of the International Society of Music Information Retrieval Conference (ISMIR), pages 417–422, Taipei, Taiwan, 2014.
  • [VdH13] Anja Volk and W Bas de Haas. A corpus-based study on ragtime syncopation. In Proceedings of the International Society for Music Information Retrieval Conference, pages 163–168, Curitiba, Brazil, 2013.
  • [VdODS13] Aaron Van den Oord, Sander Dieleman, and Benjamin Schrauwen. Deep content-based music recommendation. In Advances in neural information processing systems (NIPS), pages 2643–2651, Lake Tahoe, USA, 2013.
  • [Whi94] John D White. Comprehensive musical analysis. Scarecrow Press, 1994.