Overview of Tasks and Investigation of Subjective Evaluation Methods in Environmental Sound Synthesis and Conversion

08/27/2019
by   Yuki Okamoto, et al.
0

Synthesizing and converting environmental sounds have the potential for many applications such as supporting movie and game production, data augmentation for sound event detection and scene classification. Conventional works on synthesizing and converting environmental sounds are based on a physical modeling or concatenative approach. However, there are a limited number of works that have addressed environmental sound synthesis and conversion with statistical generative models; thus, this research area is not yet well organized. In this paper, we review problem definitions, applications, and evaluation methods of environmental sound synthesis and conversion. We then report on environmental sound synthesis using sound event labels, in which we focus on the current performance of statistical environmental sound synthesis and investigate how we should conduct subjective experiments on environmental sound synthesis.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

page 5

07/09/2020

RWCP-SSD-Onomatopoeia: Onomatopoeic Word Dataset for Environmental Sound Synthesis

Environmental sound synthesis is a technique for generating a natural en...
02/11/2021

Onoma-to-wave: Environmental sound synthesis from onomatopoeic words

In this paper, we propose a new framework for environmental sound synthe...
12/01/2021

Environmental Sound Extraction Using Onomatopoeia

Onomatopoeia, which is a character sequence that phonetically imitates a...
04/15/2019

Proximal binaural sound can induce subjective frisson

Sound frisson is a subjective experience wherein people tend to perceive...
09/06/2020

Overview and Evaluation of Sound Event Localization and Detection in DCASE 2019

Sound event localization and detection is a novel area of research that ...
09/14/2018

A Multi-Stage Algorithm for Acoustic Physical Model Parameters Estimation

One of the challenges in computational acoustics is the identification o...
09/25/2012

Environmental Sounds Spectrogram Classification using Log-Gabor Filters and Multiclass Support Vector Machines

This paper presents novel approaches for efficient feature extraction us...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Sound synthesis and conversion are techniques for generating a natural sound using a statistical model that associates input information with the generated sound. Sound synthesis and conversion methods with the aim of generating speech or music have been widely developed [1, 2, 3]. Recently, some researchers have also developed methods for environmental sound synthesis and conversion that can be applied to support movie and game production [4], the generation of content for virtual reality (VR) [5], and data augmentation for sound event detection and scene classification [6]. Many studies on environmental sound synthesis and conversion have taken a physical modeling or concatenative approach [7, 8, 6]

. On the other hand, there have been fewer studies on environmental sound synthesis and conversion based on statistical generative models such as deep learning approaches. To the best of our knowledge, there is no literature giving an overview of the problem definitions and evaluation methods for environmental sound synthesis and conversion. Moreover, there have been no investigation of subjective evaluation methods for environmental sound synthesis and conversion.

In this paper, we therefore review problem definitions, applications, and evaluation methods of environmental sound synthesis and conversion. We then report on environmental sound synthesis based on WaveNet [9], which successfully synthesizes human voices, to discuss the current performance of statistical environmental sound synthesis. Moreover, we investigate subjective evaluation methods of environmental sound synthesis.

The remainder of this paper is structured as follows. In Sec. 2, we review problem definitions of environmental sound synthesis and conversion, their applications, and evaluation methods. In Sec. 3, subjective experiments carried out to evaluate the performance of sound event synthesis using a WaveNet-based method are reported. Finally, we summarize and conclude this paper in Sec. 4.

Figure 1: Problem definition of sound scene synthesis
Figure 2: Problem definition of environmental sound synthesis using sound event labels

2 Problem Definitions of Environmental Sound Synthesis and Conversion

In this section, we review applications, problem definitions, and evaluation methods of environmental sound synthesis and conversion, specifically environmental sound synthesis using event or scene labels (Sec. 2.1), environmental sound synthesis using onomatopoeic words (Sec. 2.2), environmental sound conversion (Sec. 2.3), and environmental sound synthesis/conversion using multimedia (Sec. 2.4).

2.1 Environmental Sound Synthesis Using Sound Event and Scene Labels

When providing movies or games with background sounds or sound effects, we need to listen to many sounds in a large sound database and select the most suitable one for the scene or sound event, which is a time-consuming part of movie or game production. To address this issue, a statistical method for synthesizing an environmental sound well representing a sound event or scene, which utilizes the sound event or scene labels as below as an input, has been proposed [10]. Figures 1 and 2 illustrate the processes of environmental sound synthesis using the sound event or scene labels as the inputs of the systems, where we call these research tasks sound event synthesis (SES) and sound scene synthesis (SSS), respectively. Another issue is that the construction of an environmental sound dataset is very time-consuming compared with the construction of a speech or music dataset [11]

. In recent studies, environmental sound analysis based on deep neural networks has required a large number of sounds to achieve a reasonable performance. To overcome this problem of a shortage of environmental sound datasets, SES and SSS can be applied for data augmentation in environmental sound analysis.

To generate environmental sounds by a statistical approach, Kong et al. [10] have proposed a method of environmental sound synthesis utilizing a conditional SampleRNN [12]

with sound scene labels represented as one-hot vectors.

A method of evaluating synthesized environmental sounds is an important subject in this research area. When we apply SES or SSS to data augmentation for sound event detection or acoustic scene classification, it is reasonable to evaluate the methods of SES or SSS via their event detection or scene classification performance with augmented data. On the other hand, in the case of utilizing the sound synthesized by SES or SSS itself, it has not been investigated in detail how the synthesis method should be evaluated. In this paper, we focus on the subjective evaluation method for environmental sound synthesis in Sec. 3.

Figure 3: Problem definition of environmental sound synthesis using onomatopoeic words

On the other hand, the subjective evaluation of sounds is very time-consuming; thus, it is desirable to test methods for environmental sound synthesis and conversion with an objective evaluation of synthesized sounds. There are some methods of objective evaluation such as the perceptual evaluation of speech quality (PESQ) [13], perceptual objective listening quality analysis (POLQA) [14], and perceived evaluation of audio quality (PEAQ) [15], which are used for the evaluation of the speech quality in telecommunication or audio quality via codecs. However, the objective evaluation method for synthesized or converted environmental sounds has not been investigated.

2.2 Environmental Sound Synthesis Using Onomatopoeic Words

The SES and SSS discussed in Sec. 2.1 control synthesized environmental sounds only using the sound event or scene labels; thus, they cannot control synthesized sounds without types of sound or scenes. For instance, when synthesizing the sound of a car horn, it cannot be determined in advance whether SES will synthesize a horn sound with a continuous high tone (e.g, peeeeeeeeee) or one with an intermittent low tone (e.g, beep beep beep). To control synthesized environmental sounds more finely, we can apply environmental sound synthesis using onomatopoeic words as an input of the system, as shown in Fig. 3. For SES using onomatopoeic words, Ikawa et al. [16] have proposed a method that converts onomatopoeic words to wave forms of environmental sounds using an encoder–decoder model.

Figure 4: Problem definition of environmental sound conversion
Sound length 1–2 s
Sampling rate 16000
Wavefrom encoding 16-bit linear PCM (real sound)
8-bit -law (synthesized sound)
Filter size 2
Learning rate 0.001
Batch size 5
Receptive field 64 ms
# dilations
# residual channels 32
# dilation channels 32
# quantization channels 256
# skip channels 512
Table 2: Number of synthesized sounds used for subjective test
Experiment # labels # samples # listeners # total
in each label samples
Exp. 1 10 5 24 1,200
Exp. 2 10 4 24 960
Exp. 3 10 2 24 480
Table 1: Experimental conditions

2.3 Environmental Sound Conversion

Sound event synthesis using onomatopoeic words is a flexible way of synthesizing environmental sounds; however, it is still difficult to control the generated environmental sounds as intended. One way to address this problem is to synthesize environmental sounds not with sound event labels or onomatopoeic words but with the environmental sound or voice as the input of the system, as shown in Fig. 4. We call this kind of task a sound event conversion (SEC) or sound scene conversion (SSC). When we have some background sounds or sound effects but they are not suitable for the movie or game, environmental sound conversion can also be applied to obtain desirable sounds. For instance, when we have the horn sound of car X and a video including car Y, we can convert the horn sound of car X to that of car Y using SEC without re-recording the horn sound of car Y.

To convert environmental sounds to other audio signals, Grinstein et al. [17] and Mital [18] have applied a neural-style transfer-based method [19], which enables the “style” and “content” of an audio to be independently manipulated and copied to another audio signal.

Figure 5: Confusion matrix of classification accuracy for original audio samples in terms of recall

2.4 Environmental Sound Synthesis/Conversion Using Multimedia

Some researchers have addressed environmental sound synthesis and conversion using multimedia information as an input such as images. For instance, Zhou et al. have proposed a method for synthesizing environmental sounds from images that is based on SampleRNN [5].

3 Investigation of Subjective Evaluation Method

3.1 Experimental Conditions

In this section, by evaluating SES using sound event labels based on the conditional WaveNet [9], we discuss the current performance of environmental sound synthesis and how we should conduct a subjective test to evaluate a method under development. For the evaluation, we considered 10 different sound events (manual coffee grinder, cup clinking, alarm clock ringing, whistle, maracas, drum, electric shaver, trash box banging, tearing paper, bell ringing) contained in the RWCP-SSD (Real World Computing Partnership-Sound Scene Database) [20]. We used a total of 1,000 samples (100 samples 10 sound events), in which 95 samples of each sound event were used for model training and the others were used for the subjective test. Table 2 shows the experimental conditions and parameters used for WaveNet. Samples of sounds synthesized by WaveNet are available at [21].

Figure 6: Confusion matrix of classification accuracy for synthesized audio samples in terms of recall
Figure 7: Spectrograms of real and synthesized environmental sounds

Many works on speech and music synthesis have been conducted using subjective tests to evaluate the quality of synthesized sounds. For example, in speech synthesis, speech intelligibility and naturalness are often used as evaluation metrics. On the other hand, there have been no works in which methods of subjective tests in environmental sound synthesis and conversion were investigated in detail; thus, we here discuss how we should conduct subjective tests for environmental sound synthesis. For synthesized sounds, it is important that (I) they are distinguishable from other types of environmental sound, (II) they are not distinguishable from real sounds, and (III) they have as high naturalness as real environmental sounds. On the basis of these considerations, we conducted the following experiments:

  • Experiment I: evaluation of intelligibility of synthesized sounds
    After listening to a synthesized sound, the listener selected a sound event label that best represented the sound. As a comparison, the listener also similarly evaluated real environmental sounds.

  • Experiment II: evaluation of distinguishability of real and synthesized sounds
    We conducted a preference AB test. After listening to a pair of real and synthesized sounds in random order, the listener selected the one that sounded more real.

  • Experiment III: evaluation of naturalness of synthesized sounds
    We conducted a five-scale mean opinion score (MOS) test. After listening to a real or synthesized sound presented randomly, the listener scored the naturalness from 1 (very unnatural as an environmental sound) to 5 (very natural as an environmental sound).

Experiments were conducted with 24 listeners (13 males and 11 females) in a quiet environment at Ritsumeikan University. In Table 2, the number of samples used in each experiment is listed. In the experiments, a Roland QUAD-CAPTURE UA-55 audio interface and SONY MDR-CD900ST headphones were used.

Figure 8: Recognition rate of real sounds

3.2 Experimental Results and Discussion

Experiment I: the classification results of real and synthesized sounds in terms of recall are shown in Figs. 5 and 6

, respectively. The averaged F-scores for real and synthesized sounds were 86.22% and 76.30%, respectively. From these results, synthesized drum sounds are classified with a similar performance to real sounds, whereas the synthesized sounds of a cup clinking and an electric shaver tended to be more often misclassified than the real sounds. Figure 

7 shows spectrograms of real and synthesized sounds. This indicates that the synthesized sound of an electric shaver does not have the fine structure of the spectrum, which has the real sound. Thus, the difference between the spectrograms of the sounds of an electric shaver and tearing paper is likely to be unclear, and this leads to the misclassfication. From the results of experiment I, it considered that this subjective test is particularly helpful for evaluating whether the method can reproduce distinguishable sounds even when they have similar characteristics.

Experiment II: listeners identified real sounds with an average accuracy of 82.71% as shown in Fig. 8. From this result, sounds synthesized by WaveNet do not have sufficiently high quality to be indistinguishable from real sounds. This indicates that the evaluation of the distinguishability of real and synthesized sounds can be used for the comparison of conventional methods and more sophisticated methods of environmental sound synthesis that will be developed.

Experiment III

: the average MOS score for the naturalness of synthesized and real sounds and its 95% confidence interval are shown in Fig. 

9. The results indicate that the synthesized sounds of the coffee grinder, clock, and maracas had similar naturalness scores to those of real sounds. On the other hand, for the sounds of the electric shaver and the trash box banging, there are large differences in the MOS scores between the synthesized and real sounds. We consider that this is because SES using WaveNet cannot reproduce the fine structure of the synthesized spectrum (e.g., the spectrum of the electric shaver in Fig. 7). Moreover, Figs. 5, 6, and 9 show that the listeners classified both the real and synthesized whistle sounds with reasonable performance, whereas there are large differences in the MOS scores between synthesized and real sounds. This means that the evaluation of intelligibility is not satisfactory for evaluating the quality of synthesized sounds.

Thus, we propose that methods of environmental sound synthesis should be evaluated not only by testing the intelligibility of synthesized sounds but also by testing the distinguishability of real and synthesized sounds and/or the naturalness of synthesized sounds.

Figure 9: MOS score for naturalness of original and synthesized sounds

4 Conclusion

In this paper, we presented the problem definitions of sound event synthesis, sound scene synthesis, and sound event and scene conversion. We then discussed the current performance of sound event synthesis and subjective evaluation methods of environmental sound synthesis. The evaluation experiments indicate that sounds synthesized by WaveNet do not yet have sufficiently high quality to be indistinguishable from real sounds. Moreover, on the basis of our experimental results, we consider that methods of environmental sound synthesis should be evaluated by testing not only intelligibility but also distinguishability and/or naturalness.

References

  • [1] H. Zen, K. Tokuda, and A. Black, “Statistical parametric speech synthesis,” Speech Communication, vol. 51, no. 11, pp. 1039–1064, 2009.
  • [2] S. H. Mohammadi and A. Kain, “An overview of voice conversion systems,” Speech Communication, vol. 88, pp. 65–82, 2017.
  • [3] J. P. Briot, G. Hadjeres, and F. Pachet, “Deep learning techniques for music generation – a survey,” arXiv preprint arXiv:1709.01620, 2017.
  • [4] D. B. Lloyd, N. Raghuvanshi, and N. K. Govindaraju, “Sound synthesis for impact sounds in video games,” Proc. Symposium on Interactive 3D Graphics and Games. ACM, pp. 55–61, 20111.
  • [5] Y. Zhou, Z. Wang, C. Fang, T. Bui, and T. L. Berg, “Visual to sound: Generating natural sound for videos in the wild,”

    Proc. IEEE Conference on Computer Vision and Pattern Recognition

    (CVPR)
    , pp. 3550–3558, 2018.
  • [6] J. Salamon, D. MacConnell, M. Cartwright, P. Li, and J. P. Bello, “Scaper: A library for soundscape synthesis and augmentation,” Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 344–348, 2017.
  • [7] D. Schwarz, “State of the art in sound texture synthesis,” Proc. Digital Audio Effects (DAFx), pp. 221–232, 2011.
  • [8] G. Bernardes, L. Aly, and M. E. Davies, “Seed: Resynthesizing environmental sounds from examples,” Proc. the Sound and Music Computing Conference, pp. 55–62, 2016.
  • [9] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “WaveNet: A generative model for raw audio,” arXiv preprint, arXiv:1609.03499, 2016.
  • [10] Q. Kong, Y. Xu, T. Iqbal, Y. Cao, W. Wang, and M. D. Plumbley, “Acoustic scene generation with conditional sampleRNN,” Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 925–929, 2019.
  • [11] K. Imoto, “Introduction to acoustic event and scene analysis,” Acoustical Science and Technology, vol. 39, no. 3, pp. 182–188, 2018.
  • [12] S. Mehri, K. Kumar, I. Gulrajani, R. Kumar, S. Jain, J. Sotelo, A. Courville, and Y. Bengio, “SampleRNN: An unconditional end-to-end neural audio generation model,” Proc. International Conference for Learning Representations (ICLR), pp. 1–11, 2017.
  • [13] “Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs,” ITU-T Recommendation P.862, 2001.
  • [14] “Perceptual objective listening quality assessment,” ITU-T Recommendation P.863, 2011.
  • [15] “Method for objective measurements of perceived audio quality,” ITU-R Recommendation BS.1387-1, 2001.
  • [16] S. Ikawa and K. Kashino, “Generating sound words from audio signals of acoustic events with sequence-to-sequence model,” Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 346–350, 2018.
  • [17] E. Grinstein, N. Q. K. Duong, A. Ozerov, and P. Pérez, “Audio style transfer,” Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 586–590, 2018.
  • [18] P. K. Mital, “Time domain neural audio style transfer,” arXiv preprint, p. arXiv:1711.11160, 2017.
  • [19]

    L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,”

    Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2414–2423, 2016.
  • [20] S. Nakamura, K. Hiyane, F. Asano, and T. Endo, “Acoustical sound database in real environments for sound scene understanding and hands-free speech recognition,” Proc. Language Resources and Evaluation Conference (LREC), pp. 965–968, 2000.
  • [21] https://www.ksuke.net/demos.