Sound synthesis and conversion are techniques for generating a natural sound using a statistical model that associates input information with the generated sound. Sound synthesis and conversion methods with the aim of generating speech or music have been widely developed [1, 2, 3]. Recently, some researchers have also developed methods for environmental sound synthesis and conversion that can be applied to support movie and game production , the generation of content for virtual reality (VR) , and data augmentation for sound event detection and scene classification . Many studies on environmental sound synthesis and conversion have taken a physical modeling or concatenative approach [7, 8, 6]
. On the other hand, there have been fewer studies on environmental sound synthesis and conversion based on statistical generative models such as deep learning approaches. To the best of our knowledge, there is no literature giving an overview of the problem definitions and evaluation methods for environmental sound synthesis and conversion. Moreover, there have been no investigation of subjective evaluation methods for environmental sound synthesis and conversion.
In this paper, we therefore review problem definitions, applications, and evaluation methods of environmental sound synthesis and conversion. We then report on environmental sound synthesis based on WaveNet , which successfully synthesizes human voices, to discuss the current performance of statistical environmental sound synthesis. Moreover, we investigate subjective evaluation methods of environmental sound synthesis.
The remainder of this paper is structured as follows. In Sec. 2, we review problem definitions of environmental sound synthesis and conversion, their applications, and evaluation methods. In Sec. 3, subjective experiments carried out to evaluate the performance of sound event synthesis using a WaveNet-based method are reported. Finally, we summarize and conclude this paper in Sec. 4.
2 Problem Definitions of Environmental Sound Synthesis and Conversion
In this section, we review applications, problem definitions, and evaluation methods of environmental sound synthesis and conversion, specifically environmental sound synthesis using event or scene labels (Sec. 2.1), environmental sound synthesis using onomatopoeic words (Sec. 2.2), environmental sound conversion (Sec. 2.3), and environmental sound synthesis/conversion using multimedia (Sec. 2.4).
2.1 Environmental Sound Synthesis Using Sound Event and Scene Labels
When providing movies or games with background sounds or sound effects, we need to listen to many sounds in a large sound database and select the most suitable one for the scene or sound event, which is a time-consuming part of movie or game production. To address this issue, a statistical method for synthesizing an environmental sound well representing a sound event or scene, which utilizes the sound event or scene labels as below as an input, has been proposed . Figures 1 and 2 illustrate the processes of environmental sound synthesis using the sound event or scene labels as the inputs of the systems, where we call these research tasks sound event synthesis (SES) and sound scene synthesis (SSS), respectively. Another issue is that the construction of an environmental sound dataset is very time-consuming compared with the construction of a speech or music dataset 
. In recent studies, environmental sound analysis based on deep neural networks has required a large number of sounds to achieve a reasonable performance. To overcome this problem of a shortage of environmental sound datasets, SES and SSS can be applied for data augmentation in environmental sound analysis.
with sound scene labels represented as one-hot vectors.
A method of evaluating synthesized environmental sounds is an important subject in this research area. When we apply SES or SSS to data augmentation for sound event detection or acoustic scene classification, it is reasonable to evaluate the methods of SES or SSS via their event detection or scene classification performance with augmented data. On the other hand, in the case of utilizing the sound synthesized by SES or SSS itself, it has not been investigated in detail how the synthesis method should be evaluated. In this paper, we focus on the subjective evaluation method for environmental sound synthesis in Sec. 3.
On the other hand, the subjective evaluation of sounds is very time-consuming; thus, it is desirable to test methods for environmental sound synthesis and conversion with an objective evaluation of synthesized sounds. There are some methods of objective evaluation such as the perceptual evaluation of speech quality (PESQ) , perceptual objective listening quality analysis (POLQA) , and perceived evaluation of audio quality (PEAQ) , which are used for the evaluation of the speech quality in telecommunication or audio quality via codecs. However, the objective evaluation method for synthesized or converted environmental sounds has not been investigated.
2.2 Environmental Sound Synthesis Using Onomatopoeic Words
The SES and SSS discussed in Sec. 2.1 control synthesized environmental sounds only using the sound event or scene labels; thus, they cannot control synthesized sounds without types of sound or scenes. For instance, when synthesizing the sound of a car horn, it cannot be determined in advance whether SES will synthesize a horn sound with a continuous high tone (e.g, peeeeeeeeee) or one with an intermittent low tone (e.g, beep beep beep). To control synthesized environmental sounds more finely, we can apply environmental sound synthesis using onomatopoeic words as an input of the system, as shown in Fig. 3. For SES using onomatopoeic words, Ikawa et al.  have proposed a method that converts onomatopoeic words to wave forms of environmental sounds using an encoder–decoder model.
|Sound length||1–2 s|
|Wavefrom encoding||16-bit linear PCM (real sound)|
|8-bit -law (synthesized sound)|
|Receptive field||64 ms|
|# residual channels||32|
|# dilation channels||32|
|# quantization channels||256|
|# skip channels||512|
|Experiment||# labels||# samples||# listeners||# total|
|in each label||samples|
2.3 Environmental Sound Conversion
Sound event synthesis using onomatopoeic words is a flexible way of synthesizing environmental sounds; however, it is still difficult to control the generated environmental sounds as intended. One way to address this problem is to synthesize environmental sounds not with sound event labels or onomatopoeic words but with the environmental sound or voice as the input of the system, as shown in Fig. 4. We call this kind of task a sound event conversion (SEC) or sound scene conversion (SSC). When we have some background sounds or sound effects but they are not suitable for the movie or game, environmental sound conversion can also be applied to obtain desirable sounds. For instance, when we have the horn sound of car X and a video including car Y, we can convert the horn sound of car X to that of car Y using SEC without re-recording the horn sound of car Y.
2.4 Environmental Sound Synthesis/Conversion Using Multimedia
Some researchers have addressed environmental sound synthesis and conversion using multimedia information as an input such as images. For instance, Zhou et al. have proposed a method for synthesizing environmental sounds from images that is based on SampleRNN .
3 Investigation of Subjective Evaluation Method
3.1 Experimental Conditions
In this section, by evaluating SES using sound event labels based on the conditional WaveNet , we discuss the current performance of environmental sound synthesis and how we should conduct a subjective test to evaluate a method under development. For the evaluation, we considered 10 different sound events (manual coffee grinder, cup clinking, alarm clock ringing, whistle, maracas, drum, electric shaver, trash box banging, tearing paper, bell ringing) contained in the RWCP-SSD (Real World Computing Partnership-Sound Scene Database) . We used a total of 1,000 samples (100 samples 10 sound events), in which 95 samples of each sound event were used for model training and the others were used for the subjective test. Table 2 shows the experimental conditions and parameters used for WaveNet. Samples of sounds synthesized by WaveNet are available at .
Many works on speech and music synthesis have been conducted using subjective tests to evaluate the quality of synthesized sounds. For example, in speech synthesis, speech intelligibility and naturalness are often used as evaluation metrics. On the other hand, there have been no works in which methods of subjective tests in environmental sound synthesis and conversion were investigated in detail; thus, we here discuss how we should conduct subjective tests for environmental sound synthesis. For synthesized sounds, it is important that (I) they are distinguishable from other types of environmental sound, (II) they are not distinguishable from real sounds, and (III) they have as high naturalness as real environmental sounds. On the basis of these considerations, we conducted the following experiments:
Experiment I: evaluation of intelligibility of synthesized sounds
After listening to a synthesized sound, the listener selected a sound event label that best represented the sound. As a comparison, the listener also similarly evaluated real environmental sounds.
Experiment II: evaluation of distinguishability of real and synthesized sounds
We conducted a preference AB test. After listening to a pair of real and synthesized sounds in random order, the listener selected the one that sounded more real.
Experiment III: evaluation of naturalness of synthesized sounds
We conducted a five-scale mean opinion score (MOS) test. After listening to a real or synthesized sound presented randomly, the listener scored the naturalness from 1 (very unnatural as an environmental sound) to 5 (very natural as an environmental sound).
Experiments were conducted with 24 listeners (13 males and 11 females) in a quiet environment at Ritsumeikan University. In Table 2, the number of samples used in each experiment is listed. In the experiments, a Roland QUAD-CAPTURE UA-55 audio interface and SONY MDR-CD900ST headphones were used.
3.2 Experimental Results and Discussion
, respectively. The averaged F-scores for real and synthesized sounds were 86.22% and 76.30%, respectively. From these results, synthesized drum sounds are classified with a similar performance to real sounds, whereas the synthesized sounds of a cup clinking and an electric shaver tended to be more often misclassified than the real sounds. Figure7 shows spectrograms of real and synthesized sounds. This indicates that the synthesized sound of an electric shaver does not have the fine structure of the spectrum, which has the real sound. Thus, the difference between the spectrograms of the sounds of an electric shaver and tearing paper is likely to be unclear, and this leads to the misclassfication. From the results of experiment I, it considered that this subjective test is particularly helpful for evaluating whether the method can reproduce distinguishable sounds even when they have similar characteristics.
Experiment II: listeners identified real sounds with an average accuracy of 82.71% as shown in Fig. 8. From this result, sounds synthesized by WaveNet do not have sufficiently high quality to be indistinguishable from real sounds. This indicates that the evaluation of the distinguishability of real and synthesized sounds can be used for the comparison of conventional methods and more sophisticated methods of environmental sound synthesis that will be developed.
: the average MOS score for the naturalness of synthesized and real sounds and its 95% confidence interval are shown in Fig.9. The results indicate that the synthesized sounds of the coffee grinder, clock, and maracas had similar naturalness scores to those of real sounds. On the other hand, for the sounds of the electric shaver and the trash box banging, there are large differences in the MOS scores between the synthesized and real sounds. We consider that this is because SES using WaveNet cannot reproduce the fine structure of the synthesized spectrum (e.g., the spectrum of the electric shaver in Fig. 7). Moreover, Figs. 5, 6, and 9 show that the listeners classified both the real and synthesized whistle sounds with reasonable performance, whereas there are large differences in the MOS scores between synthesized and real sounds. This means that the evaluation of intelligibility is not satisfactory for evaluating the quality of synthesized sounds.
Thus, we propose that methods of environmental sound synthesis should be evaluated not only by testing the intelligibility of synthesized sounds but also by testing the distinguishability of real and synthesized sounds and/or the naturalness of synthesized sounds.
In this paper, we presented the problem definitions of sound event synthesis, sound scene synthesis, and sound event and scene conversion. We then discussed the current performance of sound event synthesis and subjective evaluation methods of environmental sound synthesis. The evaluation experiments indicate that sounds synthesized by WaveNet do not yet have sufficiently high quality to be indistinguishable from real sounds. Moreover, on the basis of our experimental results, we consider that methods of environmental sound synthesis should be evaluated by testing not only intelligibility but also distinguishability and/or naturalness.
-  H. Zen, K. Tokuda, and A. Black, “Statistical parametric speech synthesis,” Speech Communication, vol. 51, no. 11, pp. 1039–1064, 2009.
-  S. H. Mohammadi and A. Kain, “An overview of voice conversion systems,” Speech Communication, vol. 88, pp. 65–82, 2017.
-  J. P. Briot, G. Hadjeres, and F. Pachet, “Deep learning techniques for music generation – a survey,” arXiv preprint arXiv:1709.01620, 2017.
-  D. B. Lloyd, N. Raghuvanshi, and N. K. Govindaraju, “Sound synthesis for impact sounds in video games,” Proc. Symposium on Interactive 3D Graphics and Games. ACM, pp. 55–61, 20111.
-  Y. Zhou, Z. Wang, C. Fang, T. Bui, and T. L. Berg, “Visual to sound: Generating natural sound for videos in the wild,” (CVPR), pp. 3550–3558, 2018.
-  J. Salamon, D. MacConnell, M. Cartwright, P. Li, and J. P. Bello, “Scaper: A library for soundscape synthesis and augmentation,” Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 344–348, 2017.
-  D. Schwarz, “State of the art in sound texture synthesis,” Proc. Digital Audio Effects (DAFx), pp. 221–232, 2011.
-  G. Bernardes, L. Aly, and M. E. Davies, “Seed: Resynthesizing environmental sounds from examples,” Proc. the Sound and Music Computing Conference, pp. 55–62, 2016.
-  A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “WaveNet: A generative model for raw audio,” arXiv preprint, arXiv:1609.03499, 2016.
-  Q. Kong, Y. Xu, T. Iqbal, Y. Cao, W. Wang, and M. D. Plumbley, “Acoustic scene generation with conditional sampleRNN,” Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 925–929, 2019.
-  K. Imoto, “Introduction to acoustic event and scene analysis,” Acoustical Science and Technology, vol. 39, no. 3, pp. 182–188, 2018.
-  S. Mehri, K. Kumar, I. Gulrajani, R. Kumar, S. Jain, J. Sotelo, A. Courville, and Y. Bengio, “SampleRNN: An unconditional end-to-end neural audio generation model,” Proc. International Conference for Learning Representations (ICLR), pp. 1–11, 2017.
-  “Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs,” ITU-T Recommendation P.862, 2001.
-  “Perceptual objective listening quality assessment,” ITU-T Recommendation P.863, 2011.
-  “Method for objective measurements of perceived audio quality,” ITU-R Recommendation BS.1387-1, 2001.
-  S. Ikawa and K. Kashino, “Generating sound words from audio signals of acoustic events with sequence-to-sequence model,” Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 346–350, 2018.
-  E. Grinstein, N. Q. K. Duong, A. Ozerov, and P. Pérez, “Audio style transfer,” Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 586–590, 2018.
-  P. K. Mital, “Time domain neural audio style transfer,” arXiv preprint, p. arXiv:1711.11160, 2017.
L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,”Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2414–2423, 2016.
-  S. Nakamura, K. Hiyane, F. Asano, and T. Endo, “Acoustical sound database in real environments for sound scene understanding and hands-free speech recognition,” Proc. Language Resources and Evaluation Conference (LREC), pp. 965–968, 2000.
-  https://www.ksuke.net/demos.