Feasibility of Post-Editing Speech Transcriptions with a Mismatched Crowd

by   Purushotam Radadia, et al.
Tata Consultancy Services

Manual correction of speech transcription can involve a selection from plausible transcriptions. Recent work has shown the feasibility of employing a mismatched crowd for speech transcription. However, it is yet to be established whether a mismatched worker has sufficiently fine-granular speech perception to choose among the phonetically proximate options that are likely to be generated from the trellis of an ASRU. Hence, we consider five languages, Arabic, German, Hindi, Russian and Spanish. For each we generate synthetic, phonetically proximate, options which emulate post-editing scenarios of varying difficulty. We consistently observe non-trivial crowd ability to choose among fine-granular options.



There are no comments yet.


page 1

page 2

page 3


Post-edit Analysis of Collective Biography Generation

Text generation is increasingly common but often requires manual post-ed...

Context-Aware Prosody Correction for Text-Based Speech Editing

Text-based speech editors expedite the process of editing speech recordi...

When Waiting is not an Option : Learning Options with a Deliberation Cost

Recent work has shown that temporally extended actions (options) can be ...

Genetic approach for arabic part of speech tagging

With the growing number of textual resources available, the ability to u...

Manual Post-editing of Automatically Transcribed Speeches from the Icelandic Parliament - Althingi

The design objectives for an automatic transcription system are to produ...

Translator2Vec: Understanding and Representing Human Post-Editors

The combination of machines and humans for translation is effective, wit...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

The ASR errors are unavoidable, especially in case of low-resourced languages. Human intervention is required to correct such errors. Crowds can be utilized for post-editing ASR transcripts [McGraw et al.2010, Wald2011]. The Scribe system proposed in [Lasecki et al.2012] allows non-experts to caption the speech in near real time (less than 5 sec latency). The study reported in [Gaur2015] analyzes latency of transcriptions achieved from crowd based on the errors in an ASR output. In above mentioned studies, a worker must be familiar with the spoken language. This can limit the crowd base for under-resourced languages due to the significant mismatch between population of native speakers of languages in the world and the crowd workers who speak that language [Pavlick et al.2014]. Therefore, in [Jyothi and Hasegawa-Johnson2015, Radadia et al.2016] the use of a mismatched crowd, which is unfamiliar with the source language has been explored for transcription. Nevertheless, to the best of our knowledge, the utility of mismatched crowd is not explored for transcription post editing. Our contribution in this paper is to employ a mismatched crowd that performs transcription voting tasks for a given utterance. We consider five different languages in our study: Arabic, German, Hindi, Russian, Spanish languages.

2 Corpus Creation

Word Selection: We select words each from, viz., Arabic, German, Hindi, Russian and Spanish. The words are sampled from publically available pronunciation dictionaries that are being used in ASR systems111https://sourceforge.net/projects/cmusphinx/. The pronunciation dictionary contains words and their phonetic decompositions in terms of arpabets. The words are selected in such a way that the overall corpus becomes phonetically rich for each language. In addition, the length of selected words is limited to 4-8 arpabets.

Synthetic Utterances: Since, the word utterances in natural speech can be dependent on the speaker as well as the acoustic conditions of the recording environment, we use synthetic utterances in our experiments. We synthesize language specific word utterances using Google speech synthesis interface via SoundOfText222http://soundoftext.com/ utility.

Generation of Synthetic Options: Since, we are interested in studying the ability of the crowd to capture the fine grained phonetic information, the valid dictionary word may not provide the proper set of options which are phonetically proximate. Hence, we aim to generate them synthetically. We introduced phonetic substitution errors to generate the perturbed sequence. Further, we only allow the vowel to vowel and consonant to consonant substitutions and not cross substitutions. This helps in maintaining the pronunciability of the word. We present a worker with a set of 5 options (4 strings and a "none of the above"). The 4 sequences are generated from the reference sequence to form an option set for a given word. We considered 6 such option sets for each word in the corpus. The sets are, , , , , and . This results in 24 different sequences for each reference sequence. The set indicates that the alternate arpabet sequences are 0, 1, 2 and 4 substitutions away from reference, respectively.

Phoneme to Grapheme Modeling: Since the crowd worker may neither be familiar with the phone set nor the script of a spoken language, we convert all the phoneme sequences to the valid script that workers are familiar with. We chose Roman(English) script for this purpose. To transform the phoneme sequence to the Roman letters (Graphemes), Phoneme to Grapheme models (P2G) models are used. We used Carmel Finite State Toolkit [Graehl] to build P2G model for each language. Consequently, the trained P2G models enable us to create the Roman letter sequences of the perturbed arpabet sequences of the alternate list.

Task Allocation and Crowd Work: We used the CrowdFlower platform for our experiments. We created 6 tasks for each of the 500 words from the considered 5 languages. Each task was repeated thrice to study vote aggregation, as well as remove spam bias. Each worker was allocated 15 randomly selected tasks and thus we employed 3000 workers to complete 15000 voting tasks.

3 Experimental Results

Crowd Bias against Rejection:

We observed that crowd distribution is skewed towards the quadrant where accuracy if true world listed is greater than 50%, while the accuracy if word is not-listed is less than 50%. This indicates that majority of the crowd fails to exercise the Reject option when similar sounding options are available in the list. Hence, in the remainder of the paper, we focus on the cases where the correct word is always present in the list shown to the worker.

Performance of Crowd workers: Table 1 shows that as the task difficulty increases (i.e., the options becomes more phonetically similar), the performance is degraded. We highlight that even an accuracy of 45%, as observed for Arabic over the hardest options, is significantly better than that of random guess accuracy of 20% (or 25%). Meanwhile, for the Indo-European languages the average accuracy, especially for the relatively easy tasks () is greater than 75%.

ar de hi ru es

Table 1: Voting accuracies across languages

Language Tree Analysis: The native language of a transcriber can significantly impact his/her perception of the mismatched speech. Table 2 shows the performance of workers, belonging to certain languages/language-families. The language families considered in the Table 2 are represented as: Indo-Aryan={Bengali, Bihari, Urdu, Sinhalese, Gujarati, Marathi, Nepali}, Slavic={Polish, Czech, Croatian, Ukranian, Bulgerian, Serbian, Slovenian, Slovak }, Romance={Italian, Portugese, French} and Tonal={Chinese, Vietnamese}. It is evident that for each of the five languages the best accuracies were exhibited by the natives. The fact that natives are unable to provide near 100% accuracy can be attributed to spam, but, also to the fact that task involves utterances from multiple languages, thus, denying the listener of a context. In addition, one can observe some correlation across related languages, for example, the second best performance for Hindi was by the Indo-Aryan group, for Russian was by the Slavic group and for Spanish was by the Romance group. We could not establish such a pattern for German (where English should have been expected to be the second best.) The correlations in speech perceptions may indeed be influenced by modern cultural influences and shared words, rather than just linguistic connections.

ar de hi ru es Arabic Turkish German English Hindi Indo-Aryan Russian Slavic Spanish Romance Tonal

Table 2: Impact of worker’s language on Accuracies

Feasibility of Aggregating Labels: We consider two very simple vote aggregation strategies. The first one is the traditional majority vote decoding, where, the aggregated label is the majority label if a simple majority exists, else if all three labels are distinct, we simply declare an error. The second algorithm, seeks to break such three way ties, by building worker reputations on the tasks where a majority does exist. The reputation is described by the count of correct votes in simple pure majorities. In a three-way tie, the worker with the highest such repute is declared a winner. Table 3 shows the performance of the Marjority Vote (MV) as well as the Majority Vote with Tie-Breaks (TB). When compared to Table 1, MV provides a gain in accuracy of (1-2)% for Arabic, (2-4)% for German, (3-6)% for Hindi, (3-4)% for Russian and (2-5)% for Spanish. Additionally, it can be observed that reputation based TB provides an additional gain over MV of (4-5)% for Arabic, (4-5)% for German, (2-5)% for Hindi, (3)% for Russian and (3-4)% for Spanish.

MV TB MV TB MV TB ar de hi ru es

Table 3: Majority vote with tie breaking

4 Conclusion

It was observed that a mismatched crowd is reluctant to use a Reject Option, in future, we may have to provide improved interaction to un-bias the crowd. However, in the cases where the true value is a part of the option list it was observed that a mismatched crowd can indeed identify the correct option even among fine-granular (phonetically proximate) choices. There is early evidence that some language pairs (or demographies) though non-native may be better matches, similarly there is evidence that the voting errors are not correlated as shown by the gain in accuracy through even majority voting based strategies.


  • [Gaur2015] Gaur, Y. 2015. The effects of automatic speech recognition quality on human transcription latency. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility, 367–368. ACM.
  • [Graehl] Graehl, J. Carmel finite state toolkit.
  • [Jyothi and Hasegawa-Johnson2015] Jyothi, P., and Hasegawa-Johnson, M. 2015. Acquiring speech transcriptions using mismatched crowdsourcing. In AAAI, 1263–1269.
  • [Lasecki et al.2012] Lasecki, W.; Miller, C.; Sadilek, A.; Abumoussa, A.; Borrello, D.; Kushalnagar, R.; and Bigham, J. 2012. Real-time captioning by groups of non-experts. In Proceedings of the 25th annual ACM symposium on User interface software and technology, 23–34. ACM.
  • [McGraw et al.2010] McGraw, I.; Lee, C.; Hetherington, I. L.; Seneff, S.; and Glass, J. R. 2010. Collecting voices from the cloud. In LREC.
  • [Pavlick et al.2014] Pavlick, E.; Post, M.; Irvine, A.; Kachaev, D.; and Callison-Burch, C. 2014. The language demographics of amazon mechanical turk. Transactions of the Association for Computational Linguistics 2:79–92.
  • [Radadia et al.2016] Radadia, P.; Kumar, R.; Kalra, K.; Karande, S.; and Lodha, S. 2016. On employing a highly mismatched crowd for speech transcription. In INTERSPEECH. "To appear".
  • [Wald2011] Wald, M. 2011. Crowdsourcing correction of speech recognition captioning errors. In Proceedings of the International Cross-Disciplinary Conference on Web Accessibility, W4A ’11, 22:1–22:2. New York, NY, USA: ACM.