Crowdsourcing approach for subjective evaluation of echo impairment

10/25/2020
by   Ross Cutler, et al.
0

The quality of acoustic echo cancellers (AECs) in real-time communication systems is typically evaluated using objective metrics like ERLE and PESQ, and less commonly with lab-based subjective tests like ITU-T Rec. P.831. We will show that these objective measures are not well correlated to subjective measures. We then introduce an open-source crowdsourcing approach for subjective evaluation of echo impairment which can be used to evaluate the performance of AECs. We provide a study that shows this tool is highly reproducible. This new tool has been recently used in the ICASSP 2021 AEC Challenge which made the challenge possible to do quickly and cost effectively.

READ FULL TEXT
research
10/06/2021

AECMOS: A speech quality assessment metric for echo impairment

Traditionally, the quality of acoustic echo cancellers is evaluated usin...
research
05/16/2017

Subjective Knowledge Acquisition and Enrichment Powered By Crowdsourcing

Knowledge bases (KBs) have attracted increasing attention due to its gre...
research
03/24/2016

Evaluating the Performance of Offensive Linemen in the NFL

How does one objectively measure the performance of an individual offens...
research
03/08/2022

Reproducible Subjective Evaluation

Human perceptual studies are the gold standard for the evaluation of man...
research
10/25/2020

A Crowdsourcing Extension of the ITU-T Recommendation P.835 with Validation

The quality of the speech communication systems, which include noise sup...
research
06/01/2020

A time-scale modification dataset with subjective quality labels

Time Scale Modification (TSM) is a well-researched field; however, no ef...
research
06/17/2019

Crowdsourcing in the Absence of Ground Truth -- A Case Study

Crowdsourcing information constitutes an important aspect of human-in-th...

Please sign up or login with your details

Forgot password? Click here to reset