On Design of Problem Token Questions in Quality of Experience Surveys

by   Jayant Gupchup, et al.

User surveys for Quality of Experience (QoE) are a critical source of information. In addition to the common "star rating" used to estimate Mean Opinion Score (MOS), more detailed survey questions (problem tokens) about specific areas provide valuable insight into the factors impacting QoE. This paper explores two aspects of the problem token questionnaire design. First, we study the bias introduced by fixed question order, and second, we study the challenge of selecting a subset of questions to keep the token set small. Based on 900,000 calls gathered using a randomized controlled experiment from a live system, we find that the order bias can be significantly reduced by randomizing the display order of tokens. The difference in response rate varies based on token position and display design. It is worth noting that the users respond to the randomized-order variant at levels that are comparable to the fixed-order variant. The effective selection of a subset of token questions is achieved by extracting tokens that provide the highest information gain over user ratings. This selection is known to be in the class of NP-hard problems. We apply a well-known greedy submodular maximization method on our dataset to capture 94 of the information using just 30


Analysis of Problem Tokens to Rank Factors Impacting Quality in VoIP Applications

User-perceived quality-of-experience (QoE) in internet telephony systems...

A survey on the parameterized complexity of the independent set and (connected) dominating set reconfiguration problems

A graph vertex-subset problem defines which subsets of the vertices of a...

Token Swapping on Trees

The input to the token swapping problem is a graph with vertices v_1, v_...

UTXO in Digital Currencies: Account-based or Token-based? Or Both?

There are different interpretations of the terms "tokens" and "token-bas...

A principled framework for the design and analysis of token algorithms

We consider a decentralized optimization problem, in which n nodes colla...

Dynamic Question Ordering in Online Surveys

Online surveys have the potential to support adaptive questions, where l...

To incentivize or not: Impact of blockchain-based cryptoeconomic tokens on human information sharing behavior

Cryptoeconomic incentives in the form of blockchain-based tokens are see...

Please sign up or login with your details

Forgot password? Click here to reset