Ranking Clarifying Questions Based on Predicted User Engagement

by   Tom Lotze, et al.

To improve online search results, clarification questions can be used to elucidate the information need of the user. This research aims to predict the user engagement with the clarification pane as an indicator of relevance based on the lexical information: query, question, and answers. Subsequently, the predicted user engagement can be used as a feature to rank the clarification panes. Regression and classification are applied for predicting user engagement and compared to naive heuristic baselines (e.g. mean) on the new MIMICS dataset [20]. An ablation study is carried out using a RankNet model to determine whether the predicted user engagement improves clarification pane ranking performance. The prediction models were able to improve significantly upon the naive baselines, and the predicted user engagement feature significantly improved the RankNet results in terms of NDCG and MRR. This research demonstrates the potential for ranking clarification panes based on lexical information only and can serve as a first neural baseline for future research to improve on. The code is available online.


Analysing the Effect of Clarifying Questions on Document Ranking in Conversational Search

Recent research on conversational search highlights the importance of mi...

User Engagement Prediction for Clarification in Search

Clarification is increasingly becoming a vital factor in various topics ...

Modeling Engagement Dynamics of Online Discussions using Relativistic Gravitational Theory

Online discussions are valuable resources to study user behaviour on a d...

Systematic improvement of user engagement with academic titles using computational linguistics

This paper describes a novel approach to systematically improve informat...

Who will stay? Using Deep Learning to predict engagement of citizen scientists

Citizen science and machine learning should be considered for monitoring...

Predicting User Engagement Status for Online Evaluation of Intelligent Assistants

Evaluation of intelligent assistants in large-scale and online settings ...

Can Population-based Engagement Improve Personalisation? A Novel Dataset and Experiments

This work explores how population-based engagement prediction can addres...