Log In Sign Up

Predicting User Engagement Status for Online Evaluation of Intelligent Assistants

by   Rui Meng, et al.

Evaluation of intelligent assistants in large-scale and online settings remains an open challenge. User behavior-based online evaluation metrics have demonstrated great effectiveness for monitoring large-scale web search and recommender systems. Therefore, we consider predicting user engagement status as the very first and critical step to online evaluation for intelligent assistants. In this work, we first proposed a novel framework for classifying user engagement status into four categories – fulfillment, continuation, reformulation and abandonment. We then demonstrated how to design simple but indicative metrics based on the framework to quantify user engagement levels. We also aim for automating user engagement prediction with machine learning methods. We compare various models and features for predicting engagement status using four real-world datasets. We conducted detailed analyses on features and failure cases to discuss the performance of current models as well as challenges.


page 1

page 2

page 3

page 4


User Engagement Prediction for Clarification in Search

Clarification is increasingly becoming a vital factor in various topics ...

Who will stay? Using Deep Learning to predict engagement of citizen scientists

Citizen science and machine learning should be considered for monitoring...

From Optimizing Engagement to Measuring Value

Most recommendation engines today are based on predicting user engagemen...

Ranking Clarifying Questions Based on Predicted User Engagement

To improve online search results, clarification questions can be used to...

Synerise at RecSys 2021: Twitter user engagement prediction with a fast neural model

In this paper we present our 2nd place solution to ACM RecSys 2021 Chall...

Vision-based Engagement Detection in Virtual Reality

User engagement modeling for manipulating actions in vision-based interf...

On Variational Inference for User Modeling in Attribute-Driven Collaborative Filtering

Recommender Systems have become an integral part of online e-Commerce pl...