Multimodal Engagement Analysis from Facial Videos in the Classroom

by   Ömer Sümer, et al.

Student engagement is a key construct for learning and teaching. While most of the literature explored the student engagement analysis on computer-based settings, this paper extends that focus to classroom instruction. To best examine student visual engagement in the classroom, we conducted a study utilizing the audiovisual recordings of classes at a secondary school over one and a half month's time, acquired continuous engagement labeling per student (N=15) in repeated sessions, and explored computer vision methods to classify engagement levels from faces in the classroom. We trained deep embeddings for attentional and emotional features, training Attention-Net for head pose estimation and Affect-Net for facial expression recognition. We additionally trained different engagement classifiers, consisting of Support Vector Machines, Random Forest, Multilayer Perceptron, and Long Short-Term Memory, for both features. The best performing engagement classifiers achieved AUCs of .620 and .720 in Grades 8 and 12, respectively. We further investigated fusion strategies and found score-level fusion either improves the engagement classifiers or is on par with the best performing modality. We also investigated the effect of personalization and found that using only 60-seconds of person-specific data selected by margin uncertainty of the base classifier yielded an average AUC improvement of .084. 4.Our main aim with this work is to provide the technical means to facilitate the manual data analysis of classroom videos in research on teaching quality and in the context of teacher training.


page 5

page 7

page 11

page 14


Engagement Recognition using Deep Learning and Facial Expression

Engagement is a key indicator of the quality of learning experience, and...

The Wits Intelligent Teaching System: Detecting Student Engagement During Lectures Using Convolutional Neural Networks

To perform contingent teaching and be responsive to students' needs duri...

Bootstrap Model Ensemble and Rank Loss for Engagement Intensity Regression

This paper presents our approach for the engagement intensity regression...

MATT: Multimodal Attention Level Estimation for e-learning Platforms

This work presents a new multimodal system for remote attention level es...

Detecting Disengagement in Virtual Learning as an Anomaly

Student engagement is an important factor in meeting the goals of virtua...

Prediction and Localization of Student Engagement in the Wild

Student engagement localization can play a key role in designing success...

Predicting Stress in Remote Learning via Advanced Deep Learning Technologies

COVID-19 has driven most schools to remote learning through online meeti...

Please sign up or login with your details

Forgot password? Click here to reset