An Educational System for Personalized Teacher Recommendation in K-12 Online Classrooms

07/15/2021 ∙ by Jiahao Chen, et al. ∙ 5

In this paper, we propose a simple yet effective solution to build practical teacher recommender systems for online one-on-one classes. Our system consists of (1) a pseudo matching score module that provides reliable training labels; (2) a ranking model that scores every candidate teacher; (3) a novelty boosting module that gives additional opportunities to new teachers; and (4) a diversity metric that guardrails the recommended results to reduce the chance of collision. Offline experimental results show that our approach outperforms a wide range of baselines. Furthermore, we show that our approach is able to reduce the number of student-teacher matching attempts from 7.22 to 3.09 in a five-month observation on a third-party online education platform.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Because of the better accessibility and immersive learning experience, one-on-one class stands out among all the different forms of online courses [chen2019multimodal, liang2005few, blatchford2011examining, blatchford2003class, finn1999tennessee]. In one-on-one courses, teacher recommender systems play an important role in helping students find their most appropriate teachers [li2020identify, xu2020automatic]. However, teacher recommendation presents numerous challenges that come from the following special characteristics of real-world educational scenarios:

  • Limited sizes of demand and supply: The number of teachers in supply side is incredibly smaller compared to Internet-scaled inventories. Moreover, different from item based recommendation where popular items can be suggested to millions of users simultaneously, a teacher can only take a very limited amount of students and students may only take one or two classes at each semester.

  • Lack of gold standard: There is no ground truth showing how good a match is between a teacher and a student. The rating based mechanism doesn’t work since ratings from K-12 students are very noisy and unreliable.

  • Cold-start teachers: The online educational marketplace is dynamic and there are always new teachers joining. It is important to give such new teachers opportunities to let them take students instead of keeping recommending existing best performing teachers.

  • High-demand diversity: It is undesirable to recommend the same set of teachers to students and the teacher recommender systems are supposed to reduce chances that two students want to book the same teacher at the same time.

The objective of this work is to study and develop approaches that can be used for personalized teacher recommendation for online classes. More specifically, we design techniques to (1) generate robust pseudo training labels as ground truth for learning patterns of good matches between students and teachers; (2) boost newly arrived teachers by giving incentives to their ranking scores when generating the recommended candidates; and (3) fairly evaluate and guard the diversity of recommendation results by the proposed measure of teacher diversity. We compare our approach with a wide range of baselines and evaluate its benefits on a real-world online one-on-one class dataset. We also deploy our algorithm into the real production environment and demonstrate its effectiveness in terms of number of matching attempts.

2 The Framework

In this section we will discuss the details about our teacher recommendation framework for the online one-on-one courses. Our framework is made up of four key components: (1) the pseudo matching scores module; (2) the ranking model; (3) the novelty boosting module ; and (4) the diversity metric.

Pseudo Matching Scores One of the most challenging problems in building teacher recommender systems is the missing of ground truth. To remedy above problem, we choose to generate the pseudo matching scores from students’ dropouts. Our mechanism relies on the assumption that matching scores reflect student preferences, which are approximated by the number of one-on-one courses between each teacher and student. In addition, we capture of the recency effect of dropout cases by using an exponential function. We design the pseudo matching scores as follows:

Definition 1

positive pseudo matching score. For student who has completed the class, let be the collection of teachers who have ever taught student and denotes the th teacher, denotes the total number of teachers who have taught student . Let be the number of courses taught by teacher . The positive pseudo matching scores of is defined as , where is the positive matching score function.

Definition 2

negative pseudo matching score. For student who has dropped the class, with similar notations in Definition 1, the negative pseudo matching scores of is defined as , where is the negative matching score function.

According to Definitions 1 and 2, the pseudo matching scores range from -1 to 1. It reaches the maximum value of 1 when a student never requests a change of teacher and completes the entire class and it goes to the minimum value of -1 when a student immediately quits after the first course.

The Ranking Model The ranking model learns from a collection of teacher-student pairs with pseudo matching scores. We design the following three categories of features: (1) demographic features: the demographic information of both students and teachers, such as gender, schools, etc. (2) in-class features: the class behavioral features from both students and teachers, such as lengths of talking time, the number of spoken sentences, etc. (3) historical features:

the historical features aggregate each teacher’s past teaching performance, which includes total numbers of courses and historical dropout rates, etc. In this work, we choose to use gradient boosting decision tree (GBDT)

[friedman2001greedy] as our ranking model due to its robustness and generalization capability.

Novelty Boosting for New Teachers We design a novelty boosting component that gives extra ranking incentives to new teachers and enhances the chances of successful matches for new teachers. The novelty boosting score for teacher is defined as follows:

(1)

where represents the index set of all taught students and represents the total number of courses taught by teacher . , and are positive hyper parameters. Moreover, we measure the overall effect of novelty boosting by computing the overall new teacher ratios in the top-recommended candidates, where represents the set of recommended teachers for student and is the indicator function that indicates whether the teacher is a new teacher.

Diversity Measurement Diversity is important when conducting teacher recommendations in K-12 online scenarios. In this work, we propose a diversity guardrail measurement , where represents the set of students needed online one-on-one instructors. The diversity scores range from 0 to 1. It reaches the maximum value of 1 when each student’s recommendation results don’t overlap.

3 Experiments

Offline Evaluation The offline evaluation of recommendation is different from standard binary classification tasks where we can only partially observe the ground truth. When designing an effective offline evaluation environment, we will only focus on the “good” matches between students and teachers and ensure that a positively matched teacher exists in our recommended candidate list. Therefore, the performance is mainly evaluated by recall. Besides, we measure the effects of new teacher ratio and diversity. The hyper parameters in novelty boosting score function are =0.04, =1, and for each student we select top 200 teachers as our recommended candidates. We collect a real-world dataset with 3,672 students, 2,139 teachers, and 8,072 student-teacher matches. Here, to simulate “good” matches, we first compute the pseudo matching scores for all 8,072 student-teacher matches and randomly select 821 pairs whose positive scores are over 0.5 as our test data. We compare our approach with ItemCF [sarwar2001item], SVD [golub1970singular], NMF [gu2010collaborative], DeepFM [guo2017deepfm],W&D [cheng2016wide]. The results are shown in Table 1. The proposed approach has competitive performance against the widely used recommendation models. Please note that the matrix factorization based baselines, such as ItemCF, SVD and NMF, cannot seamlessly integrate new teachers into their corresponding rating matrices and hence fail to recommend new teachers.

model Precision Recall Diversity New Teacher Ratio

Our
0.0017 0.2545 0.7454 0.0333
Wide&Deep 0.0016 0.2335 0.7446 0.0070
DeepFM 0.0016 0.2351 0.7438 0.0013
ItemCF 0.0014 0.2011 0.7232 N/A
SVD 0.0013 0.1909 0.7233 N/A
NMF 0.0013 0.1924 0.7233 N/A
Table 1: Results on our offline educational dataset.

Online Experiments We deploy our algorithm to a real production environment. We continuously observe the change of the mean value of the number of times a student requires to change their teacher. Over the five-month observation period (2020/01 - 2020/05), we found that the number of matching attempts decreased from 7.22 to 3.09, reflecting that our algorithm can accurately make more good recommendations to teachers.

4 Conclusion

In this paper, we present an end-to-end teacher recommendation framework for online one-on-one classes in the real-world scenario. The results on the real-world educational teacher recommendation dataset show that our proposed system can not only accurately recommend teachers in terms of recall but give more opportunities to new teachers in terms of new teacher ratios. Meanwhile, we guardrail the overall recommendation quality in terms of diversity experimentally. In online experiments, the proposed model is deployed in the real production system and the results show that the proposed approach is able to greatly reduce the number of matching attempts.

Acknowledgment

This work was supported in part by National Key R&D Program of China, under Grant No. 2020AAA0104500 and in part by Beijing Nova Program (Z201100006820068) from Beijing Municipal Science & Technology Commission.

References