Using Elo Rating as a Metric for Comparative Judgement in Educational Assessment

04/04/2022
by   Andy Gray, et al.
0

Marking and feedback are essential features of teaching and learning, across the overwhelming majority of educational settings and contexts. However, it can take a great deal of time and effort for teachers to mark assessments, and to provide useful feedback to the students. Furthermore, it also creates a significant cognitive load on the assessors, especially in ensuring fairness and equity. Therefore, an alternative approach to marking called comparative judgement (CJ) has been proposed in the educational space. Inspired by the law of comparative judgment (LCJ). This pairwise comparison for as many pairs as possible can then be used to rank all submissions. Studies suggest that CJ is highly reliable and accurate while making it quick for the teachers. Alternative studies have questioned this claim suggesting that the process can increase bias in the results as the same submission is shown many times to an assessor for increasing reliability. Additionally, studies have also found that CJ can result in the overall marking process taking longer than a more traditional method of marking as information about many pairs must be collected. In this paper, we investigate Elo, which has been extensively used in rating players in zero-sum games such as chess. We experimented on a large-scale Twitter dataset on the topic of a recent major UK political event ("Brexit", the UK's political exit from the European Union) to ask users which tweet they found funnier between a pair selected from ten tweets. Our analysis of the data reveals that the Elo rating is statistically significantly similar to the CJ ranking with a Kendall's tau score of 0.96 and a p-value of 1.5x10^(-5). We finish with an informed discussion regarding the potential wider application of this approach to a range of educational contexts.

READ FULL TEXT
research
05/22/2023

Leveraging Human Feedback to Scale Educational Datasets: Combining Crowdworkers and Comparative Judgement

Machine Learning models have many potentially beneficial applications in...
research
05/11/2022

An Approach to Adaptive Microlearning in Higher Education

Current changes in society and the education system, cumulated with the ...
research
10/09/2022

Cross-strait Variations on Two Near-synonymous Loanwords xie2shang1 and tan2pan4: A Corpus-based Comparative Study

This study attempts to investigate cross-strait variations on two typica...
research
08/25/2023

A Bayesian Active Learning Approach to Comparative Judgement

Assessment is a crucial part of education. Traditional marking is a sour...
research
05/16/2023

Efficacy of Educational Misinformation Games

Misinformation has become a significant issue in today's society, with t...
research
01/03/2022

GeoGebra Discovery in Context

In our contribution we will reflect, through a collection of selected ex...
research
09/17/2022

Local political control in educational policy: Evidence from decentralized teacher pay reform under England's local education authorities

In 2012, the School Teachers' Review Body discontinued central guidance ...

Please sign up or login with your details

Forgot password? Click here to reset