An Application of HodgeRank to Online Peer Assessment

03/07/2018 ∙ by Tse-Yu Lin, et al. ∙ 0

Bias and heterogeneity in peer assessment can lead to the issue of unfair scoring in the educational field. To deal with this problem, we propose a reference ranking method for an online peer assessment system using HodgeRank. Such a scheme provides instructors with an objective scoring reference based on mathematics.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

In this paper, we construct a reference score for online peer assessments based on HodgeRank [5]. Peer assessment is a process in which students grade their peers’ assignments [2, 6].

A peer assignment system is used to enhance students’ learning process, especially in higher education. Through such a system, students are given the opportunity to not only learn knowledge from textbooks and instructors, but also from the process of making judgements on assignments completed by their peers. This process helps them understand the weaknesses and strengths in the work of others, and then to review their own.

However, there are some practical issues associated with a peer assignment system. For example, students tend to give significantly higher grades than senior graders or professionals (see  [3] for more details). Also, students have a tendency to give grades within a range, with the center of such a range often being based on the first grade they gave. Therefore, bias and heterogeneity can occur in a peer assignment system.

There are various ranking methods on peer assessment problem, such as PeerRank [7] and Borda-like aggregation algorithm [1]. PeerRank, a famous method based on a iterative process to solve the fixed-point equation. PeerRank has many interesting properties from the view of linear algebra. Borda-like aggregation algorithm, a random matheod based on the theory of random graphs and voting theory, whcih provides some probabilistic explanation on peer assessment problem.

In this paper, we propose another ranking scheme to deal with peer assessment problems that uses HodgeRank, a statistical preference aggregation problem from pairwise comparison data. The purpose of HodgeRank is to find a global ranking system based on pairwise comparison data. HodgeRank can not only generate a ranking order, but also highlight inconsistencies in the comparisons (see [5] for more detail). We apply HodgeRank to the problems in online assessment and display ranking results from HodgeRank and PeerRank in turn.

We will briefly introduce HodegRank and its useful properties in next section.

2 HodgeRank

HodgeRank, a statistical ranking method based on combinatorial Hodge theory to find a consistent ranking. Rigorously speaking, HodgeRank is one solution of a graph Laplacian problem with minimum Euclidean norm.

Now, we start from notations borrowed from graph theory.

Consider a connected graph , where is the set of alternatives to be ranked, and , consists of some unordered pairs from .

In this paper, represents the set of students to be ranked by their peers, and collects the information of pairwise comparisons. i.e., if students and are compared at least once.

Denote to be the number of assignments. Then for each assignment , pairwise comparison data on a graph of assignment , is given by so that

is skew-symmetry. i.e.,

for all . if grade of the student is higher than student by credits. For example, on hundred-mark system.

For each , a weight matrix is associated as follows: if , and otherwise. Set .

Let be a -by- matrix. The goal of the HodgeRank is find a ranking so that

(1)

However, equations (1) can not be admissible in general. Consider the following example,

If there exists such that (1) hold. Then

which leads to a contradiction. That is, it is impossible to solve (1) for any skew-symmetric matrix . Therefore, we should consider the least square solution of (1) instead. Before we rewrite above problem, we need to introduce some notations below.

Definition 2.1

[5] Denote

the space of global ranking, and the combinatorial gradient operator

is an operator defined from , the set of all function from to (or the space of all potential functions), to , as follows

From the example above, it is easy to find that if for some , then for any . However, the converse might not be true in general. That is, denote

the set of all skew-symmetric matrices, and let

then .

With these notations above, then the above problem becomes the following optimization problem:

That is, once a graph is given, then the weight on edge determines an optimization problem. Conversely, a graph can intuitively arise from the ranking data.

Let be a set of -by- skew-symmetric matrices, and is associated as above.

Then an undirected graph can be defined by and

In this case, we can treat as a edge flow on

in the sense of combinatorial vector calculus.

In conclusion, we have the following relation between graph and

Hence, the optimization problem of a skew-symmetric least square problem can be view as an optimization problem of edge flow on a graph.

Definition 2.2

(Consistency) [5] Let be a pairwise ranking edge flow on a graph .

  • X is called consistency on if

    and

  • X is called globally consistency on if for some

Note that if is called globally consistency, then is consistency on any 3-clique , where .

Now, consider the weighted trace induced by . i.e.,

for , where represents the Hadamard product or elementwise product.

With this weighted inner product, we obtain two orthogonal complement of

Since , we have and we can get further orthogonal direct sum decomposition of as follows:

where .

This decomposition is called the combinatorial Hodge decomposition. For more detail about the theory of combinatorial Hodge decomposition, please refer [5] for more detail.

We now state one useful theorem in [5].

Theorem 2.1

[5]

  1. The minimum norm solution of (1) is the solution of the normal equation:

    where , and

    is the combinatorial curl operator of .

  2. The minimum norm solution of (1) is

    where represents the Moore-Penrose pseudo inverse of the matrix .

The Hodge decomposition indicates the solution of , while the theorem 2.1 shows how to calculate the minimum solution by solving the normal equation. In the next section, we display how to apply HodgeRank to the online peer assessment problem.

3 Online peer assessment problem

As previously mentioned, bias and heterogeneity can lead to unfair scoring in online peer assessments. Students usually grade other students based on the first score they gave, which causes bias. However, since scores are usually compared with others, we can use this comparison behavior to reconstruct true ranking.

The data we used in this section were collected from an undergraduate calculus course. In this course, 133 students were asked to upload their GeoGebra  [4] assignments. Each student was then asked to review five randomly chosen assignments completed by their peers to receive partial credits in return. There are 13 assignments during one semester.

Note that ne key point of the HodgeRank is the connectedness of the graph generated by pairwise comparison data. From table  1 above, we can easily see that after half the semester passed, comparison data between students forms a connected graph. Hence, we can apply HodgeRank to calculate the ranking of all the students after assignment 7.

Assignment # 1 2 3 4 5 6
# of components 21 5 4 3 2 2 1
Table 1: Number of components with respect to the number of assignments

The traditional method for finalizing peer assessment consists of either using an average cumulative score or a truncated average score. Although these approaches might have some statistical meaning, they cannot avoid bias and heterogeneity in peer assessment.

Figure 1 displays the cumulative score, PeerRank and HodgeRank, respectively. Here, in the setting of PeerRank. For what these parameters represent in PeerRank, please refer to  [7] for more discussion.

Figure 1: Final results using different ranking methods

To compare these results, ranking results were normalized into the interval [0, 1] linearly and sorted in ascending order. In addition, to reveal the tendency of each ranking method, a steady line was plotted on the graph. There are some interesting implications that can be observed from this figure.

First, the cumulative score offers a ranking higher than the steady line. This reflects the existence of bias and heterogeneity in the cumulative average method. Second, PeerRank can be viewed as a modification of the average scoring. Third, sorted ranking result from HodgeRank is a normal distributed curve. This result can might be an explanation why HodgeRank can be solution to eliminate bias and heterogeneity by the normality.

Note that the reason why HodgeRank and PeerRank show different results is their conclusion base are totally different, while former method relies on the pairwise comparison data and latter one is applied on the average score as an initial ranking. Hence, HodgeRank provides instructors with an objective scoring reference using score difference rather than cumulative or average score.

In conclusion, this is the first time HodgeRank has been applied in the field of education. While numerical results were processed using real world data in this study, certain issues, such as how to aggregate the HodgeRank ranking method into a peer assessment system, remain unsolved. This task will be attempted as part of our future work.

References

  • [1] I. Caragiannis et al., Aggregating Partial Rankings with Applications to Peer Grading in Massive Online Open Courses, AAMAS ’15 Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, (2015).
  • [2] N. Falchikov and J. Goldfinch, Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks, Review of educational research, 70 (2000), pp. 287–322.
  • [3] S. Freeman and J. W. Parks, How accurate is peer grading?, CBE-Life Sciences Education, 9 (2010), pp. 482–488.
  • [4] M. Hohenwarter, Geogebra–ein softwaresystem für dynamische geometrie und algebra der ebene [geogebra–a software system for dynamic geometry and algebra in the plane](unpublished master’s thesis), Universität Salzburg, Salzburg, Austria, (2002).
  • [5] X. Jiang, L.-H. Lim, Y. Yao, and Y. Ye, Statistical ranking and combinatorial hodge theory, Mathematical Programming, 127 (2011), pp. 203–244.
  • [6] K. Topping, Peer assessment between students in colleges and universities, Review of educational Research, 68 (1998), pp. 249–276.
  • [7] T. Walsh, The peerrank method for peer assessment

    , ECAI’14 Proceedings of the 21st European Conference on Artificial Intelligence, (2014).