An Application of HodgeRank to Online Peer Assessment

1 Introduction

In this paper, we construct a reference score for online peer assessments based on HodgeRank [5]. Peer assessment is a process in which students grade their peers’ assignments [2, 6].

A peer assignment system is used to enhance students’ learning process, especially in higher education. Through such a system, students are given the opportunity to not only learn knowledge from textbooks and instructors, but also from the process of making judgements on assignments completed by their peers. This process helps them understand the weaknesses and strengths in the work of others, and then to review their own.

However, there are some practical issues associated with a peer assignment system. For example, students tend to give significantly higher grades than senior graders or professionals (see [3] for more details). Also, students have a tendency to give grades within a range, with the center of such a range often being based on the first grade they gave. Therefore, bias and heterogeneity can occur in a peer assignment system.

There are various ranking methods on peer assessment problem, such as PeerRank [7] and Borda-like aggregation algorithm [1]. PeerRank, a famous method based on a iterative process to solve the fixed-point equation. PeerRank has many interesting properties from the view of linear algebra. Borda-like aggregation algorithm, a random matheod based on the theory of random graphs and voting theory, whcih provides some probabilistic explanation on peer assessment problem.

In this paper, we propose another ranking scheme to deal with peer assessment problems that uses HodgeRank, a statistical preference aggregation problem from pairwise comparison data. The purpose of HodgeRank is to find a global ranking system based on pairwise comparison data. HodgeRank can not only generate a ranking order, but also highlight inconsistencies in the comparisons (see [5] for more detail). We apply HodgeRank to the problems in online assessment and display ranking results from HodgeRank and PeerRank in turn.

We will briefly introduce HodegRank and its useful properties in next section.

2 HodgeRank

HodgeRank, a statistical ranking method based on combinatorial Hodge theory to find a consistent ranking. Rigorously speaking, HodgeRank is one solution of a graph Laplacian problem with minimum Euclidean norm.

Now, we start from notations borrowed from graph theory.

Consider a connected graph $G = (V, E)$ , where $V = {1, 2, \dots, n}$ is the set of alternatives to be ranked, and $E \subseteq V \times V$ , consists of some unordered pairs from $V$ .

In this paper, $V$ represents the set of students to be ranked by their peers, and $E$ collects the information of pairwise comparisons. i.e., $(i, j) \in E$ if students $i$ and $j$ are compared at least once.

Denote $Λ$ to be the number of assignments. Then for each assignment $α \in Λ$ , pairwise comparison data on a graph $G$ of assignment $α$ , is given by $Y^{α} : E \to R$ so that $Y^{α}$

is skew-symmetry. i.e.,

Y_{i j}^{α} = - Y_{j i}^{α}

for all

i, j \in V

Y_{i j}^{α} > 0

if grade of the student

j

is higher than student

i

Y_{i j}^{α}

credits. For example,

Y_{i j}^{α} \in [- 100, 100]

on hundred-mark system.

For each $α \in Λ$ , a weight matrix $W^{α} = [w_{i j}^{α}]$ is associated as follows: $w_{i j}^{α} > 0$ if $Y_{i j}^{α} \neq 0$ , and $0$ otherwise. Set $W = \sum α \in Λ W^{α}$ .

Let $Y = \sum α \in Λ Y^{α}$ be a $n$ -by- $n$ matrix. The goal of the HodgeRank is find a ranking $s : V \to R$ so that

Y_{i j} = s_{j} - s_{i} for all i, j \in V .

(1)

However, equations (1) can not be admissible in general. Consider the following example,

Y = ⎡ ⎢ ⎣ \begin{matrix} 0 & 1 & - 1 - 1 & 0 & - 1 1 & 1 & 0 \end{matrix} ⎤ ⎥ ⎦

If there exists $s : V \to R$ such that (1) hold. Then

1 = Y_{12} = s_{2} - s_{1} = (s_{2} - s_{3}) + (s_{3} - s_{1}) = Y_{32} + Y_{13} = 0

which leads to a contradiction. That is, it is impossible to solve (1) for any skew-symmetric matrix $Y$ . Therefore, we should consider the least square solution of (1) instead. Before we rewrite above problem, we need to introduce some notations below.

Definition 2.1

[5] Denote

M_{G} = {X \in R^{n \times n} | X_{i j} = s_{i} - s_{j} for% some s : V \to R},

the space of global ranking, and the combinatorial gradient operator

grad : F (V, R) \to M_{G}

is an operator defined from $F (V, R)$ , the set of all function from $V$ to $R$ (or the space of all potential functions), to $M_{G}$ , as follows

(grad s) (i, j) = s_{j} - s_{i} .

From the example above, it is easy to find that if $X = g r a d (s)$ for some $s \in F (V, R)$ , then $X_{i j} + X_{j k} + X_{k i} = 0$ for any $(i, j), (j, k), (k, i) \in E$ . However, the converse might not be true in general. That is, denote

A = {X \in R^{n \times n} | X^{T} = - X},

the set of all skew-symmetric matrices, and let

M_{T} = {X \in A | X_{i j} + X_{j k} + X_{k i} = 0},

then $M_{G} \subseteq M_{T}$ .

With these notations above, then the above problem becomes the following optimization problem:

min X \in M_{G} | | X - Y | |_{2, w}^{2} = min X \in M_{G} \sum (i, j) \in E w_{i j} (X_{i j} - Y_{i j})^{2}

That is, once a graph is given, then the weight on edge $E$ determines an optimization problem. Conversely, a graph can intuitively arise from the ranking data.

Let ${Y^{α} | α \in Λ}$ be a set of $n$ -by- $n$ skew-symmetric matrices, and ${W^{α} | α \in Λ}$ is associated as above.

Then an undirected graph $G = (V, E)$ can be defined by $V = {1, 2, \dots, n}$ and

E = {(i, j) \in V \times V | W_{i j} > 0} .

In this case, we can treat $X$ as a edge flow on $G$

in the sense of combinatorial vector calculus.

In conclusion, we have the following relation between graph and

Hence, the optimization problem of a skew-symmetric least square problem can be view as an optimization problem of edge flow on a graph.

Definition 2.2

(Consistency) [5] Let $X : V \times X \to R$ be a pairwise ranking edge flow on a graph $G = (G, E)$ .

X is called consistency on ${i, j, k}$ if

$(i, j), (j, k), (k, i) \in E$ and $X \in M_{T}$
X is called globally consistency on ${i, j, k}$ if $X = grad (s)$ for some $s \in F (V, R)$

Note that if $X$ is called globally consistency, then $X$ is consistency on any 3-clique ${i, j, k}$ , where $(i, j), (j, k), (k, i) \in E$ .

Now, consider the weighted trace induced by $W$ . i.e.,

< X, Y >= tr (X^{T} (W ⊙ Y)) = \sum (i, j) \in E W_{i j} X_{i j} Y_{i j}

for $X, Y \in A$ , where $⊙$ represents the Hadamard product or elementwise product.

With this weighted inner product, we obtain two orthogonal complement of $A$

A = M_{G} \oplus M_{G}^{⊥} = M_{T} \oplus M_{T}^{⊥}

Since $M_{G} \subseteq M_{T}$ , we have $M_{G}^{⊥} \supseteq M_{T}^{⊥}$ and we can get further orthogonal direct sum decomposition of $A$ as follows:

A = M_{G} \oplus M_{H} \oplus M_{T}^{⊥},

where $M_{H} = M_{T} \cap M_{G}^{⊥}$ .

This decomposition is called the combinatorial Hodge decomposition. For more detail about the theory of combinatorial Hodge decomposition, please refer [5] for more detail.

We now state one useful theorem in [5].

Theorem 2.1

[5]

The minimum norm solution $s$ of (1) is the solution of the normal equation:

$Δ_{0} s = - div Y,$

where , and

$div (Y) (i) = \sum j s . t . (i, j) \in E w_{i j} Y_{i j}$

is the combinatorial curl operator of $Y$ .
The minimum norm solution $s$ of (1) is

$s^{*} = - Δ_{0}^{†} div Y,$

where $Δ_{0}^{†}$ represents the Moore-Penrose pseudo inverse of the matrix $Δ_{0}$ .

The Hodge decomposition indicates the solution of $(???)$ , while the theorem 2.1 shows how to calculate the minimum solution by solving the normal equation. In the next section, we display how to apply HodgeRank to the online peer assessment problem.

3 Online peer assessment problem

As previously mentioned, bias and heterogeneity can lead to unfair scoring in online peer assessments. Students usually grade other students based on the first score they gave, which causes bias. However, since scores are usually compared with others, we can use this comparison behavior to reconstruct true ranking.

The data we used in this section were collected from an undergraduate calculus course. In this course, 133 students were asked to upload their GeoGebra [4] assignments. Each student was then asked to review five randomly chosen assignments completed by their peers to receive partial credits in return. There are 13 assignments during one semester.

Note that ne key point of the HodgeRank is the connectedness of the graph generated by pairwise comparison data. From table 1 above, we can easily see that after half the semester passed, comparison data between students forms a connected graph. Hence, we can apply HodgeRank to calculate the ranking of all the students after assignment 7.

Assignment #	1	2	3	4	5	6	$7 \sim 13$
# of components	21	5	4	3	2	2	1

Table 1: Number of components with respect to the number of assignments

The traditional method for finalizing peer assessment consists of either using an average cumulative score or a truncated average score. Although these approaches might have some statistical meaning, they cannot avoid bias and heterogeneity in peer assessment.

Figure 1 displays the cumulative score, PeerRank and HodgeRank, respectively. Here, $(α, β) = (0.5, 0)$ in the setting of PeerRank. For what these parameters represent in PeerRank, please refer to [7] for more discussion.

Figure 1: Final results using different ranking methods

To compare these results, ranking results were normalized into the interval [0, 1] linearly and sorted in ascending order. In addition, to reveal the tendency of each ranking method, a steady line was plotted on the graph. There are some interesting implications that can be observed from this figure.

First, the cumulative score offers a ranking higher than the steady line. This reflects the existence of bias and heterogeneity in the cumulative average method. Second, PeerRank can be viewed as a modification of the average scoring. Third, sorted ranking result from HodgeRank is a normal distributed curve. This result can might be an explanation why HodgeRank can be solution to eliminate bias and heterogeneity by the normality.

Note that the reason why HodgeRank and PeerRank show different results is their conclusion base are totally different, while former method relies on the pairwise comparison data and latter one is applied on the average score as an initial ranking. Hence, HodgeRank provides instructors with an objective scoring reference using score difference rather than cumulative or average score.

In conclusion, this is the first time HodgeRank has been applied in the field of education. While numerical results were processed using real world data in this study, certain issues, such as how to aggregate the HodgeRank ranking method into a peer assessment system, remain unsolved. This task will be attempted as part of our future work.

References

[1] I. Caragiannis et al., Aggregating Partial Rankings with Applications to Peer Grading in Massive Online Open Courses, AAMAS ’15 Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, (2015).
[2] N. Falchikov and J. Goldfinch, Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks, Review of educational research, 70 (2000), pp. 287–322.
[3] S. Freeman and J. W. Parks, How accurate is peer grading?, CBE-Life Sciences Education, 9 (2010), pp. 482–488.
[4] M. Hohenwarter, Geogebra–ein softwaresystem für dynamische geometrie und algebra der ebene [geogebra–a software system for dynamic geometry and algebra in the plane](unpublished master’s thesis), Universität Salzburg, Salzburg, Austria, (2002).
[5] X. Jiang, L.-H. Lim, Y. Yao, and Y. Ye, Statistical ranking and combinatorial hodge theory, Mathematical Programming, 127 (2011), pp. 203–244.
[6] K. Topping, Peer assessment between students in colleges and universities, Review of educational Research, 68 (1998), pp. 249–276.
[7] T. Walsh, The peerrank method for peer assessment
, ECAI’14 Proceedings of the 21st European Conference on Artificial Intelligence, (2014).

An Application of HodgeRank to Online Peer Assessment

Authors

Systematic Review of Approaches to Improve Peer Assessment at Scale

Online Learning Using Only Peer Assessment

The PeerRank Method for Peer Assessment

Discovery of Bias and Strategic Behavior in Crowdsourced Performance Assessment

Inducing Honest Reporting Without Observing Outcomes: An Application to the Peer-Review Process

Mining Supervisor Evaluation and Peer Feedback in Performance Appraisals

Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation

1 Introduction

2 HodgeRank

Definition 2.1

Definition 2.2

Theorem 2.1

3 Online peer assessment problem

References

An Application of HodgeRank to Online Peer Assessment

Authors

Systematic Review of Approaches to Improve Peer Assessment at Scale

Online Learning Using Only Peer Assessment

The PeerRank Method for Peer Assessment

Discovery of Bias and Strategic Behavior in Crowdsourced Performance Assessment

Inducing Honest Reporting Without Observing Outcomes: An Application to the Peer-Review Process

Mining Supervisor Evaluation and Peer Feedback in Performance Appraisals

Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation

This week in AI

1 Introduction

2 HodgeRank

Definition 2.1

Definition 2.2

Theorem 2.1

3 Online peer assessment problem

References