Multi-rater delta: extending the delta nominal measure of agreement between two raters to many raters

09/12/2019
by   A. Martín Andrés, et al.
0

The need to measure the degree of agreement among R raters who independently classify n subjects within K nominal categories is frequent in many scientific areas. The most popular measures are Cohen's kappa (R = 2), Fleiss' kappa, Conger's kappa and Hubert's kappa (R ≥ 2) coefficients, which have several defects. In 2004, the delta coefficient was defined for the case of R = 2, which did not have the defects of Cohen's kappa coefficient. This article extends the coefficient delta from R = 2 raters to R ≥ 2. The coefficient multi-rater delta has the same advantages as the coefficient delta with regard to the type kappa coefficients: i) it is intuitive and easy to interpret, because it refers to the proportion of replies that are concordant and non random; ii) the summands which give its value allow the degree of agreement in each category to be measured accurately, with no need to be collapsed; and iii) it is not affected by the marginal imbalance.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/10/2018

sklarsomega: An R Package for Measuring Agreement Using Sklar's Omega Coefficient

R package sklarsomega provides tools for measuring agreement using Sklar...
research
04/10/2020

Multivariate Medial Concordance

We define a multivariate medial correlation coefficient that extends the...
research
04/29/2013

The Randomized Dependence Coefficient

We introduce the Randomized Dependence Coefficient (RDC), a measure of n...
research
03/22/2021

krippendorffsalpha: An R Package for Measuring Agreement Using Krippendorff's Alpha Coefficient

R package krippendorffsalpha provides tools for measuring agreement usin...
research
10/10/2018

Local Average Estimation and Inferences for Varying Coefficient Models

The varying coefficient model is very popular in the application of fina...
research
02/21/2018

Algebra and geometry of tensors for modeling rater agreement data

We study three different quasi-symmetry models and three different mixtu...

Please sign up or login with your details

Forgot password? Click here to reset