Weighting scheme
Weighting schemes allow raters to gain partial credit for partial agreements. Weights are stored in a square matrix of size q, where q is the number of possible categories. Values in this matrix can be indexed by w_kl where k is the category assigned by the first rater and l is the category assigned by the second rater. Weights range from 0 to 1, where 0 represents no credit and 1 represents full credit. Full credit is always assigned on the diagonal (i.e., when k = l).
Example matrices are shown for q = 5 and x = {1, 2, 3, 4, 5}.
Identity weights are identity matrices and are useful when categories are distinct and unordered (i.e., nominal).
Linear weights are equal to 1 minus the distance between the categories divided by the maximum distance between any two possible categories. Here | . | represents the absolute value function. The denominator represents the maximum distance between any two categories.
Quadratic weights are similar to linear weights except that the numerator and denominator (i.e., observed and maximum distances) are squared prior to division. This squaring results in higher values than linear weights and thus a more "forgiving" scheme.
- Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220.
- Krippendorff, K. (1980). Content analysis: An introduction to its methodology. Newbury Park, CA: Sage Publications.
- Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Gaithersburg, MD: Advanced Analytics.