Weighting scheme
Weighting schemes allow observers to gain partial credit for partial agreements. Weights are stored in a square matrix of size q, where q is the number of possible categories. Values in this matrix can be indexed by w_kl where k is the category assigned by the first rater and l is the category assigned by the second rater. Weights range from 0 to 1, where 0 represents no credit and 1 represents full credit. Full credit is always assigned on the diagonal (i.e., when k = l).
Nominal weights are identity matrices.
Ordinal weights involve the ratio of pairwise combinations.
Interval (i.e., linear) weights are equal to 1 minus the distance between the categories divided by the maximum distance between any two possible categories. Here | . | represents the absolute value function. The denominator represents the maximum distance between any two categories.
Ratio weights evaluate the differences between scores relative to their magnitudes.
- Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220.
- Krippendorff, K. (1980). Content analysis: An introduction to its methodology. Newbury Park, CA: Sage Publications.
- Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Gaithersburg, MD: Advanced Analytics.