Skip to content

Scott's pi coefficient

Jump to bottom
Jeffrey M Girard edited this page Feb 23, 2016 · 36 revisions

Overview

The pi coefficient is a chance-adjusted index for the reliability of categorical measurements. It estimates chance agreement using a distribution-based approach. It assumes that observers have a conspired "quotas" for each category that they work together to meet.

History

Coming soon...

MATLAB Functions

  • FAST_PI %Calculates pi using simplified formulas
  • FULL_PI %Calculates pi using generalized formulas

Simplified Formulas

Use these formulas with two raters and two (dichotomous) categories:

pi

pi

pi

pi

pi

a is the number of items both raters assigned to the first category

d is the number of items both raters assigned to the second category

n is the total number of items

f_1 is the number of items rater A assigned to category 1

f_2 is the number of items rater A assigned to category 2

g_1 is the number of items rater B assigned to category 1

g_2 is the number of items rater B assigned to category 2

Contingency Table

Generalized Formulas

Use these formulas with multiple raters, multiple categories, and any weighting scheme:

pi

pi

pi

pi

pi

q is the total number of categories

w_kl is the weight associated with two raters assigning an item to categories k and l

r_il is the number of raters that assigned item i to category l

n' is the number of items that were coded by two or more raters

r_ik is the number of raters that assigned item i to category k

r_i is the number of raters that assigned item i to any category

n is the total number of items

References

  1. Scott, W. A. (1955). Reliability of content analysis: The case of nominal scaling. Public Opinion Quarterly, 19(3), 321–325.
  2. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382.
  3. Siegel, S., & Castellan, N. J. (1988). Nonparametric statistics for the behavioural sciences. New York, NY: McGraw-Hill.
  4. Byrt, T., Bishop, J., & Carlin, J. B. (1993). Bias, prevalence and kappa. Journal of Clinical Epidemiology, 46, 423–429.
  5. Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Gaithersburg, MD: Advanced Analytics.
Clone this wiki locally