Krippendorff's alpha coefficient

Overview

The alpha coefficient is a chance-adjusted index for the reliability of categorical measurements. It estimates chance agreement using an average-distribution-based approach. Like Scott's pi coefficient, it assumes that observers have a conspired "quota" for each category that they work together to meet. However, unlike pi, it also attempts to correct for sample size and yields a higher reliability score than the pi coefficient when the reliability experiment includes fewer items.

MATLAB Functions

• mALPHAK %Calculates alpha using vectorized formulas

Simplified Formulas

Use these formulas with two raters and two (dichotomous) categories:

---

$$p_o = \frac{n_{11} + n_{22}}{n}$$

$$m_1 = \frac{n_{+1} + n_{1+}}{2}$$

$$m_2 = \frac{n_{+2} + n_{2+}}{2}$$

$$p_c = \left( \frac{2m_1}{2n} \right) \left( \frac{2m_1 - 1}{2n - 1} \right) + \left( \frac{2m_2}{2n} \right) \left( \frac{2m_2 - 1}{2n - 1} \right)$$

$$\alpha = \frac{p_o - p_c}{1 - p_c}$$

---

$n_{11}$ is the number of items both raters assigned to category 1

$n_{22}$ is the number of items both raters assigned to category 2

$n$ is the total number of items

$n_{1+}$ is the number of items rater 1 assigned to category 1

$n_{2+}$ is the number of items rater 1 assigned to category 2

$n_{+1}$ is the number of items rater 2 assigned to category 1

$n_{+2}$ is the number of items rater 2 assigned to category 2

Generalized Formulas

Use these formulas with multiple raters, multiple categories, and any weighting scheme:

---

$$r_{ik}^\star = \sum_{l=1}^q w_{kl} r_{il}$$

$$p_o' = \frac{1}{n'} \sum_{i=1}^{n'} \sum_{k=1}^q \frac{r_{ik} (r_{ik}^\star - 1)}{\bar r (r_i - 1)}$$

$$\bar r = \frac{1}{n'} \sum_{i=1}^{n'} r_i$$

$$\varepsilon_n = \frac{1}{n' \bar r}$$

$$p_o = p'_o (1 - \varepsilon_n) + \varepsilon_n$$

$$\pi_k = \frac{1}{n'} \sum_{i=1}^{n'} \frac{r_{ik}}{\bar r}$$

$$p_c = \sum_{k,l} w_{kl} \pi_k \pi_l$$

$$\alpha = \frac{p_o - p_c}{1 - p_c}$$

---

$q$ is the total number of categories

$w_{kl}$ is the weight associated with two raters assigning an item to categories $k$ and $l$

$r_{il}$ is the number of raters that assigned item $i$ to category $l$

$n'$ is the number of items that were coded by two or more raters

$r_{ik}$ is the number of raters that assigned item $i$ to category $k$

$r_i$ is the number of raters that assigned item $i$ to any category

References

1. Krippendorff, K. (1970). Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement, 30(1), 61–70.
2. Krippendorff, K. (1980). Content analysis: An introduction to its methodology. Newbury Park, CA: Sage Publications.
3. Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77–89.
4. Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Gaithersburg, MD: Advanced Analytics.