Inter-Rater Reliability: Percent Agreement, Cohen’s Kappa, Alpha

What is inter-rater reliability?

Inter-rater reliability (in qualitative research often called intercoder reliability or intercoder agreement) is the degree to which two or more coders who work independently assign the same codes to the same units of data. High agreement suggests that your codebook is clear and that your coding does not depend on one person’s idiosyncrasies. Low agreement is a signal to sharpen definitions and decision rules.

It is related to, but different from, intra-rater reliability (one coder codes the same material twice at different times) and from internal consistency of scales such as Cronbach’s alpha in questionnaire research.

Running example: you interviewed 12 first-generation students about belonging. A fellow student independently codes 100 segments from three transcripts with your codebook, deciding for each segment whether the code PEER_SUPPORT applies.

When you need it, and when you do not

Whether reliability statistics make sense depends on your method and epistemology. In coding reliability approaches and in many forms of qualitative content analysis, a second coder and a reliability coefficient are expected, especially in health research and communication studies. In reflexive thematic analysis, Braun and Clarke (2022) explicitly reject reliability coefficients: coding is seen as interpretive, and the researcher’s subjectivity is part of the analysis, so “agreement” is not a quality criterion. Grounded theory and IPA likewise rely on other quality strategies such as memo writing, reflexivity and audit trails.

Check with your supervisor. Some departments require a second coder for every qualitative dissertation; others will see a kappa value in a reflexive TA study as a methodological inconsistency. Follow your university’s guidelines (you can upload them to Professor Digital).

Percent agreement

The simplest measure is percent agreement: the number of units on which both coders agree, divided by the total number of units.

Percent agreement = agreements / total units × 100

It is intuitive but has a serious weakness: it does not account for agreement that would occur by chance. If a code is rare and both coders mostly say “not present”, they will agree on most units even if they code randomly. Report percent agreement only together with a chance-corrected coefficient.

Cohen’s kappa: formula and worked example

Cohen (1960) introduced kappa for two coders and nominal categories. It compares the observed agreement pₒ with the agreement expected by chance pₑ, which is calculated from each coder’s marginal distribution:

κ = (pₒ − pₑ) / (1 − pₑ)

κ = 1 means perfect agreement, κ = 0 means agreement at chance level, negative values mean less agreement than chance.

Worked example: two coders decide for 100 segments whether PEER_SUPPORT applies. The cross-table below shows their decisions.

How to interpret kappa: Landis and Koch

The most frequently cited benchmarks come from Landis and Koch (1977). They are conventions, not laws: the authors themselves called them arbitrary but useful. Many methodologists consider values below 0.60 insufficient for drawing conclusions from coded data.

• Kappa paradox: with very rare or very frequent codes, kappa can be low even if percent agreement is high. Report both and the code’s prevalence.
• Calculate kappa per code, not only one overall value, so you see which codes are problematic.
• For more than two coders, Fleiss’ kappa or Krippendorff’s alpha are used instead of Cohen’s kappa.
• For ordinal categories (for example intensity ratings), weighted kappa gives partial credit for near misses.

Krippendorff’s alpha

Krippendorff’s alpha is the most general agreement coefficient. It is defined as

α = 1 − Dₒ / Dₑ

where Dₒ is the observed disagreement and Dₑ the disagreement expected by chance. Its advantages: it works for any number of coders, tolerates missing data (not every coder must code every unit) and can be used for nominal, ordinal, interval and ratio data. Krippendorff (2018) recommends α ≥ 0.800 for reliable data and treats 0.667 ≤ α < 0.800 as acceptable only for tentative conclusions.

In communication research and content analysis, alpha is often preferred to kappa. Calculating it by hand is tedious, so use statistical software or a reliability calculator; our free reliability tool also covers related coefficients.

A practical procedure for your dissertation

You do not need to double-code everything. A common and defensible procedure is to have a second coder code 10 to 25 % of the material, calculate agreement, discuss disagreements and revise the codebook. Report the result transparently.

• 1 Train the second coder with the codebook on material that is not part of the test sample.
• 2 Define the coding units in advance (for example pre-segmented passages); otherwise you mix disagreement about segmentation with disagreement about codes.
• 3 Both coders code the sample independently.
• 4 Calculate percent agreement and kappa (two coders) or alpha (more coders), per code and overall.
• 5 Discuss disagreements, refine definitions and decision rules, and note changes in the codebook.
• 6 Report: sample size, coders, coefficient, values, how disagreements were resolved.