Attribute agreement analysis and kappa
How to check go/no-go gauges and visual inspection, and how to read kappa values.
When inspectors judge parts as pass or fail (or grade them), the "measurement system" is the inspector plus the method. Attribute agreement analysis checks how consistent and how correct those judgments are.
Running a study
- Choose 30 to 50 parts, including clearly good, clearly bad and borderline parts. Know the correct answer for each (the standard).
- Have 2 or 3 inspectors judge each part 2 or 3 times in random order, without knowing the earlier results.
What it measures
- Within appraiser: does each inspector agree with themselves?
- Between appraisers: do inspectors agree with each other?
- Each appraiser vs standard and all vs standard: are they right?
- Kappa corrects agreement for what would happen by chance. 1 is perfect agreement, 0 is no better than chance.
Reading kappa
- Above 0.90: excellent.
- 0.75 to 0.90: good.
- 0.40 to 0.75: marginal; improve the method, boundary samples or training.
- Below 0.40: poor.
For ordered grades (1 to 5), Kendall's coefficient of concordance is more informative than kappa.
Questions and answers
Our inspectors agree 95% of the time. Isn't that enough?
Not if most parts are obviously good. Agreement on easy parts is guaranteed. Kappa and borderline parts in the study show whether inspectors agree where it matters.
In RS-1: Quality > Attribute MSA and Cohen's Kappa.
