Cohen's Kappa Inter-Rater Agreement

Unweighted Cohens kappa from actual matched two-rater categorical confusion counts. Not continuous Pearson correlation, weighted ordinal kappa, diagnosis or inferential reliability interval.

Description

Cohen's kappa compares observed categorical agreement with agreement expected from marginal proportions.

Cohen's kappa compares observed categorical agreement with agreement expected from marginal proportions.^[1]

The unweighted two-rater form requires a matched confusion table. The calculation and input requirements are explained below.

When to use Cohen's Kappa Inter-Rater Agreement

Enter a square confusion matrix with at least two categories and nonnegative integer counts up to 10000000.

How to use Cohen's Kappa Inter-Rater Agreement and how it works

Prepare the measurements or independently established model inputs described below, with matching units and observation scope.
Enter the required fields and run Cohen's Kappa Inter-Rater Agreement.
Read the result fields using the formula, output units, signs and ordering explained below.
Correct any rejected inputs and rerun; interpret the result within the stated model and its assumptions.

Input field	Meaning	Value type	Requirement
actualRaterConfusionCounts	Actual Rater Confusion Counts. Use the units and matching context stated in the calculation below.	array of array of number	Required; other constraints below

Enter a square confusion matrix with at least two categories and nonnegative integer counts up to 10000000. Require positive total trials. Calculate diagonal agreement and the row/column marginal expected agreement, require 1-expected positive, and return unweighted kappa. It does not supply a confidence interval.

kappa=(observedAgreement-expectedAgreement)/(1-expectedAgreement)

Apply the input ordering, units and branch conditions stated above.

Output field	Meaning	Value type
result	Object containing the calculated or selected fields below.	object
result.cohensKappa	Cohens Kappa. Interpret its unit, sign, order or availability flag as explained above.	number

json

{
  "input": {
    "actualRaterConfusionCounts": [
      [
        8,
        2
      ],
      [
        2,
        8
      ]
    ]
  },
  "output": {
    "result": {
      "cohensKappa": 0.6000000000000001
    }
  }
}

Limitations and assumptions

The calculation uses the supplied observations and model inputs. It does not collect measurements, select a reference population or coefficient, or establish that the model applies to the actual case. The stated branch conditions and output units remain part of the result.

All fields require explicit values. Unexpected fields, wrong types, nonfinite numbers and sparse arrays are rejected. Alignment, bounds and supported-token rules are explained above; extremely large inputs or nonfinite results can also fail. Input errors differ from successful false comparisons, unavailable-domain flags and empty match lists.

Alternative or Complementary methods

Use Fleiss' Kappa Multi-Rater Agreement for a complementary operation: Fleiss' kappa measures categorical agreement across multiple raters.

Use Guttman Reproducibility Coefficient for a complementary operation: Guttman reproducibility compares coded errors with all person–item responses.

References

Cohen's Kappa Inter-Rater Agreement — tool.io

Similar or alternative tools

Fleiss' Kappa Multi-Rater Agreement
Fleiss' kappa measures categorical agreement across multiple raters.
Guttman Reproducibility Coefficient
Guttman reproducibility compares coded errors with all person–item responses.