Fleiss' Kappa Multi-Rater Agreement

Fleiss multi-rater categorical kappa from actual count matrix, equal rater count; not Cohen pairwise agreement or weighted ordinal coding.

Description

Fleiss' kappa measures categorical agreement across multiple raters.

Fleiss' kappa measures categorical agreement across multiple raters.^[1]

Its count-table form requires the same number of ratings for every subject. The calculation and input requirements are explained below.

When to use Fleiss' Kappa Multi-Rater Agreement

Enter a rectangular subject-by-category matrix with at least two subjects and categories, integer counts 0–1000000 and the same row sum of at least two raters.

How to use Fleiss' Kappa Multi-Rater Agreement and how it works

Prepare the measurements or independently established model inputs described below, with matching units and observation scope.
Enter the required fields and run Fleiss' Kappa Multi-Rater Agreement.
Read the result fields using the formula, output units, signs and ordering explained below.
Correct any rejected inputs and rerun; interpret the result within the stated model and its assumptions.

Input field	Meaning	Value type	Requirement
actualSubjectByCategoryRaterCounts	Actual Subject By Category Rater Counts. Use the units and matching context stated in the calculation below.	array of array of number	Required; other constraints below

Enter a rectangular subject-by-category matrix with at least two subjects and categories, integer counts 0–1000000 and the same row sum of at least two raters. Compute mean within-subject agreement and squared pooled category proportions, require expected agreement below one and return Fleiss' kappa. No ordinal weighting is used.

fleissKappa=(meanSubjectAgreement-sum(categoryProportions^2))/(1-sum(categoryProportions^2))

Apply the input ordering, units and branch conditions stated above.

Output field	Meaning	Value type
result	Object containing the calculated or selected fields below.	object
result.fleissKappa	Fleiss Kappa. Interpret its unit, sign, order or availability flag as explained above.	number

json

{
  "input": {
    "actualSubjectByCategoryRaterCounts": [
      [
        2,
        0
      ],
      [
        0,
        2
      ]
    ]
  },
  "output": {
    "result": {
      "fleissKappa": 1
    }
  }
}

Limitations and assumptions

The calculation uses the supplied observations and model inputs. It does not collect measurements, select a reference population or coefficient, or establish that the model applies to the actual case. The stated branch conditions and output units remain part of the result.

All fields require explicit values. Unexpected fields, wrong types, nonfinite numbers and sparse arrays are rejected. Alignment, bounds and supported-token rules are explained above; extremely large inputs or nonfinite results can also fail. Input errors differ from successful false comparisons, unavailable-domain flags and empty match lists.

Alternative or Complementary methods

Use Cohen's Kappa Inter-Rater Agreement for a complementary operation: Cohen's kappa compares observed categorical agreement with agreement expected from marginal proportions.

Use Guttman Reproducibility Coefficient for a complementary operation: Guttman reproducibility compares coded errors with all person–item responses.

References

Fleiss' Kappa Multi-Rater Agreement — tool.io

Similar or alternative tools

Cohen's Kappa Inter-Rater Agreement
Cohen's kappa compares observed categorical agreement with agreement expected from marginal proportions.
Guttman Reproducibility Coefficient
Guttman reproducibility compares coded errors with all person–item responses.