Forecast Calibration

Calculate Brier score and calibration gap for binary forecasts.

Description

Binary forecast scoring compares probabilities with realized zero/one outcomes.

Binary forecast scoring compares probabilities with realized zero/one outcomes. Mean calibration gap can hide differences among individual predictions.^[1]

Calculate Brier score and calibration gap for binary forecasts.

When to use Forecast Calibration

Use this tool to calculate Brier score and calibration gap for binary forecasts.

Prepare the recorded observations, ratings, assumptions or expected/observed lists requested. Choose thresholds and definitions appropriate to the planning question.

How to use Forecast Calibration and understand its results

Prepare the recorded observations, ratings, assumptions or expected/observed lists requested. Choose thresholds and definitions appropriate to the planning question.
Enter forecastProbabilities, outcomes using the field table and calculation rules, then run the tool.
Read brierScore, meanForecastProbability, observedFrequency, calibrationGap. The explanation below defines the units, signs, ordering and any flags or placeholder values.
If the tool reports an input issue, correct the values and run it again. Apply the stated assumptions when interpreting a successfully returned result.

Input field	Meaning	Value type; see rules below	Requirement
forecastProbabilities	Forecast probabilities	list of number	Required
outcomes	Binary outcomes	list of number	Required

Enter nonempty aligned probabilities 0–1 and outcomes exactly 0 or 1. Return Brier score, means and absolute gap.

Brier=mean((probability−outcome)²); gap=|meanProbability−meanOutcome|

Units, indexing, boundary conditions and exceptional cases are defined in the calculation rules above.

Output field	Meaning	Value type
brierScore	Brier score	number; minimum: 0; maximum: 1
meanForecastProbability	Mean forecast probability	number; minimum: 0; maximum: 1
observedFrequency	Observed event frequency	number; minimum: 0; maximum: 1
calibrationGap	Absolute calibration gap	number; minimum: 0; maximum: 1

json

{
  "input": {
    "forecastProbabilities": [
      0.8,
      0.2
    ],
    "outcomes": [
      1,
      0
    ]
  },
  "output": {
    "brierScore": 0.039999999999999994,
    "meanForecastProbability": 0.5,
    "observedFrequency": 0.5,
    "calibrationGap": 0
  }
}

Limitations and assumptions

No binwise calibration curve, uncertainty or future predictive performance is estimated.

Numeric inputs, where present, must be finite. Follow the calculation-specific rules for overflow, underflow and exceptional cases. Lists are limited to 100,000 entries; the rules above may require smaller limits. Missing required fields, extra fields, wrong value types and violations of the calculation rules are input errors.

Alternative or Complementary methods

Use Delphi Consensus: Measure real-time Delphi consensus from expert ratings. For a complementary task, use State of the Future Index: Calculate a caller-defined weighted percent-change index from aligned indicators.

References

Forecast Calibration — tool.io

Similar or alternative tools

Delphi Consensus
Measure real-time Delphi consensus from expert ratings.
State of the Future Index
Calculate a caller-defined weighted percent-change index from aligned indicators.