Calibration and the Brier Score: How to Evaluate a Probability Forecast
The binary Brier-score formula, a worked calculation, reliability bins and why one average score is not enough to assess calibration.
What calibration means
A probability forecast is a number from 0 to 1, not a promise about one outcome. A group of forecasts near 70% is well calibrated if the event occurs in about 70% of comparable cases. This is a group-level property; it does not mean every individual match “wins” at a 70% rate.
The Brier score is the mean squared error between a stated probability and a binary observed outcome. For one event, p is the forecast probability and y is 1 if the event occurred or 0 if it did not; the error is (p − y)². For N forecasts, add the individual scores and divide by N. In the binary case, a lower score means lower average squared error, but comparisons require the same events and outcome definition.
A worked calculation
You assign an outcome probability p = 0.70. If it occurs (y = 1), the error is (0.70 − 1)² = 0.09. If it does not (y = 0), the error is (0.70 − 0)² = 0.49. For four forecasts—0.70/1, 0.70/0, 0.40/0 and 0.40/1—the mean Brier score is (0.09 + 0.49 + 0.16 + 0.36)/4 = 0.275.
A single result depends on both confidence and outcome. One loss at 70% does not show that a model is poorly calibrated; the check needs many forecasts, comparable events and temporal control. If probabilities were selected after reviewing results or losses are missing, the analysis is already biased; begin with a complete forecast journal.
Reliability bins and limits of one number
Group frozen forecasts into predefined probability intervals, such as 0.60–0.70 and 0.70–0.80. In each interval, compare the average forecast with the observed event frequency. If forecasts in the 0.70–0.80 bin average 0.74 but the outcome occurs in 0.62 of cases, that suggests overstatement in that group, subject to sample size. Report the count: a bin with only a few matches has wide random uncertainty.
The mean Brier score combines several attributes of probabilistic forecasts. The classical decomposition includes reliability, resolution and baseline uncertainty. So one aggregate score does not show whether forecasts are grouped correctly by probability. Read it alongside a reliability diagram, event frequency and bin sizes; compare against a simple baseline on the same matches. The basics and limits of a hit rate appear in the win-rate guide, while the confidence-interval article covers uncertainty around an observed frequency.
A practical evaluation
Keep the probability, market, forecast timestamp, data available at that time and final outcome. First separate development data from a later holdout, then calculate Brier and bins only on the frozen evaluation sample. Do not fit a correction on the same data or choose interval edges after seeing the most appealing chart.
The Brier score evaluates probability forecasts; it does not measure bet profitability. Even sound calibration does not remove commission, price, risk or uncertainty about a new match.