Probability Calibration and What Confidence Can Mean

A label or distance is not a probability

A deterministic threshold classifier assigns a label from its inputs and rules. Its distance from a boundary is a distance, not automatically a probability that the label is correct. Converting that distance into a percentage requires a defined target, a fitted mapping, and independent evaluation. A two-feature synthetic classification exercise does not acquire empirical confidence merely by using precise decimals. Name the quantity being reported: margin, model-implied state probability, forecast event probability, and observed conditional frequency are different objects with different evidential requirements.

Choose an outcome that can be evaluated

Define the event, horizon, information cutoff, and outcome-recording rule before scoring probabilities. A forecast of next-period threshold crossing differs from an estimate of an unobserved current state. Hidden states have no automatically available ground-truth label; a fitted model's own labels cannot independently validate its state probabilities. If an external outcome or proxy is used, explain the mismatch. For revised economic outcomes, declare whether verification uses the first release or a later vintage, and retain that choice throughout the comparison.

Calibration concerns repeated comparable forecasts

Calibration asks whether forecast probabilities align with event frequencies over suitable groups of cases. A reliability diagram groups predictions and compares their average probability with the observed event fraction, as described in the scikit-learn calibration reference. Show the case count and probability spread in each group, because bins can hide heterogeneity. Sparse bins and dependent observations weaken conclusions. A single event occurring or failing to occur cannot confirm or refute calibration, and a well-calibrated constant forecast need not distinguish higher-risk from lower-risk cases.

Use proper scores without calling them calibration tests

Proper scoring rules reward honest probability assessments in expectation under their stated setup; Gneiting and Raftery develop this framework. For a binary outcome y coded zero or one, use the binary Brier convention: average (p-y)^2 across cases. Lower is better for this loss. The score reflects more than calibration, including the ability to distinguish cases, so a lower score alone does not prove improved calibration. State the convention, since summing squared errors across both binary classes changes the numerical scale.

Calculate a fictional score transparently

The invented probabilities below are assigned solely for arithmetic; they are not fitted estimates or actual forecasts. With invented outcomes 0, 1, 0, 0, their squared errors are 0.04, 0.09, 0.36, and 0.01. The sum is 0.50, so the binary Brier score is 0.125. A separately predeclared constant 0.50 forecast would score 0.25 on these cases. This tiny fictional comparison establishes neither calibration nor real predictive skill. In particular, the large error on the third case is compatible with a probabilistic forecast sometimes being wrong.

Fit calibration without borrowing evaluation outcomes

When calibration is learned, fit its mapping on data separate from the final assessment, preserving chronological order and the horizon of each target. Earlier predictions used for calibration should themselves come from an appropriate out-of-sample process. Keep imputation, scaling, model selection, and any recalibration inside their declared information boundaries. Filtered state estimates can still leak future information through full-sample parameter fitting; smoothed estimates explicitly use later observations. Neither should be compared with the model's own hindsight labels and advertised as independent proof of reliable probabilities.

Report uncertainty and leave room to abstain

Report score differences, reliability summaries, coverage, and dependence-aware uncertainty alongside a fixed baseline. Show how conclusions change across evaluation periods without selecting the most favorable window. If a policy abstains on ambiguous cases, disclose the excluded fraction and evaluate those cases separately so that selective coverage is visible.

Also audit missing forecasts and unresolved outcomes. Dropping cases after discovering that they are difficult to score can alter the evaluated population. Compare competing forecasts on a declared common set, while reporting each method's total coverage. Keep pending outcomes separate from non-events; a horizon that has not finished supplies no verified binary result.

Distinguish uncertainty about inputs, fitted parameters, state definitions, and future events; one confidence number rarely represents all of them. These are principles for evaluating probability claims, not investment advice or evidence that any real regime forecast is calibrated.

Inspect the comparison

Primary references

References support the stated methods and data conventions. Original fictional examples are not observations from the cited sources.

• scikit-learn: Probability Calibration
• Gneiting and Raftery: Strictly Proper Scoring Rules, Prediction, and Estimation
• statsmodels: Markov Switching Autoregression Models

Continue the investigation