All labs
Interactive lab
Confidence calibration explorer
Two models, the same accuracy — one you can trust in a high-stakes loop, one that will hurt you.
AIDecision systems
Synthetic
Synthetic predictions, seeded and reproducible. An illustration of the mechanics, not a real model's numbers.
Loading interactive…
What it proves
Accuracy hides whether a model knows when it's right. In a system allowed to abstain, calibration — not accuracy — decides whether the confidence threshold means anything.
How it works
- Each profile draws the same stated confidences, then draws correctness from an authored calibration curve a(p) = P(correct | p).
- Expected Calibration Error is the count-weighted average distance between each bin's stated confidence and its observed accuracy.
- The abstention threshold answers only above the cut — coverage is how much it still answers, selective risk is how often it's wrong when it does.