Someone is well-calibrated if the things they predict with X% chance of happening in fact occur X% of the time. Importantly, calibration is not the same as accuracy. Calibration is about accurately assessing how good your predictions are, not making good predictions. Person A, whose predictions are marginally better than chance (60% of them come true when choosing from two options) and who is precisely 60% confident in their choices, is perfectly calibrated. In contrast, Person B, who is 99% confident in their predictions, and right 90% of the time, is more accurate than Person A, but less well-calibrated.

See also: Betting, Epistemic Modesty, Forecasting & Prediction

Being well-calibrated has value for rationalists separately from accuracy. Among other things, being well-calibrated lets you make good bets / make good decisions, communicate information helpfully to others if they know you to be well-calibrated (See Group Rationality), and helps prioritize which information is worth acquiring.

Note that all expressions of quantified confidence in beliefs can be well- or poorly- calibrated. For example, calibration applies to whether a person's 95% confidence intervals captures the true outcome 95% of the time.

List of Calibration Exercises
based on this post. Todo: find more & sort & new post for visibility in search engines?

WebApp with user created Exercises
PDF by CFAR
by CFAR
based on The Success Equation
based on Scout Mindset
by Guided Track (Also linked to by 80000 hours and OpenPhil)
by Outside the Asylum
by Peter Attia
by Quantified Intuitions
by 2pih

Exercises that are dead/unmaintained

https://www.metaculus.com/tutorials (dead link)
http://web.archive.org/web/20100529074053/http://www.acceleratingfuture.com/tom/?p=129
http://credencecalibration.com (dead link)
https://calibration.lazdini.lv (dead link)
http://web.archive.org/web/20161020032514/http://calibratedprobabilityassessment.org/
https://predictionbook.com/credence_games/try (deprecated; see also this Github Issue)
https://calibration-training.netlify.app (dead link)