Forecasting & Prediction
Forecasting or Predicting is the act of making statements about what will happen in the future (and in some cases, the past) and then scoring the predictions. Posts marked with this tag are for discussion of the practice, skill, and methodology of forecasting. Posts exclusively containing object-level lists of forecasts and predictions are in Forecasts.
Above all, don’t ask what to believe—ask what to anticipate. Every question of belief should flow from a question of anticipation, and that question of anticipation should be the center of the inquiry.
Forecasting allows individuals and institutions to test their internal models of reality. A forecaster with a good track record can have more confidence in future predictions and hence actions in the same area as they have a good track record in. Organisations with decision-makers with good track records can likewise be more confident in their choices.
Crucially, forecasting is a tool to test decision making, rather than a tool for good decision making. If your decision makers are found to be poor forecasters, that is a bad sign, but if your decision making process doesn't involve forecasting, it's not a bad sign. It's not clear that it should.
Where to start
Some common recommendations for getting into forecasting are as follows:
- Track your personal forecasts. Get a notebook, spreadsheet or fatebook.io and write what you think will happen and a % or odds chance. Follow up on it later.
- Bet fake money on manifold.markets. It's still pretty addictive, so if you have a gambling problem, please avoid.
- Take part in the monthly estimation game. Test your ability to estimate quantities. This is correlated with your ability to navigate the world well
- Forecast on metaculus.com. Questions are often pretty focused on geopolitics
- Read Superforecasting by Philip Tetlock. This is if books are a way you learn well.
Forecasting Techniques
Forecasting is hard but many top forecasters use common techniques. This suggests that forecasting is a skill that can be learnt and practised.
Base rates
Reference Class Forecasting on Wikipedia
Suppose we are trying to find the probability that an event will occur within the next 5 years. One good place to start is by asking "of all similar time periods, what fraction of the time does this event occur?". This is the base rate.
If we want to know the probability that Joe Biden is President of the United States on Nov. 1st, 2024, we could ask
- What fraction of presidential terms are fully completed (last all 4 years)? The answer to this is 49 out of the 58 total terms, or around 84%.
- On the other hand, we know that Biden has already made it through 288 days of his term. If we remove the 5 presidents who left office before that, there are 49 out of 53 or around 92%.
- But alternately, Joe Biden is pretty old (78 to be exact). If we look up death rate per year in actuarial tables, it's around 5.1% per year, so this leaves him with a ~15% chance of death or a 85% chance of surviving his term.
These are all examples of using base rates. [These examples are taken from Base Rates and Reference Classes by jsteinhardt.]
Base rates represent the outside view for a given question. They are a good place to start but can often be improved on by updating the probability according to an inside view.
Note that there are often several reference classes we could use, each implying a different base rate. The problem of deciding which class to use is known as the reference class problem.
Calibration training
A forecaster is said to be calibrated if the events they say have a X% chance of happening, happen X% of the time.
Most people are overconfident. When they say an event has a 99% chance of happening, often the events happen much less frequently than that.
This natural overconfidence can be corrected with calibration training. In calibration training, you are asked to answer a set of factual questions, assigning a probability to each of your answers.
A calibration exercise can be found here: https://www.quantifiedintuitions.org/calibration
Question decomposition
Much like Fermi estimation, questions about future events can often be decomposed into many different questions, these questions can be answered, and the answers to these questions can be used to reconstruct an answer to the original question.
Suppose you are interested in whether AI will cause a catastrophe by 2100. For AI to cause such an event, several things need to be true: (1) it needs to be possible to build advanced AI with agentic planning and strategic awareness by 2100, (2) there need to be strong incentives to apply such a system, (3) it needs to be difficult to align such a system should it be deployed, (4) a deployed and unaligned AI would act in unintended and high-impact power seeking ways causing trillions of dollars in damage, (5) of these consequences will result in the permanent disempowerment of all humanity and (6) this disempowerment will constitute an existential catastrophe. Taking the probabilities that Eli Lifland assigned to each question gives a 80%, 85%, 75%, 90%, 80% and 95% chance of events 1 through 6 respectively. Since each event is conditional on the ones before it, we can find the probability of the original question by multiplying all the probabilities together. This gives Eli Lifland a probability of existential risk from misaligned AI before 2100 to be approximately 35%. For more detail see Eli's original post here.
Decomposing questions into their constituent parts, assigning probabilities to these sub-questions, and combining these probabilities to answer the original questions is believed to improve forecasts. This is because, while each forecast is noisy, combining the estimates from many questions cancels the noise and leaves us with the signal.
Question decomposition is also good at increasing epistemic legibility. It helps forecasters to communicate to others why they've made the forecast that they did and it allows them to identify their specific points of disagreement.
Premortems
A premortem is a strategy used once you've assigned a probability to an event. You ask yourself to imagine that the forecast was wrong and you then work backwards to determine what could potentially have caused this.
It is simply a way to reframe the question "in what ways might I be wrong?" but in a way that reduces motivated reasoning caused by attachment to the bottom line.
Practice
Getting Started on the Forecasting Wiki
While the above techniques are useful, they are no substitute for actually making predictions. Get out there and make predictions! Use the above techniques. Keep track of your predictions. Periodically evaluate questions that have been resolved and review your performance. Assess the degree to which you are calibrated. Look out for systematic mistakes that you might be making. Make more predictions! Over time, like with any skill, your ability can and should improve.
Other Resources
Other resources include:
- Superforcasting by Philip Tetlock and Dan Gardener
- Intro to Forecasting by Alex Lawson
- Forecasting Newsletter by Nuño Sempere
- Map of the Prediction Market & Forecasting Ecosystem by Saul Munn
Forecasting Research
Forecasting beyond 3 years is not good. Anything above .25 is worse than random. Many questions are too specific and too far away for forecasting to be useful to them (Dillon 2020).
Difficulties in Applying Forecasting
Decision makers largely don't trust forecasts. Even if you had the perfect set of 1000 forecast that gave policy recommendations (which is usually not the case), decision makers would need to want to act on them. That they don't is a significant bottleneck to successfully use forecasting.
It is difficult to forecast things policy makers actually care about. Forecasting sites forecast things like "will Putin leave power" rather than "If Putin leaves power between July18th and the end of Aug how will that affect the likelihood of a rogue nuclear warhead". This question probably still isn't specific enough to be useful - it doesn't forecast specific policy outcomes. And if it were, decision makers would have to trust the results, which they currently largely don't. This is related to the problem of specifying good forecasting questions, especially for nebulous domains.
State of the Art
For many years there have been calls to apply forecasting techniques to non-academic domains including journalism, policy, investing and business strategy. Several organisations now exist within these niche.
Metaculus
Metaculus is a popular and established web platform for forecasting. Their questions mainly focus on geopolitics, the coronavirus pandemic and topics of interest to Effective Altruism.
They host prediction competitions with real money prizes and collect and track public predictions made by various figures.
Cultivate Labs
Cultivate Labs build tools that companies can use to crowdsource information from among their employees. This helps leadership to understand the consensus of people working on the ground and use this to improve the decisions they make.
Kalshi
Kalshi provide real money prediction markets on geopolitical events. The financial options they provide are intended to be used as hedges for political risk.
Manifold.Markets
Manifold.Markets is a prediction market platform that uses play money. It is noteworthy for its ease of use, great UI and the fact that the market creator decides how the market resolves.
QURI
QURI is a research organisation that builds tools that make it easier to make good forecasts. Their most notable tool is Squiggle - a programming language designed to be used to make legible forecasts in a wide range of contexts.
Forecasters on twitter
- Top forecasters on @Metaculus:
- @SmoLurks (ranked #1 in 2021)
- @Jotto999 (#11 in 2019)
- @ryanbeck111 (#12 in 2021)
- @SchoeneggerPhil (#15 in 2023)
- @lxrjl (#15 in 2021)
- @JgaltTweets (#16 in 2023)
- @peterwildeford (#19 in 2023)
- @nextbigfuture (#25 in 2023)
- @draaglom (#26 in 2023)
- @EgeErdil2 (#27 in 2021)
- @LinchZhang (#29 in 2020)
- @beala (#57 in 2021)
- @NathanpmYoung (#72 in 2023)
- @MatthewJBar (#79 in 2021)
- Top forecasters on @INFERpub:
- @celloMolly (#6 in 2022)
- @NunoSempere (#7 in 2022)
- @cmeinel (#12 in 2023)
- Top forecasters on @GJ_Open:
- @leonardbarrett (#1 in News 2022 tournament)
- @juan_cambeiro (#3 in Coronavirus 2022)
- Top bettors on @Polymarket:
- Top bettors on @ManifoldMarkets:
- @firstuserhere ($1m play money lifetime profit)
- @ReplyHobbes ($600k)
- @Chrisbilbo ($450k) Top forecasting teams
- @swift_centre & @SamotsvetyF:
- Official @superforecaster: