Most people who think they're good at predicting things have never kept score. Forecasting is a trainable skill, but only if three things are in place: probabilities instead of opinions, a proper scoring rule (the Brier score), and a benchmark that's actually hard to beat — the prevailing market price. This page covers all three and how to build the habit for free.
"The Fed will probably cut" is unfalsifiable — "probably" can mean 55% or 90%, and after the fact you'll remember whichever was convenient. A real forecast is a number attached to a resolvable question with a deadline: "70% that the Fed cuts at the September meeting." The number is what makes scoring — and therefore improvement — possible. This is the core method behind forecasting research popularized by Philip Tetlock's Good Judgment Project: superforecasters differ from everyone else mostly in that they use fine-grained probabilities and update them often.
The Brier score is the mean squared error between your forecast and what happened (outcome = 1 if yes, 0 if no):
| Your forecast | Event happens | Event doesn't |
|---|---|---|
| 50% | 0.25 | 0.25 |
| 70% | 0.09 | 0.49 |
| 90% | 0.01 | 0.81 |
| 99% | 0.0001 | 0.98 |
Lower is better. Two properties make it the standard: it's proper — your expected score is best when you report your honest probability, so there's no gaming it — and it punishes overconfidence brutally. A 99% call that misses costs you roughly ten 70% calls' worth of score. Answering 50% on everything scores a flat 0.25, so a sustained average below 0.25 is the first sign your numbers carry information.
Here's the trap in raw Brier scores: you can flatter yourself by picking easy questions. "Will the sun rise tomorrow, 99.9%" pads your average without any skill. The fix is to score yourself relative to the market price at the moment you forecast. Prediction markets like Kalshi and Polymarket publish a live probability for every contract — that price is a free, aggregated forecast from many independent participants with strong incentives to be right. The interesting question is never "was I right?" but "was I more accurate than the price?"
If a contract trades at 80¢ and you forecast 85%, you're claiming to know something the crowd doesn't. Over many calls, consistently being more accurate than the price is evidence of real forecasting skill; matching the price means the market already knew what you knew; consistently losing to it means your intuitions are subtracting information. That's a hard, honest bar, which is why it's the right way to practice — the same logic drives how an AI agent studies mispriced probability.
Accuracy and calibration are different failures. Take every forecast you made at ~70% and count how many resolved yes. If it's about 70%, you're calibrated at that level; if it's 55%, you're overconfident. Most beginners are overconfident in the 80–95% range and underconfident near 50–60%. You need volume for this to be readable — roughly 30–50 resolved forecasts before your Brier average separates skill from luck, and 100+ before calibration buckets stabilize. Which leads to the last step:
Forecasting skill compounds through reps and feedback loops, not through occasional big calls. Free ways to get reps:
Every trade is an implicit forecast — buying a contract at 60¢ is saying "more likely than 60%." Practicing forecasting without positions is how you find out whether your probability sense is reliable at all, the same way paper trading tests execution mechanics in simulation. Both are about self-knowledge: they tell you where your judgment is calibrated and where it isn't, before any decision rests on it.
The mean squared error between your probability and the outcome (1 or 0). Lower is better; a coin-flip 50% answer always scores 0.25, so beating 0.25 on average means your forecasts carry information.
Because outcome-only scoring rewards picking easy questions. The market price is a free, aggregated forecast — being consistently more accurate than it means you genuinely knew something the crowd didn't; matching it means the crowd already knew.
Roughly 30–50 resolved forecasts for the Brier average, 100+ for readable calibration buckets. Volume and feedback beat occasional big calls.
Yes — forecasting tournaments (Good Judgment Open, Metaculus) and free games like BearScout score you on real questions with nothing at stake. The calibration skill transfers directly.
Accuracy is whether your leans were right. Calibration is whether your 70% calls happen about 70% of the time. Overconfidence is a calibration failure, and it's the one that costs money when a position is attached.
BearScout scores your forecasts against live prediction-market prices with a Brier-based ladder. No money at stake — just you against the crowd.
Play BearScout →