BearScout scores a probability forecast in two parts: a standard Brier score against the outcome, and a comparison against the Brier score the market's ask price would have earned on the same question at the same moment. The second part is the whole point — and it forces three design decisions that sound pedantic until you build the thing: which price counts as "the market", when to snapshot it, and what to do when the order book is thin. This post documents our answers, including the ones we're not fully satisfied with.
A Brier score is the squared error between a probability forecast and a binary outcome: forecast 0.70 on an event that happens scores (1 − 0.70)² = 0.09; on one that doesn't, 0.49. Lower is better, and flat 50% guessing scores 0.25 (Brier, 1950 — the original paper is three pages and still worth reading).
But raw Brier has a known exploit: pick easy questions. Forecast 95% on near-certainties all day and your average looks superb while containing zero information the world didn't already have. Forecasting research handles this with skill scores relative to a baseline — typically climatology in weather forecasting (Jolliffe & Stephenson, Forecast Verification). For questions that trade on a prediction market, there's a sharper baseline available: the live price. It's an aggregated forecast from participants with strong incentives to be right, and it's free. Your skill is not "was I right?" but "was I more accurate than that price?"
Three candidates exist for "the market's probability" on a binary contract quoted in cents:
| Candidate | Definition | Failure mode |
|---|---|---|
| Last trade | Price of the most recent fill | Stale — can be hours old on quiet markets; ignores everything the book has learned since |
| Mid | (best bid + best ask) / 2 | Phantom — with a 40¢ bid and an 80¢ ask, "60%" is a number nobody will transact at |
| Ask | Lowest price a YES share can actually be bought for | Slightly pessimistic about the market's confidence; embeds half the spread |
We use the ask, for one reason: disagreeing with the market is only meaningful at a price where the disagreement could be expressed. If you believe YES is more likely than the market thinks, the executable version of that belief is buying at the ask. Scoring you against a phantom mid would credit you for beating a number that never existed. The cost of this choice is honest to name: the ask embeds spread, so on wide-spread markets the baseline is easier to beat on the YES side. Which is exactly why the thin-book section below exists.
For calls that lean NO, the same logic applies mirrored: the relevant executable price is the NO ask (equivalently, 100 minus the best bid on YES). A forecast below the market is scored against what it would cost to express "NO is underpriced".
Take a contract with the YES ask at 62¢ — the market-as-baseline says P(yes) = 0.62. You call 75%.
Note what this kills: matching the market earns exactly zero skill margin. Locking a call at the market's own probability is a no-op on the ladder. The only way up is justified disagreement, accumulated over many settled calls. One call proves nothing — as a rough rule of thumb you need 30–50 resolved forecasts before the average separates skill from luck.
Everything above assumes the ask is meaningful. On an illiquid market it isn't: a book showing bid 12¢ / ask 89¢ has an "ask" in name only. Scoring against it would hand out free skill margin for disagreeing with a price no one defends.
Our current handling is curation rather than correction: BearScout only offers calls on a curated pool of liquid markets, selected from series with consistent volume and tight books, instead of exposing the full exchange feed. We tried the full feed first; most of it was junk for scoring purposes — placeholder quotes, dead books, contracts with ambiguous resolution language. The concrete rules as of this writing: a market is only callable if its quoted spread is at most 10¢, there are at least 30 minutes to close, and forecasts are clamped to the 5%–95% range (the extreme tails are where scoring noise and resolution ambiguity dominate). Curating the pool costs breadth (no long-tail questions, a real limitation vs tournament platforms like Metaculus) but keeps every baseline executable.
The alternative we keep debating: allow wider books but score against a depth-weighted price, or against the ask with a spread-dependent handicap. Both add fairness in theory and unexplainable numbers in practice — a leaderboard people can't recompute by hand loses the property that makes scoring trustworthy. For now we've chosen legibility over coverage. If you have a scheme that keeps both, genuinely: tell us.
The baseline is the ask at the moment your call locks, stored with the call, immutable afterwards. Two reasons. First, hindsight drift: if the baseline could update, a call made against a 62¢ market that later trades to 90¢ would be scored against information you didn't have. Second, incentive integrity: a fixed snapshot means the game rewards being early — disagreeing with the market at 62¢ and being right scores more than joining the consensus at 90¢, the same way it would in any real forecasting context.
Settlement uses the exchange's own resolution. Voided or ambiguous markets void the call rather than guessing — no score is better than a fabricated one.
There is no money anywhere in this system: no deposits, no payouts, nothing to buy. The ladder ranks calibration statistics, and share cards carry the line "scored forecast — not investment performance" because that's literally what it is. If you want the practice loop: BearScout. If you keep your own forecast log in a spreadsheet, the free Brier & calibration calculator does the arithmetic on your history. The method matters more than the tool — write the number down before the outcome, score it against the hardest honest baseline you can find.
BearScout locks your probability on real prediction-market questions and Brier-scores it against the ask at call time. Free, pseudonymous, no real money.
Play BearScout →